Hosted control planes

Procedure

1. Edit the

   HostedCluster

   custom resource (CR) on the hosting cluster by running the following command:

   $ oc edit hostedcluster <hosted_cluster_name> -n <hosted_cluster_namespace>

2. Add the OAuth configuration in the

   HostedCluster

   CR by using the following example:

   apiVersion: hypershift.openshift.io/v1alpha1
   kind: HostedCluster
   metadata:
     name: <hosted_cluster_name> 

   1

   
     namespace: <hosted_cluster_namespace> 

   2

   
   spec:
     configuration:
       oauth:
         identityProviders:
         - openID: 

   3

   
             claims:
               email: 

   4

   
                 - <email_address>
               name: 

   5

   
                 - <display_name>
               preferredUsername: 

   6

   
                 - <preferred_username>
             clientID: <client_id> 

   7

   
             clientSecret:
               name: <client_id_secret_name> 

   8

   
             issuer: https://example.com/identity 

   9

   
           mappingMethod: lookup 

   10

   
           name: IAM
           type: OpenID

   1

   Specifies your hosted cluster name.

   2

   Specifies your hosted cluster namespace.

   3

   This provider name is prefixed to the value of the identity claim to form an identity name. The provider name is also used to build the redirect URL.

   4

   Defines a list of attributes to use as the email address.

   5

   Defines a list of attributes to use as a display name.

   6

   Defines a list of attributes to use as a preferred user name.

   7

   Defines the ID of a client registered with the OpenID provider. You must allow the client to redirect to the https://oauth-openshift.apps.<cluster_name>.<cluster_domain>/oauth2callback/<idp_provider_name> URL.

   8

   Defines a secret of a client registered with the OpenID provider.

   9

   The Issuer Identifier described in the OpenID spec. You must use https without query or fragment component.

   10

   Defines a mapping method that controls how mappings are established between identities of this provider and User objects.

3. Save the file to apply the changes.

Procedure

1. Navigate to Home → API Explorer.
2. Use the Filter by kind box to search for your

   HostedCluster

   resource.
3. Click the

   HostedCluster

   resource that you want to edit.
4. Click the Instances tab.
5. Click the Options menu next to your hosted cluster name entry and click Edit HostedCluster.

6. Add the OAuth configuration in the YAML file:

   spec:
     configuration:
       oauth:
         identityProviders:
         - openID: 

   1

   
             claims:
               email: 

   2

   
                 - <email_address>
               name: 

   3

   
                 - <display_name>
               preferredUsername: 

   4

   
                 - <preferred_username>
             clientID: <client_id> 

   5

   
             clientSecret:
               name: <client_id_secret_name> 

   6

   
             issuer: https://example.com/identity 

   7

   
           mappingMethod: lookup 

   8

   
           name: IAM
           type: OpenID

   1

   This provider name is prefixed to the value of the identity claim to form an identity name. The provider name is also used to build the redirect URL.

   2

   Defines a list of attributes to use as the email address.

   3

   Defines a list of attributes to use as a display name.

   4

   Defines a list of attributes to use as a preferred user name.

   5

   Defines the ID of a client registered with the OpenID provider. You must allow the client to redirect to the https://oauth-openshift.apps.<cluster_name>.<cluster_domain>/oauth2callback/<idp_provider_name> URL.

   6

   Defines a secret of a client registered with the OpenID provider.

   7

   The Issuer Identifier described in the OpenID spec. You must use https without query or fragment component.

   8

   Defines a mapping method that controls how mappings are established between identities of this provider and User objects.

7. Click Save.

Procedure

1. Verify that the CCO is configured in a manual mode in your hosted cluster by running the following command:

   $ oc get cloudcredentials <hosted_cluster_name> -n <hosted_cluster_namespace> -o=jsonpath={.spec.credentialsMode}

   Expected output

   Manual

2. Verify that the value for the

   serviceAccountIssuer

   resource is not empty by running the following command:

   $ oc get authentication cluster --kubeconfig <hosted_cluster_name>.kubeconfig -o jsonpath --template '{.spec.serviceAccountIssuer }'

   Example output

   https://aos-hypershift-ci-oidc-29999.s3.us-east-2.amazonaws.com/hypershift-ci-29999

Procedure

1. Update your Operator project’s

   ClusterServiceVersion

   (CSV) object:

   1. Ensure your Operator has RBAC permission to create

      CredentialsRequests

      objects:

      Example 3.1. Example clusterPermissions list

      # ...
      install:
        spec:
          clusterPermissions:
          - rules:
            - apiGroups:
              - "cloudcredential.openshift.io"
              resources:
              - credentialsrequests
              verbs:
              - create
              - delete
              - get
              - list
              - patch
              - update
              - watch

   2. Add the following annotation to claim support for this method of CCO-based workflow with AWS STS:

      # ...
      metadata:
       annotations:
         features.operators.openshift.io/token-auth-aws: "true"

2. Update your Operator project code:

   1. Get the role ARN from the environment variable set on the pod by the

      Subscription

      object. For example:

      // Get ENV var
      roleARN := os.Getenv("ROLEARN")
      setupLog.Info("getting role ARN", "role ARN = ", roleARN)
      webIdentityTokenPath := "/var/run/secrets/openshift/serviceaccount/token"

   2. Ensure you have a

      CredentialsRequest

      object ready to be patched and applied. For example:

      Example 3.2. Example CredentialsRequest object creation

      import (
         minterv1 "github.com/openshift/cloud-credential-operator/pkg/apis/cloudcredential/v1"
         corev1 "k8s.io/api/core/v1"
         metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
      )
      
      var in = minterv1.AWSProviderSpec{
         StatementEntries: []minterv1.StatementEntry{
            {
               Action: []string{
                  "s3:*",
               },
               Effect:   "Allow",
               Resource: "arn:aws:s3:*:*:*",
            },
         },
      	STSIAMRoleARN: "<role_arn>",
      }
      
      var codec = minterv1.Codec
      var ProviderSpec, _ = codec.EncodeProviderSpec(in.DeepCopyObject())
      
      const (
         name      = "<credential_request_name>"
         namespace = "<namespace_name>"
      )
      
      var CredentialsRequestTemplate = &minterv1.CredentialsRequest{
         ObjectMeta: metav1.ObjectMeta{
             Name:      name,
             Namespace: "openshift-cloud-credential-operator",
         },
         Spec: minterv1.CredentialsRequestSpec{
            ProviderSpec: ProviderSpec,
            SecretRef: corev1.ObjectReference{
               Name:      "<secret_name>",
               Namespace: namespace,
            },
            ServiceAccountNames: []string{
               "<service_account_name>",
            },
            CloudTokenPath:   "",
         },
      }

      Alternatively, if you are starting from a

      CredentialsRequest

      object in YAML form (for example, as part of your Operator project code), you can handle it differently:

      Example 3.3. Example CredentialsRequest object creation in YAML form

      // CredentialsRequest is a struct that represents a request for credentials
      type CredentialsRequest struct {
        APIVersion string `yaml:"apiVersion"`
        Kind       string `yaml:"kind"`
        Metadata   struct {
           Name      string `yaml:"name"`
           Namespace string `yaml:"namespace"`
        } `yaml:"metadata"`
        Spec struct {
           SecretRef struct {
              Name      string `yaml:"name"`
              Namespace string `yaml:"namespace"`
           } `yaml:"secretRef"`
           ProviderSpec struct {
              APIVersion     string `yaml:"apiVersion"`
              Kind           string `yaml:"kind"`
              StatementEntries []struct {
                 Effect   string   `yaml:"effect"`
                 Action   []string `yaml:"action"`
                 Resource string   `yaml:"resource"`
              } `yaml:"statementEntries"`
              STSIAMRoleARN   string `yaml:"stsIAMRoleARN"`
           } `yaml:"providerSpec"`
      
           // added new field
            CloudTokenPath   string `yaml:"cloudTokenPath"`
        } `yaml:"spec"`
      }
      
      // ConsumeCredsRequestAddingTokenInfo is a function that takes a YAML filename and two strings as arguments
      // It unmarshals the YAML file to a CredentialsRequest object and adds the token information.
      func ConsumeCredsRequestAddingTokenInfo(fileName, tokenString, tokenPath string) (*CredentialsRequest, error) {
        // open a file containing YAML form of a CredentialsRequest
        file, err := os.Open(fileName)
        if err != nil {
           return nil, err
        }
        defer file.Close()
      
        // create a new CredentialsRequest object
        cr := &CredentialsRequest{}
      
        // decode the yaml file to the object
        decoder := yaml.NewDecoder(file)
        err = decoder.Decode(cr)
        if err != nil {
           return nil, err
        }
      
        // assign the string to the existing field in the object
        cr.Spec.CloudTokenPath = tokenPath
      
        // return the modified object
        return cr, nil
      }

      Note

      Adding a

      CredentialsRequest

      object to the Operator bundle is not currently supported.

   3. Add the role ARN and web identity token path to the credentials request and apply it during Operator initialization:

      Example 3.4. Example applying CredentialsRequest object during Operator initialization

      // apply credentialsRequest on install
      credReq := credreq.CredentialsRequestTemplate
      credReq.Spec.CloudTokenPath = webIdentityTokenPath
      
      c := mgr.GetClient()
      if err := c.Create(context.TODO(), credReq); err != nil {
         if !errors.IsAlreadyExists(err) {
            setupLog.Error(err, "unable to create CredRequest")
            os.Exit(1)
         }
      }

   4. Ensure your Operator can wait for a

      Secret

      object to show up from the CCO, as shown in the following example, which is called along with the other items you are reconciling in your Operator:

      Example 3.5. Example wait for Secret object

      // WaitForSecret is a function that takes a Kubernetes client, a namespace, and a v1 "k8s.io/api/core/v1" name as arguments
      // It waits until the secret object with the given name exists in the given namespace
      // It returns the secret object or an error if the timeout is exceeded
      func WaitForSecret(client kubernetes.Interface, namespace, name string) (*v1.Secret, error) {
        // set a timeout of 10 minutes
        timeout := time.After(10 * time.Minute) 

      1

      
      
        // set a polling interval of 10 seconds
        ticker := time.NewTicker(10 * time.Second)
      
        // loop until the timeout or the secret is found
        for {
           select {
           case <-timeout:
              // timeout is exceeded, return an error
              return nil, fmt.Errorf("timed out waiting for secret %s in namespace %s", name, namespace)
                 // add to this error with a pointer to instructions for following a manual path to a Secret that will work on STS
           case <-ticker.C:
              // polling interval is reached, try to get the secret
              secret, err := client.CoreV1().Secrets(namespace).Get(context.Background(), name, metav1.GetOptions{})
              if err != nil {
                 if errors.IsNotFound(err) {
                    // secret does not exist yet, continue waiting
                    continue
                 } else {
                    // some other error occurred, return it
                    return nil, err
                 }
              } else {
                 // secret is found, return it
                 return secret, nil
              }
           }
        }
      }

      1

      The timeout value is based on an estimate of how fast the CCO might detect an added CredentialsRequest object and generate a Secret object. You might consider lowering the time or creating custom feedback for cluster administrators that could be wondering why the Operator is not yet accessing the cloud resources.

   5. Set up the AWS configuration by reading the secret created by the CCO from the credentials request and creating the AWS config file containing the data from that secret:

      Example 3.6. Example AWS configuration creation

      func SharedCredentialsFileFromSecret(secret *corev1.Secret) (string, error) {
         var data []byte
         switch {
         case len(secret.Data["credentials"]) > 0:
             data = secret.Data["credentials"]
         default:
             return "", errors.New("invalid secret for aws credentials")
         }
      
      
         f, err := ioutil.TempFile("", "aws-shared-credentials")
         if err != nil {
             return "", errors.Wrap(err, "failed to create file for shared credentials")
         }
         defer f.Close()
         if _, err := f.Write(data); err != nil {
             return "", errors.Wrapf(err, "failed to write credentials to %s", f.Name())
         }
         return f.Name(), nil
      }

      Important

      The secret is assumed to exist, but your Operator code should wait and retry when using this secret to give time to the CCO to create the secret.

      Additionally, the wait period should eventually time out and warn users that the OpenShift Container Platform cluster version, and therefore the CCO, might be an earlier version that does not support the

      CredentialsRequest

      object workflow with STS detection. In such cases, instruct users that they must add a secret by using another method.

   6. Configure the AWS SDK session, for example:

      Example 3.7. Example AWS SDK session configuration

      sharedCredentialsFile, err := SharedCredentialsFileFromSecret(secret)
      if err != nil {
         // handle error
      }
      options := session.Options{
         SharedConfigState: session.SharedConfigEnable,
         SharedConfigFiles: []string{sharedCredentialsFile},
      }

Procedure

1. To create a

   MachineConfig

   object inside of a config map in the management cluster, enter the following information:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <configmap_name>
     namespace: clusters
   data:
     config: |
       apiVersion: machineconfiguration.openshift.io/v1
       kind: MachineConfig
       metadata:
         labels:
           machineconfiguration.openshift.io/role: worker
         name: <machineconfig_name>
       spec:
         config:
           ignition:
             version: 3.2.0
           storage:
             files:
             - contents:
                 source: data:...
               mode: 420
               overwrite: true
               path: ${PATH} 

   1

   1

   Sets the path on the node where the MachineConfig object is stored.

2. After you add the object to the config map, you can apply the config map to the node pool as follows:

   $ oc edit nodepool <nodepool_name> --namespace <hosted_cluster_namespace>

   apiVersion: hypershift.openshift.io/v1alpha1
   kind: NodePool
   metadata:
   # ...
     name: nodepool-1
     namespace: clusters
   # ...
   spec:
     config:
     - name: <configmap_name> 

   1

   
   # ...

   1

   Replace <configmap_name> with the name of your config map.

Procedure

1. Add the kubelet configuration inside of a config map in the management cluster by entering the following information:

   Example ConfigMap object with the kubelet configuration

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <configmap_name> 

   1

   
     namespace: clusters
   data:
     config: |
       apiVersion: machineconfiguration.openshift.io/v1
       kind: KubeletConfig
       metadata:
         name: <kubeletconfig_name> 

   2

   
       spec:
         kubeletConfig:
           registerWithTaints:
           - key: "example.sh/unregistered"
             value: "true"
             effect: "NoExecute"

   1

   Replace <configmap_name> with the name of your config map.

   2

   Replace <kubeletconfig_name> with the name of the KubeletConfig resource.

2. Apply the config map to the node pool by entering the following command:

   $ oc edit nodepool <nodepool_name> --namespace clusters 

   1

   1

   Replace <nodepool_name> with the name of your node pool.

   Example NodePool resource configuration

   apiVersion: hypershift.openshift.io/v1alpha1
   kind: NodePool
   metadata:
   # ...
     name: nodepool-1
     namespace: clusters
   # ...
   spec:
     config:
     - name: <configmap_name> 

   1

   
   # ...

   1

   Replace <configmap_name> with the name of your config map.

Procedure

1. Create a config map that contains a valid tuned manifest, and reference the manifest in a node pool. In the following example, a

   Tuned

   manifest defines a profile that sets

   vm.dirty_ratio

   to 55 on nodes that contain the

   tuned-1-node-label

   node label with any value. Save the following

   ConfigMap

   manifest in a file named

   tuned-1.yaml

   :

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: tuned-1
         namespace: clusters
       data:
         tuning: |
           apiVersion: tuned.openshift.io/v1
           kind: Tuned
           metadata:
             name: tuned-1
             namespace: openshift-cluster-node-tuning-operator
           spec:
             profile:
             - data: |
                 [main]
                 summary=Custom OpenShift profile
                 include=openshift-node
                 [sysctl]
                 vm.dirty_ratio="55"
               name: tuned-1-profile
             recommend:
             - priority: 20
               profile: tuned-1-profile

   Note

   If you do not add any labels to an entry in the

   spec.recommend

   section of the Tuned spec, node-pool-based matching is assumed, so the highest priority profile in the

   spec.recommend

   section is applied to nodes in the pool. Although you can achieve more fine-grained node-label-based matching by setting a label value in the Tuned

   .spec.recommend.match

   section, node labels will not persist during an upgrade unless you set the

   .spec.management.upgradeType

   value of the node pool to

   InPlace

   .

2. Create the

   ConfigMap

   object in the management cluster:

   $ oc --kubeconfig="$MGMT_KUBECONFIG" create -f tuned-1.yaml

3. Reference the

   ConfigMap

   object in the

   spec.tuningConfig

   field of the node pool, either by editing a node pool or creating one. In this example, assume that you have only one

   NodePool

   , named

   nodepool-1

   , which contains 2 nodes.

       apiVersion: hypershift.openshift.io/v1alpha1
       kind: NodePool
       metadata:
         ...
         name: nodepool-1
         namespace: clusters
       ...
       spec:
         ...
         tuningConfig:
         - name: tuned-1
       status:
       ...

   Note

   You can reference the same config map in multiple node pools. In hosted control planes, the Node Tuning Operator appends a hash of the node pool name and namespace to the name of the Tuned CRs to distinguish them. Outside of this case, do not create multiple TuneD profiles of the same name in different Tuned CRs for the same hosted cluster.

1. List the

   Tuned

   objects in the hosted cluster:

   $ oc --kubeconfig="$HC_KUBECONFIG" get tuned.tuned.openshift.io -n openshift-cluster-node-tuning-operator

   Example output

   NAME       AGE
   default    7m36s
   rendered   7m36s
   tuned-1    65s

2. List the

   Profile

   objects in the hosted cluster:

   $ oc --kubeconfig="$HC_KUBECONFIG" get profile.tuned.openshift.io -n openshift-cluster-node-tuning-operator

   Example output

   NAME                           TUNED            APPLIED   DEGRADED   AGE
   nodepool-1-worker-1            tuned-1-profile  True      False      7m43s
   nodepool-1-worker-2            tuned-1-profile  True      False      7m14s

   Note

   If no custom profiles are created, the

   openshift-node

   profile is applied by default.

3. To confirm that the tuning was applied correctly, start a debug shell on a node and check the sysctl values:

   $ oc --kubeconfig="$HC_KUBECONFIG" debug node/nodepool-1-worker-1 -- chroot /host sysctl vm.dirty_ratio

   Example output

   vm.dirty_ratio = 55

Procedure

1. Create a namespace and an Operator group:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sriov-network-operator
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sriov-network-operators
     namespace: openshift-sriov-network-operator
   spec:
     targetNamespaces:
     - openshift-sriov-network-operator

2. Create a subscription to the SR-IOV Operator:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sriov-network-operator-subsription
     namespace: openshift-sriov-network-operator
   spec:
     channel: stable
     name: sriov-network-operator
     config:
       nodeSelector:
         node-role.kubernetes.io/worker: ""
     source: s/qe-app-registry/redhat-operators
     sourceNamespace: openshift-marketplace

Verification

1. To verify that the SR-IOV Operator is ready, run the following command and view the resulting output:

   $ oc get csv -n openshift-sriov-network-operator

   Example output

   NAME                                         DISPLAY                   VERSION               REPLACES                                     PHASE
   sriov-network-operator.4.15.0-202211021237   SR-IOV Network Operator   4.15.0-202211021237   sriov-network-operator.4.15.0-202210290517   Succeeded

2. To verify that the SR-IOV pods are deployed, run the following command:

   $ oc get pods -n openshift-sriov-network-operator

Procedure

1. Create a Butane config file,

   99-worker-chrony.bu

   , that includes the contents of the

   chrony.conf

   file. For more information about Butane, see "Creating machine configs with Butane".

   Example 99-worker-chrony.bu configuration

   # ...
   variant: openshift
   version: 4.15.0
   metadata:
     name: 99-worker-chrony
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644  

   1

   
       overwrite: true
       contents:
         inline: |
           pool 0.rhel.pool.ntp.org iburst  

   2

   
           driftfile /var/lib/chrony/drift
           makestep 1.0 3
           rtcsync
           logdir /var/log/chrony
   # ...

   1

   Specify an octal value mode for the mode field in the machine config file. After creating the file and applying the changes, the mode field is converted to a decimal value.

   2

   Specify any valid, reachable time source, such as the one provided by your Dynamic Host Configuration Protocol (DHCP) server.

   Note

   For machine-to-machine communication, the NTP on the User Datagram Protocol (UDP) port is

   123

   . If you configured an external NTP time server, you must open UDP port

   123

   .

2. Use Butane to generate a

   MachineConfig

   object file,

   99-worker-chrony.yaml

   , that contains a configuration that Butane sends to the nodes. Run the following command:

   $ butane 99-worker-chrony.bu -o 99-worker-chrony.yaml

   Example 99-worker-chrony.yaml configuration

   # Generated by Butane; do not edit
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: <machineconfig_name>
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - contents:
               source: data:...
             mode: 420
             overwrite: true
             path: /example/path

3. Add the contents of the

   99-worker-chrony.yaml

   file inside of a config map in the management cluster:

   Example config map

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: <configmap_name>
     namespace: <namespace>  

   1

   
   data:
     config: |
       apiVersion: machineconfiguration.openshift.io/v1
       kind: MachineConfig
       metadata:
         labels:
           machineconfiguration.openshift.io/role: worker
         name: <machineconfig_name>
       spec:
         config:
           ignition:
             version: 3.2.0
           storage:
             files:
             - contents:
                 source: data:...
               mode: 420
               overwrite: true
               path: /example/path
   # ...

   1

   Replace <namespace> with the name of your namespace where you created the node pool, such as clusters.

4. Apply the config map to your node pool by running the following command:

   $ oc edit nodepool <nodepool_name> --namespace <hosted_cluster_namespace>

   Example NodePool configuration

   apiVersion: hypershift.openshift.io/v1alpha1
   kind: NodePool
   metadata:
   # ...
     name: nodepool-1
     namespace: clusters
   # ...
   spec:
     config:
     - name: <configmap_name> 

   1

   
   # ...

   1

   Replace <configmap_name> with the name of your config map.

5. Add the list of your NTP servers in the

   infra-env.yaml

   file, which defines the

   InfraEnv

   custom resource (CR):

   Example infra-env.yaml file

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   # ...
   spec:
     additionalNTPSources:
     - <ntp_server> 

   1

   
     - <ntp_server1>
     - <ntp_server2>
   # ...

   1

   Replace <ntp_server> with the name of your NTP server. For more details about creating a host inventory and the InfraEnv CR, see "Creating a host inventory".

6. Apply the

   InfraEnv

   CR by running the following command:

   $ oc apply -f infra-env.yaml

Procedure

1. Open the

   HostedCluster

   CR for editing on the hosting cluster by running the following command:

   $ oc edit hostedcluster <hosted_cluster_name> -n <hosted_cluster_namespace>

2. Define the feature set by entering a value in the

   featureSet

   field. For example:

   apiVersion: hypershift.openshift.io/v1beta1
   kind: HostedCluster
   metadata:
     name: <hosted_cluster_name> 

   1

   
     namespace: <hosted_cluster_namespace> 

   2

   
   spec:
     configuration:
       featureGate:
         featureSet: TechPreviewNoUpgrade 

   3

   1

   Specifies your hosted cluster name.

   2

   Specifies your hosted cluster namespace.

   3

   This feature set is a subset of the current Technology Preview features.

   Warning

   Enabling the

   TechPreviewNoUpgrade

   feature set on your cluster cannot be undone and prevents minor version updates. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them. Do not enable this feature set on production clusters.

3. Save the file to apply the changes.

Procedure

1. Create a secret with your custom certificate by entering the following command:

   $ oc create secret tls sample-hosted-kas-custom-cert \
     --cert=path/to/cert.crt \
     --key=path/to/key.key \
     -n <hosted_cluster_namespace>

2. Update your

   HostedCluster

   configuration with the custom certificate details, as shown in the following example:

   spec:
     configuration:
       apiServer:
         servingCerts:
           namedCertificates:
           - names: 

   1

   
             - api-custom-cert-sample-hosted.sample-hosted.example.com
             servingCertificate: 

   2

   
               name: sample-hosted-kas-custom-cert

   1

   The list of DNS names that the certificate is valid for.

   2

   The name of the secret that contains the custom certificate.

3. Apply the changes to your

   HostedCluster

   configuration by entering the following command:

   $ oc apply -f <hosted_cluster_config>.yaml

Procedure

1. In your provider platform, configure the DNS record so that the

   kubeAPIServerDNSName

   URL points to the IP address that the Kubernetes API server is being exposed to. The DNS record must be properly configured and resolvable from your cluster.

   Example command to configure the DNS record

   $ dig + short kubeAPIServerDNSName

2. In your

   HostedCluster

   specification, modify the

   kubeAPIServerDNSName

   field, as shown in the following example:

   apiVersion: hypershift.openshift.io/v1beta1
   kind: HostedCluster
   metadata:
     name: <hosted_cluster_name>
     namespace: <hosted_cluster_namespace>
   spec:
     configuration:
       apiServer:
         servingCerts:
           namedCertificates:
           - names: 

   1

   
             - api-custom-cert-sample-hosted.sample-hosted.example.com
             servingCertificate: 

   2

   
               name: sample-hosted-kas-custom-cert
     kubeAPIServerDNSName: api-custom-cert-sample-hosted.sample-hosted.example.com 

   3

   
   # ...

   1

   The list of DNS names that the certificate is valid for. The names listed in this field cannot be the same as the names specified in the spec.servicePublishingStrategy.*hostname field.

   2

   The name of the secret that contains the custom certificate.

   3

   This field accepts a URI that will be used as the API server endpoint.

3. Apply the configuration by entering the following command:

   $ oc -f <hosted_cluster_spec>.yaml

   After the configuration is applied, the HyperShift Operator generates a new

   kubeconfig

   secret that points to your custom DNS domain.

4. Retrieve the

   kubeconfig

   secret by using the CLI or the console.

   1. To retrieve the secret by using the CLI, enter the following command:

      $ kubectl get secret <hosted_cluster_name>-custom-admin-kubeconfig \
        -n <cluster_namespace> \
        -o jsonpath='{.data.kubeconfig}' | base64 -d

   2. To retrieve the secret by using the console, go to your hosted cluster and click Download Kubeconfig.

      Note

      You cannot consume the new

      kubeconfig

      secret by using the show login command option in the console.

Procedure

1. Verify that the DNS record is properly configured and resolved.

2. Check that the TLS certificates for the custom domain are valid, verifying that the SAN is correct for your domain, by entering the following command:

   $ oc get secret \
     -n clusters <serving_certificate_name> \
     -o jsonpath='{.data.tls\.crt}' | base64 \
     -d |openssl x509 -text -noout -

3. Ensure that network connectivity to the custom domain is working.

4. In the

   HostedCluster

   resource, verify that the status shows the correct custom

   kubeconfig

   information, as shown in the following example:

   Example HostedCluster status

   status:
     customKubeconfig:
       name: sample-hosted-custom-admin-kubeconfig

5. Check the

   kube-apiserver

   logs in the

   HostedControlPlane

   namespace by entering the following command:

   $ oc logs -n <hosted_control_plane_namespace> \
     -l app=kube-apiserver -f -c kube-apiserver

1. Upgrade an OpenShift Container Platform cluster to the latest version. For more information, see "Updating a cluster using the web console" or "Updating a cluster using the CLI".
2. Upgrade the multicluster engine Operator to the latest version. For more information, see "Updating installed Operators".
3. Upgrade the hosted cluster and node pools from the previous OpenShift Container Platform version to the latest version. For more information, see "Updating a control plane in a hosted cluster" and "Updating node pools in a hosted cluster".

Procedure

1. Add the

   hypershift.openshift.io/force-upgrade-to=<openshift_release_image>

   annotation to the hosted cluster by entering the following command:

   $ oc annotate hostedcluster -n <hosted_cluster_namespace> <hosted_cluster_name> "hypershift.openshift.io/force-upgrade-to=<openshift_release_image>" --overwrite 

   1

   2

   1

   Replace <hosted_cluster_name> and <hosted_cluster_namespace> with your hosted cluster name and hosted cluster namespace, respectively.

   2

   The <openshift_release_image> variable specifies the new OpenShift Container Platform release image that you want to upgrade to, for example, quay.io/openshift-release-dev/ocp-release:4.y.z-x86_64. Replace <4.y.z> with the supported OpenShift Container Platform version.

2. Change the

   spec.release.image

   value in the hosted cluster by entering the following command:

   $ oc patch hostedcluster <hosted_cluster_name> -n <hosted_cluster_namespace> --type=merge -p '{"spec":{"release":{"image":"<openshift_release_image>"}}}'

Procedure

1. Select All clusters.
2. Navigate to Infrastructure → Clusters to view managed hosted clusters.
3. Click the Upgrade available link to update the control plane and node pools.

Procedure

1. Create the

   hypershift-operator-install-flags

   config map in the

   local-cluster

   namespace. See the following example configuration:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: hypershift-operator-install-flags
     namespace: local-cluster
   data:
     installFlagsToAdd: "--monitoring-dashboards --metrics-set=All" 

   1

   
     installFlagsToRemove: ""

   1

   The --monitoring-dashboards --metrics-set=All flag adds the monitoring dashboard for all metrics.

2. Wait a couple of minutes for the HyperShift Operator deployment in the

   hypershift

   namespace to be updated to include the following environment variable:

       - name: MONITORING_DASHBOARDS
         value: "1"

   When monitoring dashboards are enabled, for each hosted cluster that the HyperShift Operator manages, the Operator creates a config map named

   cp-<hosted_cluster_namespace>-<hosted_cluster_name>

   in the

   openshift-config-managed

   namespace, where

   <hosted_cluster_namespace>

   is the namespace of the hosted cluster and

   <hosted_cluster_name>

   is the name of the hosted cluster. As a result, a new dashboard is added in the administrative console of the management cluster.

3. To view the dashboard, log in to the management cluster’s console and go to the dashboard for the hosted cluster by clicking Observe → Dashboards.
4. Optional: To disable monitoring dashboards in a hosted cluster, remove the

   --monitoring-dashboards --metrics-set=All

   flag from the

   hypershift-operator-install-flags

   config map. When you delete a hosted cluster, its corresponding dashboard is also deleted.

1. Backing up the hosted cluster on the source management cluster
2. Restoring the hosted cluster on a destination management cluster
3. Deleting the hosted cluster from the source management cluster