Hosted control planes

Procedure

1. Create the ConfigMap resource by running the following command:

   $ oc create -f <your-config-map-file.yaml>

   Replace <your-config-map-file.yaml> with the name of your YAML file that contains your hcp-sizing-baseline config map.

2. Create the hcp-sizing-baseline config map in the local-cluster namespace to specify the measurements you want to override. Your config map might resemble the following YAML file:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: hcp-sizing-baseline
     namespace: local-cluster
   data:
     incrementalCPUUsagePer1KQPS: "9.0"
     memoryRequestPerHCP: "18"
     minimumQPSPerHCP: "50.0"

3. Delete the hypershift-addon-agent deployment to restart the hypershift-addon-agent pod by running the following command:

   $ oc delete deployment hypershift-addon-agent -n open-cluster-management-agent-addon

1. Log in to your hub cluster by running the following command:

   $ oc login

2. Edit the hypershift-addon-deploy-config add-on deployment configuration specification by running the following command:

   $ oc edit addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine

3. Add the disableMetrics=true customized variable to the specification, as shown in the following example:

   apiVersion: addon.open-cluster-management.io/v1alpha1
   kind: AddOnDeploymentConfig
   metadata:
     name: hypershift-addon-deploy-config
     namespace: multicluster-engine
   spec:
     customizedVariables:
     - name: hcMaxNumber
       value: "80"
     - name: hcThresholdNumber
       value: "60"
     - name: disableMetrics 1
       value: "true"

   1

   The disableMetrics=true customized variable disables metric service monitoring for both new and existing hypershift-addon managed cluster add-ons.

4. Apply the changes to the configuration specification by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Get the URL to download the hcp binary by running the following command:

   $ oc get ConsoleCLIDownload hcp-cli-download -o json | jq -r ".spec"

2. Download the hcp binary by running the following command:

   $ wget <hcp_cli_download_url> 1

   1

   Replace hcp_cli_download_url with the URL that you obtained from the previous step.

3. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

4. Make the hcp binary file executable by running the following command:

   $ chmod +x hcp

5. Move the hcp binary file to a directory in your path by running the following command:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. From the OpenShift Container Platform web console, click the Help icon → Command Line Tools.
2. Click Download hcp CLI for your platform.

3. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

4. Run the following command to make the binary file executable:

   $ chmod +x hcp

5. Run the following command to move the binary file to a directory in your path:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. Navigate to the content gateway and download the hcp binary.

2. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

3. Make the hcp binary file executable by running the following command:

   $ chmod +x hcp

4. Move the hcp binary file to a directory in your path by running the following command:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. Run the following command to enable the feature:

   $ oc patch mce multiclusterengine --type=merge -p '{"spec":{"overrides":{"components":[{"name":"hypershift","enabled": true}]}}}' 1

   1

   The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

2. Run the following command to verify that the hypershift and hypershift-local-hosting features are enabled in the MultiClusterEngine custom resource:

   $ oc get mce multiclusterengine -o yaml 1

   1

   The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

   Example output

   apiVersion: multicluster.openshift.io/v1
   kind: MultiClusterEngine
   metadata:
     name: multiclusterengine
   spec:
     overrides:
       components:
       - name: hypershift
         enabled: true
       - name: hypershift-local-hosting
         enabled: true

Procedure

1. Create an S3 bucket that has public access to host OIDC discovery documents for your clusters:

   1. To create the bucket in the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

   2. To create the bucket in a region other than the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name> \
        --create-bucket-configuration LocationConstraint=<region> \
        --region <region>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

2. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift operator.
3. Save the secret in the local-cluster namespace.

4. See the following table to verify that the secret contains the following fields:

   Table 4.1. Required fields for the AWS secret

   Field name	Description
   bucket	Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters.
   credentials	A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket.
   region	Specifies the region of the S3 bucket.

5. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> --from-file=credentials=<path>/.aws/credentials --from-literal=bucket=<s3_bucket> --from-literal=region=<region> -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. Run the following command to add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery:

   $ oc label secret hypershift-operator-oidc-provider-s3-credentials -n local-cluster cluster.open-cluster-management.io/backup=true

Procedure

1. Get the Amazon Resource Name (ARN) of your user by running the following command:

   $ aws sts get-caller-identity --query "Arn" --output text

   Example output

   arn:aws:iam::1234567890:user/<aws_username>

   Use this output as the value for <arn> in the next step.

2. Create a JSON file named trust-relationship.json that contains the trust relationship configuration for your role. See the following example:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": <arn> 1
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

   1

   Replace <arn> with the ARN of your user that you noted in the previous step.

3. Create the Identity and Access Management (IAM) role by running the following command:

   $ aws iam create-role \
   --role-name <name> \1
   --assume-role-policy-document file://<file_name>.json \2
   --query "Role.Arn"

   1

   Replace <name> with the role name, for example, hcp-cli-role.

   2

   Replace <file_name> with the file name, for example, assume-role-policy.json.

   Example output

   arn:aws:iam::820196288204:role/myrole

4. Create a JSON file named policy.json that contains the following permission policies for your role:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "EC2",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateDhcpOptions",
                   "ec2:DeleteSubnet",
                   "ec2:ReplaceRouteTableAssociation",
                   "ec2:DescribeAddresses",
                   "ec2:DescribeInstances",
                   "ec2:DeleteVpcEndpoints",
                   "ec2:CreateNatGateway",
                   "ec2:CreateVpc",
                   "ec2:DescribeDhcpOptions",
                   "ec2:AttachInternetGateway",
                   "ec2:DeleteVpcEndpointServiceConfigurations",
                   "ec2:DeleteRouteTable",
                   "ec2:AssociateRouteTable",
                   "ec2:DescribeInternetGateways",
                   "ec2:DescribeAvailabilityZones",
                   "ec2:CreateRoute",
                   "ec2:CreateInternetGateway",
                   "ec2:RevokeSecurityGroupEgress",
                   "ec2:ModifyVpcAttribute",
                   "ec2:DeleteInternetGateway",
                   "ec2:DescribeVpcEndpointConnections",
                   "ec2:RejectVpcEndpointConnections",
                   "ec2:DescribeRouteTables",
                   "ec2:ReleaseAddress",
                   "ec2:AssociateDhcpOptions",
                   "ec2:TerminateInstances",
                   "ec2:CreateTags",
                   "ec2:DeleteRoute",
                   "ec2:CreateRouteTable",
                   "ec2:DetachInternetGateway",
                   "ec2:DescribeVpcEndpointServiceConfigurations",
                   "ec2:DescribeNatGateways",
                   "ec2:DisassociateRouteTable",
                   "ec2:AllocateAddress",
                   "ec2:DescribeSecurityGroups",
                   "ec2:RevokeSecurityGroupIngress",
                   "ec2:CreateVpcEndpoint",
                   "ec2:DescribeVpcs",
                   "ec2:DeleteSecurityGroup",
                   "ec2:DeleteDhcpOptions",
                   "ec2:DeleteNatGateway",
                   "ec2:DescribeVpcEndpoints",
                   "ec2:DeleteVpc",
                   "ec2:CreateSubnet",
                   "ec2:DescribeSubnets"
               ],
               "Resource": "*"
           },
           {
               "Sid": "ELB",
               "Effect": "Allow",
               "Action": [
                   "elasticloadbalancing:DeleteLoadBalancer",
                   "elasticloadbalancing:DescribeLoadBalancers",
                   "elasticloadbalancing:DescribeTargetGroups",
                   "elasticloadbalancing:DeleteTargetGroup"
               ],
               "Resource": "*"
           },
           {
               "Sid": "IAMPassRole",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:*:iam::*:role/*-worker-role",
               "Condition": {
                   "ForAnyValue:StringEqualsIfExists": {
                       "iam:PassedToService": "ec2.amazonaws.com"
                   }
               }
           },
           {
               "Sid": "IAM",
               "Effect": "Allow",
               "Action": [
                   "iam:CreateInstanceProfile",
                   "iam:DeleteInstanceProfile",
                   "iam:GetRole",
                   "iam:UpdateAssumeRolePolicy",
                   "iam:GetInstanceProfile",
                   "iam:TagRole",
                   "iam:RemoveRoleFromInstanceProfile",
                   "iam:CreateRole",
                   "iam:DeleteRole",
                   "iam:PutRolePolicy",
                   "iam:AddRoleToInstanceProfile",
                   "iam:CreateOpenIDConnectProvider",
                   "iam:ListOpenIDConnectProviders",
                   "iam:DeleteRolePolicy",
                   "iam:UpdateRole",
                   "iam:DeleteOpenIDConnectProvider",
                   "iam:GetRolePolicy"
               ],
               "Resource": "*"
           },
           {
               "Sid": "Route53",
               "Effect": "Allow",
               "Action": [
                   "route53:ListHostedZonesByVPC",
                   "route53:CreateHostedZone",
                   "route53:ListHostedZones",
                   "route53:ChangeResourceRecordSets",
                   "route53:ListResourceRecordSets",
                   "route53:DeleteHostedZone",
                   "route53:AssociateVPCWithHostedZone",
                   "route53:ListHostedZonesByName"
               ],
               "Resource": "*"
           },
           {
               "Sid": "S3",
               "Effect": "Allow",
               "Action": [
                   "s3:ListAllMyBuckets",
                   "s3:ListBucket",
                   "s3:DeleteObject",
                   "s3:DeleteBucket"
               ],
               "Resource": "*"
           }
       ]
   }

5. Attach the policy.json file to your role by running the following command:

   $ aws iam put-role-policy \
     --role-name <role_name> \1
     --policy-name <policy_name> \2
     --policy-document file://policy.json 3

   1

   Replace <role_name> with the name of your role.

   2

   Replace <policy_name> with your policy name.

   3

   The policy.json file contains the permission policies for your role.

6. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command:

   $ aws sts get-session-token --output json > sts-creds.json

   Example sts-creds.json file

   {
                 "Credentials": {
                     "AccessKeyId": "ASIA1443CE0GN2ATHWJU",
                     "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1”,
                     "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=",
                     "Expiration": "2025-05-16T04:19:32+00:00"
                 }
             }

Procedure

1. Create an AWS credential secret for the HyperShift Operator and name it hypershift-operator-private-link-credentials. The secret must reside in the managed cluster namespace that is the namespace of the managed cluster being used as the management cluster. If you used local-cluster, create the secret in the local-cluster namespace.
2. See the following table to confirm that the secret contains the required fields:

Procedure

1. To create a hosted cluster on AWS, run the following command:

   $ hcp create cluster aws \
       --name <hosted_cluster_name> \1
       --infra-id <infra_id> \2
       --base-domain <basedomain> \3
       --sts-creds <path_to_sts_credential_file> \4
       --pull-secret <path_to_pull_secret> \5
       --region <region> \6
       --generate-ssh \
       --node-pool-replicas <node_pool_replica_count> \7
       --namespace <hosted_cluster_namespace> \8
       --role-arn <role_name> \9
       --render-into <file_name>.yaml 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify your infrastructure name. You must provide the same value for <hosted_cluster_name> and <infra_id>. Otherwise the cluster might not appear correctly in the multicluster engine for Kubernetes Operator console.

   3

   Specify your base domain, for example, example.com.

   4

   Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

   5

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   6

   Specify the AWS region name, for example, us-east-1.

   7

   Specify the node pool replica count, for example, 3.

   8

   By default, all HostedCluster and NodePool custom resources are created in the clusters namespace. You can use the --namespace <namespace> parameter, to create the HostedCluster and NodePool custom resources in a specific namespace.

   9

   Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   10

   If you want to indicate whether the EC2 instance runs on shared or single tenant hardware, include this field. The --render-into flag renders Kubernetes resources into the YAML file that you specify in this field. Then, continue to the next step to edit the YAML file.

2. If you included the --render-into flag in the previous command, edit the specified YAML file. Edit the NodePool specification in the YAML file to indicate whether the EC2 instance should run on shared or single-tenant hardware, similar to the following example:

   Example YAML file

   apiVersion: hypershift.openshift.io/v1beta1
   kind: NodePool
   metadata:
     name: <nodepool_name> 1
   spec:
     platform:
       aws:
         placement:
           tenancy: "default" 2

   1

   Specify the name of the NodePool resource.

   2

   Specify a valid value for tenancy: "default", "dedicated", or "host". Use "default" when node pool instances run on shared hardware. Use "dedicated" when each node pool instance runs on single-tenant hardware. Use "host" when node pool instances run on your pre-allocated dedicated hosts.

Verification

1. Verify the status of your hosted cluster to check that the value of AVAILABLE is True. Run the following command:

   $ oc get hostedclusters -n <hosted_cluster_namespace>

2. Get a list of your node pools by running the following command:

   $ oc get nodepools --namespace <hosted_cluster_namespace>

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig --to=- > kubeconfig-<hosted-cluster-name>

2. Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   dns                                        4.10.26   True        False         False      2m52s
   image-registry                             4.10.26   True        False         False      2m8s
   ingress                                    4.10.26   True        False         False      22m

3. You can also view the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-node-tuning-operator             tuned-dhw5p                                               1/1     Running            0               109s
   openshift-cluster-storage-operator                 cluster-storage-operator-5f784969f5-vwzgz                 1/1     Running            1 (113s ago)    20m
   openshift-cluster-storage-operator                 csi-snapshot-controller-6b7687b7d9-7nrfw                  1/1     Running            0               3m8s
   openshift-console                                  console-5cbf6c7969-6gk6z                                  1/1     Running            0               119s
   openshift-console                                  downloads-7bcd756565-6wj5j                                1/1     Running            0               4m3s
   openshift-dns-operator                             dns-operator-77d755cd8c-xjfbn                             2/2     Running            0               21m
   openshift-dns                                      dns-default-kfqnh                                         2/2     Running            0               113s

1. Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system

2. Apply the YAML content by entering the following command:

   $ oc apply -f configure-metallb.yaml

   Example output

   metallb.metallb.io/metallb created

3. Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: metallb
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.216.32-192.168.216.122 1

   1

   Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network.

4. Apply the YAML content by entering the following command:

   $ oc apply -f create-ip-address-pool.yaml

   Example output

   ipaddresspool.metallb.io/metallb created

5. Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: l2advertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
      - metallb

6. Apply the YAML content by entering the following command:

   $ oc apply -f l2advertisement.yaml

   Example output

   l2advertisement.metallb.io/metallb created

Procedure

1. Run the following command:

   NODEPOOL_NAME=${CLUSTER_NAME}-work
   NODEPOOL_REPLICAS=5
   
   $ oc scale nodepool/$NODEPOOL_NAME --namespace clusters --replicas=$NODEPOOL_REPLICAS

2. After a few moments, enter the following command to see the status of the node pool:

   $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

   Example output

   NAME                  STATUS   ROLES    AGE     VERSION
   example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
   example-n6prw         Ready    worker   116m    v1.27.4+18eadca
   example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
   example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
   example-twxns         Ready    worker   88s     v1.27.4+18eadca

Procedure

1. Verify that the HostedCluster resource transitioned to the completed state by entering the following command:

   $ oc get --namespace clusters hostedclusters <hosted_cluster_name>

   Example output

   NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   clusters    example   4.12.2    example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands:

   $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig

   $ oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.12.2   True        False         False      2m38s
   csi-snapshot-controller                    4.12.2   True        False         False      4m3s
   dns                                        4.12.2   True        False         False      2m52s
   image-registry                             4.12.2   True        False         False      2m8s
   ingress                                    4.12.2   True        False         False      22m
   kube-apiserver                             4.12.2   True        False         False      23m
   kube-controller-manager                    4.12.2   True        False         False      23m
   kube-scheduler                             4.12.2   True        False         False      23m
   kube-storage-version-migrator              4.12.2   True        False         False      4m52s
   monitoring                                 4.12.2   True        False         False      69s
   network                                    4.12.2   True        False         False      4m3s
   node-tuning                                4.12.2   True        False         False      2m22s
   openshift-apiserver                        4.12.2   True        False         False      23m
   openshift-controller-manager               4.12.2   True        False         False      23m
   openshift-samples                          4.12.2   True        False         False      2m15s
   operator-lifecycle-manager                 4.12.2   True        False         False      22m
   operator-lifecycle-manager-catalog         4.12.2   True        False         False      23m
   operator-lifecycle-manager-packageserver   4.12.2   True        False         False      23m
   service-ca                                 4.12.2   True        False         False      4m41s
   storage                                    4.12.2   True        False         False      4m43s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1

   1

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Create a hosted cluster by entering the following command:

   $ hcp create cluster agent \
     --name=<hosted_cluster_name> \1
     --pull-secret=<path_to_pull_secret> \2
     --agent-namespace=<hosted_control_plane_namespace> \3
     --base-domain=<basedomain> \4
     --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
     --etcd-storage-class=<etcd_storage_class> \6
     --ssh-key  <path_to_ssh_key> \7
     --namespace <hosted_cluster_namespace> \8
     --control-plane-availability-policy HighlyAvailable \9
     --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Verify that you have a default storage class configured for your cluster. Otherwise, you might end up with pending PVCs. Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

Procedure

1. Obtain the kubeconfig file for your new hosted cluster by entering the following command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

2. Use the kubeconfig file to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   csi-snapshot-controller                    4.10.26   True        False         False      4m3s
   dns                                        4.10.26   True        False         False      2m52s

3. View the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-samples-operator                 cluster-samples-operator-6b5bcb9dff-kpnbc                 2/2     Running            0               20m
   openshift-monitoring                               alertmanager-main-0                                       6/6     Running            0               100s
   openshift-monitoring                               openshift-state-metrics-677b9fb74f-qqp6g                  3/3     Running            0               104s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Create a YAML file to contain the configuration. See the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name>
     namespace: <hosted_control_plane_namespace>
   spec:
     cpuArchitecture: s390x
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <ssh_public_key>

2. Save the file as infraenv-config.yaml.

3. Apply the configuration by entering the following command:

   $ oc apply -f infraenv-config.yaml

4. To fetch the URL to download the PXE images, such as, initrd.img, kernel.img, or rootfs.img, which allows IBM Z machines to join as agents, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

Procedure

1. Run the following command to scale the NodePool object to two nodes:

   $ oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Run the following command to see the status of a specific scaled agent:

   $ oc -n <hosted_control_plane_namespace> get agent -o \
     jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} \
     Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
   BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

3. Run the following command to see the transition phases:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER           APPROVED       ROLE        STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true          auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a                      true          auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true          auto-assign

4. Run the following command to generate the kubeconfig file to access the hosted cluster:

   $ hcp create kubeconfig --namespace <clusters_namespace> --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

   Example output

   NAME                             STATUS   ROLES    AGE      VERSION
   worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
   worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

   Example output

   NAME                                CLUSTER  NODENAME PROVIDERID     PHASE     AGE   VERSION
   hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
   hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

7. Run the following command to check the cluster version:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

   Example output

   NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version   4.15.0-ec.2   True        False         40h     Cluster version is 4.15.0-ec.2

8. Run the following command to check the cluster operator status:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER         APPROVED   ROLE          STAGE
   4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
   d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

3. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
   BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
   BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

   Example output

   NAME           STATUS   ROLES    AGE     VERSION
   ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
   ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machines

   Example output

   NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
   hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
   hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

   The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

7. Enter the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

   Example output

   NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
   
   NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
   clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
   clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb
     labels:
       openshift.io/cluster-monitoring: "true"
     annotations:
       workload.openshift.io/allowed: management
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator-operatorgroup
     namespace: metallb
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: metallb
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

2. Save the file as metallb-operator-config.yaml.

3. Enter the following command to apply the configuration:

   $ oc apply -f metallb-operator-config.yaml

4. After the Operator is running, create the MetalLB instance:

   1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb

   2. Save the file as metallb-instance-config.yaml.

   3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml

5. Configure the MetalLB Operator by creating two resources:

   • An IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

   • A BGPAdvertisement resource to advertise the load balancer IP addresses that the IPAddressPool resource provides through the BGP protocol.

     1. Create a YAML file to contain the configuration:

        apiVersion: metallb.io/v1beta1
        kind: IPAddressPool
        metadata:
          name: <ip_address_pool_name> 1
          namespace: metallb
        spec:
          protocol: layer2
          autoAssign: false
          addresses:
            - <ingress_ip>-<ingress_ip> 2
        ---
        apiVersion: metallb.io/v1beta1
        kind: BGPAdvertisement
        metadata:
          name: <bgp_advertisement_name> 3
          namespace: metallb
        spec:
          ipAddressPools:
            - <ip_address_pool_name> 4

        1 4

        Specify the IPAddressPool resource name.

        2

        Specify the IP address for your environment, for example, 192.168.122.23.

        3

        Specify the BGPAdvertisement resource name.

     2. Save the file as ipaddresspool-bgpadvertisement-config.yaml.

     3. Create the resources by entering the following command:

        $ oc apply -f ipaddresspool-bgpadvertisement-config.yaml

6. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

   1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer

   2. Save the metallb-loadbalancer-service.yaml file.

   3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml

   4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

   5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

1. Verify that the spec.nodeDrainTimeout value in your NodePool resource is greater than 0s. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace and <nodepool_name> with the node pool name. Run the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep nodeDrainTimeout

   Example output

   nodeDrainTimeout: 30s

2. If the spec.nodeDrainTimeout value is not greater than 0s, modify the value by running the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec":{"nodeDrainTimeout": "30m"}}' --type=merge

3. Enable machine health checks by setting the spec.management.autoRepair field to true in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":true}}}' --type=merge

4. Verify that the NodePool resource is updated with the autoRepair: true value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Disable machine health checks by setting the spec.management.autoRepair field to false in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":false}}}' --type=merge

2. Verify that the NodePool resource is updated with the autoRepair: false value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair