Chapter 4. Deploying hosted control planes

Procedure

1. Create an S3 bucket that has public access to host OIDC discovery documents for your clusters:

   1. To create the bucket in the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

   2. To create the bucket in a region other than the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name> \
        --create-bucket-configuration LocationConstraint=<region> \
        --region <region>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

2. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift operator.
3. Save the secret in the local-cluster namespace.

4. See the following table to verify that the secret contains the following fields:

   Table 4.1. Required fields for the AWS secret

   Field name	Description
   bucket	Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters.
   credentials	A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket.
   region	Specifies the region of the S3 bucket.

5. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> --from-file=credentials=<path>/.aws/credentials --from-literal=bucket=<s3_bucket> --from-literal=region=<region> -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. Run the following command to add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery:

   $ oc label secret hypershift-operator-oidc-provider-s3-credentials -n local-cluster cluster.open-cluster-management.io/backup=true

Procedure

1. Get the Amazon Resource Name (ARN) of your user by running the following command:

   $ aws sts get-caller-identity --query "Arn" --output text

   Example output

   arn:aws:iam::1234567890:user/<aws_username>

   Use this output as the value for <arn> in the next step.

2. Create a JSON file named trust-relationship.json that contains the trust relationship configuration for your role. See the following example:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": <arn> 1
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

   1

   Replace <arn> with the ARN of your user that you noted in the previous step.

3. Create the Identity and Access Management (IAM) role by running the following command:

   $ aws iam create-role \
   --role-name <name> \1
   --assume-role-policy-document file://<file_name>.json \2
   --query "Role.Arn"

   1

   Replace <name> with the role name, for example, hcp-cli-role.

   2

   Replace <file_name> with the file name, for example, assume-role-policy.json.

   Example output

   arn:aws:iam::820196288204:role/myrole

4. Create a JSON file named policy.json that contains the following permission policies for your role:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "EC2",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateDhcpOptions",
                   "ec2:DeleteSubnet",
                   "ec2:ReplaceRouteTableAssociation",
                   "ec2:DescribeAddresses",
                   "ec2:DescribeInstances",
                   "ec2:DeleteVpcEndpoints",
                   "ec2:CreateNatGateway",
                   "ec2:CreateVpc",
                   "ec2:DescribeDhcpOptions",
                   "ec2:AttachInternetGateway",
                   "ec2:DeleteVpcEndpointServiceConfigurations",
                   "ec2:DeleteRouteTable",
                   "ec2:AssociateRouteTable",
                   "ec2:DescribeInternetGateways",
                   "ec2:DescribeAvailabilityZones",
                   "ec2:CreateRoute",
                   "ec2:CreateInternetGateway",
                   "ec2:RevokeSecurityGroupEgress",
                   "ec2:ModifyVpcAttribute",
                   "ec2:DeleteInternetGateway",
                   "ec2:DescribeVpcEndpointConnections",
                   "ec2:RejectVpcEndpointConnections",
                   "ec2:DescribeRouteTables",
                   "ec2:ReleaseAddress",
                   "ec2:AssociateDhcpOptions",
                   "ec2:TerminateInstances",
                   "ec2:CreateTags",
                   "ec2:DeleteRoute",
                   "ec2:CreateRouteTable",
                   "ec2:DetachInternetGateway",
                   "ec2:DescribeVpcEndpointServiceConfigurations",
                   "ec2:DescribeNatGateways",
                   "ec2:DisassociateRouteTable",
                   "ec2:AllocateAddress",
                   "ec2:DescribeSecurityGroups",
                   "ec2:RevokeSecurityGroupIngress",
                   "ec2:CreateVpcEndpoint",
                   "ec2:DescribeVpcs",
                   "ec2:DeleteSecurityGroup",
                   "ec2:DeleteDhcpOptions",
                   "ec2:DeleteNatGateway",
                   "ec2:DescribeVpcEndpoints",
                   "ec2:DeleteVpc",
                   "ec2:CreateSubnet",
                   "ec2:DescribeSubnets"
               ],
               "Resource": "*"
           },
           {
               "Sid": "ELB",
               "Effect": "Allow",
               "Action": [
                   "elasticloadbalancing:DeleteLoadBalancer",
                   "elasticloadbalancing:DescribeLoadBalancers",
                   "elasticloadbalancing:DescribeTargetGroups",
                   "elasticloadbalancing:DeleteTargetGroup"
               ],
               "Resource": "*"
           },
           {
               "Sid": "IAMPassRole",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:*:iam::*:role/*-worker-role",
               "Condition": {
                   "ForAnyValue:StringEqualsIfExists": {
                       "iam:PassedToService": "ec2.amazonaws.com"
                   }
               }
           },
           {
               "Sid": "IAM",
               "Effect": "Allow",
               "Action": [
                   "iam:CreateInstanceProfile",
                   "iam:DeleteInstanceProfile",
                   "iam:GetRole",
                   "iam:UpdateAssumeRolePolicy",
                   "iam:GetInstanceProfile",
                   "iam:TagRole",
                   "iam:RemoveRoleFromInstanceProfile",
                   "iam:CreateRole",
                   "iam:DeleteRole",
                   "iam:PutRolePolicy",
                   "iam:AddRoleToInstanceProfile",
                   "iam:CreateOpenIDConnectProvider",
                   "iam:ListOpenIDConnectProviders",
                   "iam:DeleteRolePolicy",
                   "iam:UpdateRole",
                   "iam:DeleteOpenIDConnectProvider",
                   "iam:GetRolePolicy"
               ],
               "Resource": "*"
           },
           {
               "Sid": "Route53",
               "Effect": "Allow",
               "Action": [
                   "route53:ListHostedZonesByVPC",
                   "route53:CreateHostedZone",
                   "route53:ListHostedZones",
                   "route53:ChangeResourceRecordSets",
                   "route53:ListResourceRecordSets",
                   "route53:DeleteHostedZone",
                   "route53:AssociateVPCWithHostedZone",
                   "route53:ListHostedZonesByName"
               ],
               "Resource": "*"
           },
           {
               "Sid": "S3",
               "Effect": "Allow",
               "Action": [
                   "s3:ListAllMyBuckets",
                   "s3:ListBucket",
                   "s3:DeleteObject",
                   "s3:DeleteBucket"
               ],
               "Resource": "*"
           }
       ]
   }

5. Attach the policy.json file to your role by running the following command:

   $ aws iam put-role-policy \
     --role-name <role_name> \1
     --policy-name <policy_name> \2
     --policy-document file://policy.json 3

   1

   Replace <role_name> with the name of your role.

   2

   Replace <policy_name> with your policy name.

   3

   The policy.json file contains the permission policies for your role.

6. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command:

   $ aws sts get-session-token --output json > sts-creds.json

   Example sts-creds.json file

   {
                 "Credentials": {
                     "AccessKeyId": "ASIA1443CE0GN2ATHWJU",
                     "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1”,
                     "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=",
                     "Expiration": "2025-05-16T04:19:32+00:00"
                 }
             }

Procedure

1. Create an AWS credential secret for the HyperShift Operator and name it hypershift-operator-private-link-credentials. The secret must reside in the managed cluster namespace that is the namespace of the managed cluster being used as the management cluster. If you used local-cluster, create the secret in the local-cluster namespace.
2. See the following table to confirm that the secret contains the required fields:

Procedure

1. To create a hosted cluster on AWS, run the following command:

   $ hcp create cluster aws \
       --name <hosted_cluster_name> \1
       --infra-id <infra_id> \2
       --base-domain <basedomain> \3
       --sts-creds <path_to_sts_credential_file> \4
       --pull-secret <path_to_pull_secret> \5
       --region <region> \6
       --generate-ssh \
       --node-pool-replicas <node_pool_replica_count> \7
       --namespace <hosted_cluster_namespace> \8
       --role-arn <role_name> \9
       --render-into <file_name>.yaml 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify your infrastructure name. You must provide the same value for <hosted_cluster_name> and <infra_id>. Otherwise the cluster might not appear correctly in the multicluster engine for Kubernetes Operator console.

   3

   Specify your base domain, for example, example.com.

   4

   Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json.

   5

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   6

   Specify the AWS region name, for example, us-east-1.

   7

   Specify the node pool replica count, for example, 3.

   8

   By default, all HostedCluster and NodePool custom resources are created in the clusters namespace. You can use the --namespace <namespace> parameter, to create the HostedCluster and NodePool custom resources in a specific namespace.

   9

   Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole.

   10

   If you want to indicate whether the EC2 instance runs on shared or single tenant hardware, include this field. The --render-into flag renders Kubernetes resources into the YAML file that you specify in this field. Then, continue to the next step to edit the YAML file.

2. If you included the --render-into flag in the previous command, edit the specified YAML file. Edit the NodePool specification in the YAML file to indicate whether the EC2 instance should run on shared or single-tenant hardware, similar to the following example:

   Example YAML file

   apiVersion: hypershift.openshift.io/v1beta1
   kind: NodePool
   metadata:
     name: <nodepool_name> 1
   spec:
     platform:
       aws:
         placement:
           tenancy: "default" 2

   1

   Specify the name of the NodePool resource.

   2

   Specify a valid value for tenancy: "default", "dedicated", or "host". Use "default" when node pool instances run on shared hardware. Use "dedicated" when each node pool instance runs on single-tenant hardware. Use "host" when node pool instances run on your pre-allocated dedicated hosts.

Verification

1. Verify the status of your hosted cluster to check that the value of AVAILABLE is True. Run the following command:

   $ oc get hostedclusters -n <hosted_cluster_namespace>

2. Get a list of your node pools by running the following command:

   $ oc get nodepools --namespace <hosted_cluster_namespace>

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig --to=- > kubeconfig-<hosted-cluster-name>

2. Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   dns                                        4.10.26   True        False         False      2m52s
   image-registry                             4.10.26   True        False         False      2m8s
   ingress                                    4.10.26   True        False         False      22m

3. You can also view the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-node-tuning-operator             tuned-dhw5p                                               1/1     Running            0               109s
   openshift-cluster-storage-operator                 cluster-storage-operator-5f784969f5-vwzgz                 1/1     Running            1 (113s ago)    20m
   openshift-cluster-storage-operator                 csi-snapshot-controller-6b7687b7d9-7nrfw                  1/1     Running            0               3m8s
   openshift-console                                  console-5cbf6c7969-6gk6z                                  1/1     Running            0               119s
   openshift-console                                  downloads-7bcd756565-6wj5j                                1/1     Running            0               4m3s
   openshift-dns-operator                             dns-operator-77d755cd8c-xjfbn                             2/2     Running            0               21m
   openshift-dns                                      dns-default-kfqnh                                         2/2     Running            0               113s

1. Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system

2. Apply the YAML content by entering the following command:

   $ oc apply -f configure-metallb.yaml

   Example output

   metallb.metallb.io/metallb created

3. Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: metallb
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.216.32-192.168.216.122 1

   1

   Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network.

4. Apply the YAML content by entering the following command:

   $ oc apply -f create-ip-address-pool.yaml

   Example output

   ipaddresspool.metallb.io/metallb created

5. Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: l2advertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
      - metallb

6. Apply the YAML content by entering the following command:

   $ oc apply -f l2advertisement.yaml

   Example output

   l2advertisement.metallb.io/metallb created

Procedure

1. Run the following command:

   NODEPOOL_NAME=${CLUSTER_NAME}-work
   NODEPOOL_REPLICAS=5
   
   $ oc scale nodepool/$NODEPOOL_NAME --namespace clusters --replicas=$NODEPOOL_REPLICAS

2. After a few moments, enter the following command to see the status of the node pool:

   $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

   Example output

   NAME                  STATUS   ROLES    AGE     VERSION
   example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
   example-n6prw         Ready    worker   116m    v1.27.4+18eadca
   example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
   example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
   example-twxns         Ready    worker   88s     v1.27.4+18eadca

Procedure

1. Verify that the HostedCluster resource transitioned to the completed state by entering the following command:

   $ oc get --namespace clusters hostedclusters <hosted_cluster_name>

   Example output

   NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   clusters    example   4.12.2    example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands:

   $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig

   $ oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.12.2   True        False         False      2m38s
   csi-snapshot-controller                    4.12.2   True        False         False      4m3s
   dns                                        4.12.2   True        False         False      2m52s
   image-registry                             4.12.2   True        False         False      2m8s
   ingress                                    4.12.2   True        False         False      22m
   kube-apiserver                             4.12.2   True        False         False      23m
   kube-controller-manager                    4.12.2   True        False         False      23m
   kube-scheduler                             4.12.2   True        False         False      23m
   kube-storage-version-migrator              4.12.2   True        False         False      4m52s
   monitoring                                 4.12.2   True        False         False      69s
   network                                    4.12.2   True        False         False      4m3s
   node-tuning                                4.12.2   True        False         False      2m22s
   openshift-apiserver                        4.12.2   True        False         False      23m
   openshift-controller-manager               4.12.2   True        False         False      23m
   openshift-samples                          4.12.2   True        False         False      2m15s
   operator-lifecycle-manager                 4.12.2   True        False         False      22m
   operator-lifecycle-manager-catalog         4.12.2   True        False         False      23m
   operator-lifecycle-manager-packageserver   4.12.2   True        False         False      23m
   service-ca                                 4.12.2   True        False         False      4m41s
   storage                                    4.12.2   True        False         False      4m43s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1

   1

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Create a hosted cluster by entering the following command:

   $ hcp create cluster agent \
     --name=<hosted_cluster_name> \1
     --pull-secret=<path_to_pull_secret> \2
     --agent-namespace=<hosted_control_plane_namespace> \3
     --base-domain=<basedomain> \4
     --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
     --etcd-storage-class=<etcd_storage_class> \6
     --ssh-key  <path_to_ssh_key> \7
     --namespace <hosted_cluster_namespace> \8
     --control-plane-availability-policy HighlyAvailable \9
     --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Verify that you have a default storage class configured for your cluster. Otherwise, you might end up with pending PVCs. Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

Procedure

1. Obtain the kubeconfig file for your new hosted cluster by entering the following command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

2. Use the kubeconfig file to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   csi-snapshot-controller                    4.10.26   True        False         False      4m3s
   dns                                        4.10.26   True        False         False      2m52s

3. View the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-samples-operator                 cluster-samples-operator-6b5bcb9dff-kpnbc                 2/2     Running            0               20m
   openshift-monitoring                               alertmanager-main-0                                       6/6     Running            0               100s
   openshift-monitoring                               openshift-state-metrics-677b9fb74f-qqp6g                  3/3     Running            0               104s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Create a YAML file to contain the configuration. See the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name>
     namespace: <hosted_control_plane_namespace>
   spec:
     cpuArchitecture: s390x
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <ssh_public_key>

2. Save the file as infraenv-config.yaml.

3. Apply the configuration by entering the following command:

   $ oc apply -f infraenv-config.yaml

4. To fetch the URL to download the PXE images, such as, initrd.img, kernel.img, or rootfs.img, which allows IBM Z machines to join as agents, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

Procedure

1. Run the following command to scale the NodePool object to two nodes:

   $ oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Run the following command to see the status of a specific scaled agent:

   $ oc -n <hosted_control_plane_namespace> get agent -o \
     jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} \
     Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
   BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

3. Run the following command to see the transition phases:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER           APPROVED       ROLE        STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true          auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a                      true          auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true          auto-assign

4. Run the following command to generate the kubeconfig file to access the hosted cluster:

   $ hcp create kubeconfig --namespace <clusters_namespace> --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

   Example output

   NAME                             STATUS   ROLES    AGE      VERSION
   worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
   worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

   Example output

   NAME                                CLUSTER  NODENAME PROVIDERID     PHASE     AGE   VERSION
   hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
   hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

7. Run the following command to check the cluster version:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

   Example output

   NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version   4.15.0-ec.2   True        False         40h     Cluster version is 4.15.0-ec.2

8. Run the following command to check the cluster operator status:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy HighlyAvailable \9
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 10

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   The default value for the control plane availability policy is HighlyAvailable.

   10

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s