Hosted control planes

Procedure

1. Create the ConfigMap resource by running the following command:

   $ oc create -f <your-config-map-file.yaml>

   Replace <your-config-map-file.yaml> with the name of your YAML file that contains your hcp-sizing-baseline config map.

2. Create the hcp-sizing-baseline config map in the local-cluster namespace to specify the measurements you want to override. Your config map might resemble the following YAML file:

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: hcp-sizing-baseline
     namespace: local-cluster
   data:
     incrementalCPUUsagePer1KQPS: "9.0"
     memoryRequestPerHCP: "18"
     minimumQPSPerHCP: "50.0"

3. Delete the hypershift-addon-agent deployment to restart the hypershift-addon-agent pod by running the following command:

   $ oc delete deployment hypershift-addon-agent -n open-cluster-management-agent-addon

1. Log in to your hub cluster by running the following command:

   $ oc login

2. Edit the hypershift-addon-deploy-config add-on deployment configuration specification by running the following command:

   $ oc edit addondeploymentconfig hypershift-addon-deploy-config -n multicluster-engine

3. Add the disableMetrics=true customized variable to the specification, as shown in the following example:

   apiVersion: addon.open-cluster-management.io/v1alpha1
   kind: AddOnDeploymentConfig
   metadata:
     name: hypershift-addon-deploy-config
     namespace: multicluster-engine
   spec:
     customizedVariables:
     - name: hcMaxNumber
       value: "80"
     - name: hcThresholdNumber
       value: "60"
     - name: disableMetrics 1
       value: "true"

   1

   The disableMetrics=true customized variable disables metric service monitoring for both new and existing hypershift-addon managed cluster add-ons.

4. Apply the changes to the configuration specification by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Get the URL to download the hcp binary by running the following command:

   $ oc get ConsoleCLIDownload hcp-cli-download -o json | jq -r ".spec"

2. Download the hcp binary by running the following command:

   $ wget <hcp_cli_download_url> 1

   1

   Replace hcp_cli_download_url with the URL that you obtained from the previous step.

3. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

4. Make the hcp binary file executable by running the following command:

   $ chmod +x hcp

5. Move the hcp binary file to a directory in your path by running the following command:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. From the OpenShift Container Platform web console, click the Help icon → Command Line Tools.
2. Click Download hcp CLI for your platform.

3. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

4. Run the following command to make the binary file executable:

   $ chmod +x hcp

5. Run the following command to move the binary file to a directory in your path:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. Navigate to the content gateway and download the hcp binary.

2. Unpack the downloaded archive by running the following command:

   $ tar xvzf hcp.tar.gz

3. Make the hcp binary file executable by running the following command:

   $ chmod +x hcp

4. Move the hcp binary file to a directory in your path by running the following command:

   $ sudo mv hcp /usr/local/bin/.

Procedure

1. Run the following command to enable the feature:

   $ oc patch mce multiclusterengine --type=merge -p '{"spec":{"overrides":{"components":[{"name":"hypershift","enabled": true}]}}}' 1

   1

   The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

2. Run the following command to verify that the hypershift and hypershift-local-hosting features are enabled in the MultiClusterEngine custom resource:

   $ oc get mce multiclusterengine -o yaml 1

   1

   The default MultiClusterEngine resource instance name is multiclusterengine, but you can get the MultiClusterEngine name from your cluster by running the following command: $ oc get mce.

   Example output

   apiVersion: multicluster.openshift.io/v1
   kind: MultiClusterEngine
   metadata:
     name: multiclusterengine
   spec:
     overrides:
       components:
       - name: hypershift
         enabled: true
       - name: hypershift-local-hosting
         enabled: true

Procedure

1. Create an S3 bucket that has public access to host OIDC discovery documents for your clusters:

   1. To create the bucket in the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

   2. To create the bucket in a region other than the us-east-1 region, enter the following code:

      aws s3api create-bucket --bucket <bucket_name> \
        --create-bucket-configuration LocationConstraint=<region> \
        --region <region>
      aws s3api delete-public-access-block --bucket <bucket_name>
      echo '{
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": "*",
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket_name>/*"
              }
          ]
      }' | envsubst > policy.json
      aws s3api put-bucket-policy --bucket <bucket_name> --policy file://policy.json

2. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift operator.
3. Save the secret in the local-cluster namespace.

4. See the following table to verify that the secret contains the following fields:

   Table 3.1. Required fields for the AWS secret

   Field name	Description
   bucket	Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters.
   credentials	A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket.
   region	Specifies the region of the S3 bucket.

5. To create an AWS secret, run the following command:

   $ oc create secret generic <secret_name> --from-file=credentials=<path>/.aws/credentials --from-literal=bucket=<s3_bucket> --from-literal=region=<region> -n local-cluster

   Note

   Disaster recovery backup for the secret is not automatically enabled. Run the following command to add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery:

   $ oc label secret hypershift-operator-oidc-provider-s3-credentials -n local-cluster cluster.open-cluster-management.io/backup=true

Procedure

1. Get the Amazon Resource Name (ARN) of your user by running the following command:

   $ aws sts get-caller-identity --query "Arn"

   Note the ARN of your user to use in the next step.

2. Create a JSON file named trust-relationship.json that contains the trust relationship configuration for your role. See the following example:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": <arn> 1
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

   1

   Replace <arn> with the ARN of your user that you noted in the previous step.

3. Create the Identity and Access Management (IAM) role by running the following command:

   $ aws iam create-role \
   --role-name <name> \1
   --assume-role-policy-document file://<file_name>.json \2
   --query "Role.Arn"

   1

   Replace <name> with the role name, for example, hcp-cli-role.

   2

   Replace <file_name> with the file name, for example, assume-role-policy.json.

   Example output

   arn:aws:iam::820196288204:role/myrole

4. Create a JSON file named policy.json that contains the following permission policies for your role:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "EC2",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateDhcpOptions",
                   "ec2:DeleteSubnet",
                   "ec2:ReplaceRouteTableAssociation",
                   "ec2:DescribeAddresses",
                   "ec2:DescribeInstances",
                   "ec2:DeleteVpcEndpoints",
                   "ec2:CreateNatGateway",
                   "ec2:CreateVpc",
                   "ec2:DescribeDhcpOptions",
                   "ec2:AttachInternetGateway",
                   "ec2:DeleteVpcEndpointServiceConfigurations",
                   "ec2:DeleteRouteTable",
                   "ec2:AssociateRouteTable",
                   "ec2:DescribeInternetGateways",
                   "ec2:DescribeAvailabilityZones",
                   "ec2:CreateRoute",
                   "ec2:CreateInternetGateway",
                   "ec2:RevokeSecurityGroupEgress",
                   "ec2:ModifyVpcAttribute",
                   "ec2:DeleteInternetGateway",
                   "ec2:DescribeVpcEndpointConnections",
                   "ec2:RejectVpcEndpointConnections",
                   "ec2:DescribeRouteTables",
                   "ec2:ReleaseAddress",
                   "ec2:AssociateDhcpOptions",
                   "ec2:TerminateInstances",
                   "ec2:CreateTags",
                   "ec2:DeleteRoute",
                   "ec2:CreateRouteTable",
                   "ec2:DetachInternetGateway",
                   "ec2:DescribeVpcEndpointServiceConfigurations",
                   "ec2:DescribeNatGateways",
                   "ec2:DisassociateRouteTable",
                   "ec2:AllocateAddress",
                   "ec2:DescribeSecurityGroups",
                   "ec2:RevokeSecurityGroupIngress",
                   "ec2:CreateVpcEndpoint",
                   "ec2:DescribeVpcs",
                   "ec2:DeleteSecurityGroup",
                   "ec2:DeleteDhcpOptions",
                   "ec2:DeleteNatGateway",
                   "ec2:DescribeVpcEndpoints",
                   "ec2:DeleteVpc",
                   "ec2:CreateSubnet",
                   "ec2:DescribeSubnets"
               ],
               "Resource": "*"
           },
           {
               "Sid": "ELB",
               "Effect": "Allow",
               "Action": [
                   "elasticloadbalancing:DeleteLoadBalancer",
                   "elasticloadbalancing:DescribeLoadBalancers",
                   "elasticloadbalancing:DescribeTargetGroups",
                   "elasticloadbalancing:DeleteTargetGroup"
               ],
               "Resource": "*"
           },
           {
               "Sid": "IAMPassRole",
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:*:iam::*:role/*-worker-role",
               "Condition": {
                   "ForAnyValue:StringEqualsIfExists": {
                       "iam:PassedToService": "ec2.amazonaws.com"
                   }
               }
           },
           {
               "Sid": "IAM",
               "Effect": "Allow",
               "Action": [
                   "iam:CreateInstanceProfile",
                   "iam:DeleteInstanceProfile",
                   "iam:GetRole",
                   "iam:UpdateAssumeRolePolicy",
                   "iam:GetInstanceProfile",
                   "iam:TagRole",
                   "iam:RemoveRoleFromInstanceProfile",
                   "iam:CreateRole",
                   "iam:DeleteRole",
                   "iam:PutRolePolicy",
                   "iam:AddRoleToInstanceProfile",
                   "iam:CreateOpenIDConnectProvider",
                   "iam:ListOpenIDConnectProviders",
                   "iam:DeleteRolePolicy",
                   "iam:UpdateRole",
                   "iam:DeleteOpenIDConnectProvider",
                   "iam:GetRolePolicy"
               ],
               "Resource": "*"
           },
           {
               "Sid": "Route53",
               "Effect": "Allow",
               "Action": [
                   "route53:ListHostedZonesByVPC",
                   "route53:CreateHostedZone",
                   "route53:ListHostedZones",
                   "route53:ChangeResourceRecordSets",
                   "route53:ListResourceRecordSets",
                   "route53:DeleteHostedZone",
                   "route53:AssociateVPCWithHostedZone",
                   "route53:ListHostedZonesByName"
               ],
               "Resource": "*"
           },
           {
               "Sid": "S3",
               "Effect": "Allow",
               "Action": [
                   "s3:ListAllMyBuckets",
                   "s3:ListBucket",
                   "s3:DeleteObject",
                   "s3:DeleteBucket"
               ],
               "Resource": "*"
           }
       ]
   }

5. Attach the policy.json file to your role by running the following command:

   $ aws iam put-role-policy \
     --role-name <role_name> \1
     --policy-name <policy_name> \2
     --policy-document file://policy.json 3

   1

   Replace <role_name> with the name of your role.

   2

   Replace <policy_name> with your policy name.

   3

   The policy.json file contains the permission policies for your role.

6. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command:

   $ aws sts get-session-token --output json > sts-creds.json

   Example sts-creds.json file

   {
                 "Credentials": {
                     "AccessKeyId": "ASIA1443CE0GN2ATHWJU",
                     "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1”,
                     "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=",
                     "Expiration": "2025-05-16T04:19:32+00:00"
                 }
             }

Procedure

1. Create an AWS credential secret for the HyperShift Operator and name it hypershift-operator-private-link-credentials. The secret must reside in the managed cluster namespace that is the namespace of the managed cluster being used as the management cluster. If you used local-cluster, create the secret in the local-cluster namespace.
2. See the following table to confirm that the secret contains the required fields:

Verification

1. Verify the status of your hosted cluster to check that the value of AVAILABLE is True. Run the following command:

   $ oc get hostedclusters -n <hosted_cluster_namespace>

2. Get a list of your node pools by running the following command:

   $ oc get nodepools --namespace <hosted_cluster_namespace>

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy SingleReplica \
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig --to=- > kubeconfig-<hosted-cluster-name>

2. Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   dns                                        4.10.26   True        False         False      2m52s
   image-registry                             4.10.26   True        False         False      2m8s
   ingress                                    4.10.26   True        False         False      22m

3. You can also view the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-node-tuning-operator             tuned-dhw5p                                               1/1     Running            0               109s
   openshift-cluster-storage-operator                 cluster-storage-operator-5f784969f5-vwzgz                 1/1     Running            1 (113s ago)    20m
   openshift-cluster-storage-operator                 csi-snapshot-controller-6b7687b7d9-7nrfw                  1/1     Running            0               3m8s
   openshift-console                                  console-5cbf6c7969-6gk6z                                  1/1     Running            0               119s
   openshift-console                                  downloads-7bcd756565-6wj5j                                1/1     Running            0               4m3s
   openshift-dns-operator                             dns-operator-77d755cd8c-xjfbn                             2/2     Running            0               21m
   openshift-dns                                      dns-default-kfqnh                                         2/2     Running            0               113s

1. Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: MetalLB
   metadata:
     name: metallb
     namespace: metallb-system

2. Apply the YAML content by entering the following command:

   $ oc apply -f configure-metallb.yaml

   Example output

   metallb.metallb.io/metallb created

3. Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool
   metadata:
     name: metallb
     namespace: metallb-system
   spec:
     addresses:
     - 192.168.216.32-192.168.216.122 1

   1

   Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network.

4. Apply the YAML content by entering the following command:

   $ oc apply -f create-ip-address-pool.yaml

   Example output

   ipaddresspool.metallb.io/metallb created

5. Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file:

   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement
   metadata:
     name: l2advertisement
     namespace: metallb-system
   spec:
     ipAddressPools:
      - metallb

6. Apply the YAML content by entering the following command:

   $ oc apply -f l2advertisement.yaml

   Example output

   l2advertisement.metallb.io/metallb created

Procedure

1. Run the following command:

   NODEPOOL_NAME=${CLUSTER_NAME}-work
   NODEPOOL_REPLICAS=5
   
   $ oc scale nodepool/$NODEPOOL_NAME --namespace clusters --replicas=$NODEPOOL_REPLICAS

2. After a few moments, enter the following command to see the status of the node pool:

   $ oc --kubeconfig $CLUSTER_NAME-kubeconfig get nodes

   Example output

   NAME                  STATUS   ROLES    AGE     VERSION
   example-9jvnf         Ready    worker   97s     v1.27.4+18eadca
   example-n6prw         Ready    worker   116m    v1.27.4+18eadca
   example-nc6g4         Ready    worker   117m    v1.27.4+18eadca
   example-thp29         Ready    worker   4m17s   v1.27.4+18eadca
   example-twxns         Ready    worker   88s     v1.27.4+18eadca

Procedure

1. Verify that the HostedCluster resource transitioned to the completed state by entering the following command:

   $ oc get --namespace clusters hostedclusters <hosted_cluster_name>

   Example output

   NAMESPACE   NAME      VERSION   KUBECONFIG                 PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
   clusters    example   4.12.2    example-admin-kubeconfig   Completed   True        False         The hosted control plane is available

2. Verify that all the cluster operators in the hosted cluster are online by entering the following commands:

   $ hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig

   $ oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.12.2   True        False         False      2m38s
   csi-snapshot-controller                    4.12.2   True        False         False      4m3s
   dns                                        4.12.2   True        False         False      2m52s
   image-registry                             4.12.2   True        False         False      2m8s
   ingress                                    4.12.2   True        False         False      22m
   kube-apiserver                             4.12.2   True        False         False      23m
   kube-controller-manager                    4.12.2   True        False         False      23m
   kube-scheduler                             4.12.2   True        False         False      23m
   kube-storage-version-migrator              4.12.2   True        False         False      4m52s
   monitoring                                 4.12.2   True        False         False      69s
   network                                    4.12.2   True        False         False      4m3s
   node-tuning                                4.12.2   True        False         False      2m22s
   openshift-apiserver                        4.12.2   True        False         False      23m
   openshift-controller-manager               4.12.2   True        False         False      23m
   openshift-samples                          4.12.2   True        False         False      2m15s
   operator-lifecycle-manager                 4.12.2   True        False         False      22m
   operator-lifecycle-manager-catalog         4.12.2   True        False         False      23m
   operator-lifecycle-manager-packageserver   4.12.2   True        False         False      23m
   service-ca                                 4.12.2   True        False         False      4m41s
   storage                                    4.12.2   True        False         False      4m43s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1

   1

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Create a hosted cluster by entering the following command:

   $ hcp create cluster agent \
     --name=<hosted_cluster_name> \1
     --pull-secret=<path_to_pull_secret> \2
     --agent-namespace=<hosted_control_plane_namespace> \3
     --base-domain=<basedomain> \4
     --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
     --etcd-storage-class=<etcd_storage_class> \6
     --ssh-key  <path_to_ssh_key> \7
     --namespace <hosted_cluster_namespace> \8
     --control-plane-availability-policy SingleReplica \
     --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Verify that you have a default storage class configured for your cluster. Otherwise, you might end up with pending PVCs. Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported {ocp-short} version that you want to use, for example, 4.14.0-x86_64.

Procedure

1. Obtain the kubeconfig file for your new hosted cluster by entering the following command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

2. Use the kubeconfig file to view the cluster Operators of the hosted cluster. Enter the following command:

   $ oc get co --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   console                                    4.10.26   True        False         False      2m38s
   csi-snapshot-controller                    4.10.26   True        False         False      4m3s
   dns                                        4.10.26   True        False         False      2m52s

3. View the running pods on your hosted cluster by entering the following command:

   $ oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name>

   Example output

   NAMESPACE                                          NAME                                                      READY   STATUS             RESTARTS        AGE
   kube-system                                        konnectivity-agent-khlqv                                  0/1     Running            0               3m52s
   openshift-cluster-samples-operator                 cluster-samples-operator-6b5bcb9dff-kpnbc                 2/2     Running            0               20m
   openshift-monitoring                               alertmanager-main-0                                       6/6     Running            0               100s
   openshift-monitoring                               openshift-state-metrics-677b9fb74f-qqp6g                  3/3     Running            0               104s

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy SingleReplica \
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Create a YAML file to contain the configuration. See the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name>
     namespace: <hosted_control_plane_namespace>
   spec:
     cpuArchitecture: s390x
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <ssh_public_key>

2. Save the file as infraenv-config.yaml.

3. Apply the configuration by entering the following command:

   $ oc apply -f infraenv-config.yaml

4. To fetch the URL to download the PXE images, such as, initrd.img, kernel.img, or rootfs.img, which allows IBM Z machines to join as agents, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

Procedure

1. Run the following command to scale the NodePool object to two nodes:

   $ oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as {ocp-short} nodes. The agents pass through the transition phases in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Run the following command to see the status of a specific scaled agent:

   $ oc -n <hosted_control_plane_namespace> get agent -o \
     jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} \
     Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
   BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

3. Run the following command to see the transition phases:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER           APPROVED       ROLE        STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true          auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a                      true          auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true          auto-assign

4. Run the following command to generate the kubeconfig file to access the hosted cluster:

   $ hcp create kubeconfig --namespace <clusters_namespace> --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig

5. After the agents reach the added-to-existing-cluster state, verify that you can see the {ocp-short} nodes by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

   Example output

   NAME                             STATUS   ROLES    AGE      VERSION
   worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
   worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

   Example output

   NAME                                CLUSTER  NODENAME PROVIDERID     PHASE     AGE   VERSION
   hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
   hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

7. Run the following command to check the cluster version:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

   Example output

   NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version   4.15.0-ec.2   True        False         40h     Cluster version is 4.15.0-ec.2

8. Run the following command to check the cluster operator status:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

Procedure

1. Create the hosted control plane namespace by entering the following command:

   $ oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>

   Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters. Replace <hosted_cluster_name> with your hosted cluster name.

2. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command:

   $ hcp create cluster agent \
       --name=<hosted_cluster_name> \1
       --pull-secret=<path_to_pull_secret> \2
       --agent-namespace=<hosted_control_plane_namespace> \3
       --base-domain=<basedomain> \4
       --api-server-address=api.<hosted_cluster_name>.<basedomain> \5
       --etcd-storage-class=<etcd_storage_class> \6
       --ssh-key  <path_to_ssh_public_key> \7
       --namespace <hosted_cluster_namespace> \8
       --control-plane-availability-policy SingleReplica \
       --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9

   1

   Specify the name of your hosted cluster, for instance, example.

   2

   Specify the path to your pull secret, for example, /user/name/pullsecret.

   3

   Specify your hosted control plane namespace, for example, clusters-example. Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command.

   4

   Specify your base domain, for example, krnl.es.

   5

   The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster.

   6

   Specify the etcd storage class name, for example, lvm-storageclass.

   7

   Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

   8

   Specify your hosted cluster namespace.

   9

   Specify the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64. If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest.

3. After a few moments, verify that your hosted control plane pods are up and running by entering the following command:

   $ oc -n <hosted_control_plane_namespace> get pods

   Example output

   NAME                                             READY   STATUS    RESTARTS   AGE
   capi-provider-7dcf5fc4c4-nr9sq                   1/1     Running   0          4m32s
   catalog-operator-6cd867cc7-phb2q                 2/2     Running   0          2m50s
   certified-operators-catalog-884c756c4-zdt64      1/1     Running   0          2m51s
   cluster-api-f75d86f8c-56wfz                      1/1     Running   0          4m32s

Procedure

1. Scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER         APPROVED   ROLE          STAGE
   4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
   d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

3. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
   BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
   BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

   Example output

   NAME           STATUS   ROLES    AGE     VERSION
   ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
   ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machines

   Example output

   NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
   hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
   hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

   The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

7. Enter the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

   Example output

   NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
   
   NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
   clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
   clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb
     labels:
       openshift.io/cluster-monitoring: "true"
     annotations:
       workload.openshift.io/allowed: management
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator-operatorgroup
     namespace: metallb
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: metallb
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

2. Save the file as metallb-operator-config.yaml.

3. Enter the following command to apply the configuration:

   $ oc apply -f metallb-operator-config.yaml

4. After the Operator is running, create the MetalLB instance:

   1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb

   2. Save the file as metallb-instance-config.yaml.

   3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml

5. Configure the MetalLB Operator by creating two resources:

   • An IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

   • A BGPAdvertisement resource to advertise the load balancer IP addresses that the IPAddressPool resource provides through the BGP protocol.

     1. Create a YAML file to contain the configuration:

        apiVersion: metallb.io/v1beta1
        kind: IPAddressPool
        metadata:
          name: <ip_address_pool_name> 1
          namespace: metallb
        spec:
          protocol: layer2
          autoAssign: false
          addresses:
            - <ingress_ip>-<ingress_ip> 2
        ---
        apiVersion: metallb.io/v1beta1
        kind: BGPAdvertisement
        metadata:
          name: <bgp_advertisement_name> 3
          namespace: metallb
        spec:
          ipAddressPools:
            - <ip_address_pool_name> 4

        1 4

        Specify the IPAddressPool resource name.

        2

        Specify the IP address for your environment, for example, 192.168.122.23.

        3

        Specify the BGPAdvertisement resource name.

     2. Save the file as ipaddresspool-bgpadvertisement-config.yaml.

     3. Create the resources by entering the following command:

        $ oc apply -f ipaddresspool-bgpadvertisement-config.yaml

6. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

   1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer

   2. Save the metallb-loadbalancer-service.yaml file.

   3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml

   4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

   5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64.

1. Verify that the spec.nodeDrainTimeout value in your NodePool resource is greater than 0s. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace and <nodepool_name> with the node pool name. Run the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep nodeDrainTimeout

   Example output

   nodeDrainTimeout: 30s

2. If the spec.nodeDrainTimeout value is not greater than 0s, modify the value by running the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec":{"nodeDrainTimeout": "30m"}}' --type=merge

3. Enable machine health checks by setting the spec.management.autoRepair field to true in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":true}}}' --type=merge

4. Verify that the NodePool resource is updated with the autoRepair: true value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Disable machine health checks by setting the spec.management.autoRepair field to false in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":false}}}' --type=merge

2. Verify that the NodePool resource is updated with the autoRepair: false value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER         APPROVED   ROLE          STAGE
   4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
   d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

3. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
   BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
   BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

   Example output

   NAME           STATUS   ROLES    AGE     VERSION
   ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
   ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machines

   Example output

   NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
   hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
   hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

   The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

7. Enter the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

   Example output

   NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
   
   NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
   clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
   clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb
     labels:
       openshift.io/cluster-monitoring: "true"
     annotations:
       workload.openshift.io/allowed: management
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator-operatorgroup
     namespace: metallb
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: metallb
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

2. Save the file as metallb-operator-config.yaml.

3. Enter the following command to apply the configuration:

   $ oc apply -f metallb-operator-config.yaml

4. After the Operator is running, create the MetalLB instance:

   1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb

   2. Save the file as metallb-instance-config.yaml.

   3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml

5. Configure the MetalLB Operator by creating two resources:

   • An IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

   • A BGPAdvertisement resource to advertise the load balancer IP addresses that the IPAddressPool resource provides through the BGP protocol.

     1. Create a YAML file to contain the configuration:

        apiVersion: metallb.io/v1beta1
        kind: IPAddressPool
        metadata:
          name: <ip_address_pool_name> 1
          namespace: metallb
        spec:
          protocol: layer2
          autoAssign: false
          addresses:
            - <ingress_ip>-<ingress_ip> 2
        ---
        apiVersion: metallb.io/v1beta1
        kind: BGPAdvertisement
        metadata:
          name: <bgp_advertisement_name> 3
          namespace: metallb
        spec:
          ipAddressPools:
            - <ip_address_pool_name> 4

        1 4

        Specify the IPAddressPool resource name.

        2

        Specify the IP address for your environment, for example, 192.168.122.23.

        3

        Specify the BGPAdvertisement resource name.

     2. Save the file as ipaddresspool-bgpadvertisement-config.yaml.

     3. Create the resources by entering the following command:

        $ oc apply -f ipaddresspool-bgpadvertisement-config.yaml

6. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

   1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer

   2. Save the metallb-loadbalancer-service.yaml file.

   3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml

   4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

   5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.14.0-x86_64.

1. Verify that the spec.nodeDrainTimeout value in your NodePool resource is greater than 0s. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace and <nodepool_name> with the node pool name. Run the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep nodeDrainTimeout

   Example output

   nodeDrainTimeout: 30s

2. If the spec.nodeDrainTimeout value is not greater than 0s, modify the value by running the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec":{"nodeDrainTimeout": "30m"}}' --type=merge

3. Enable machine health checks by setting the spec.management.autoRepair field to true in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":true}}}' --type=merge

4. Verify that the NodePool resource is updated with the autoRepair: true value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Disable machine health checks by setting the spec.management.autoRepair field to false in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":false}}}' --type=merge

2. Verify that the NodePool resource is updated with the autoRepair: false value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Create a YAML file to configure an InfraEnv resource. See the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name> \1
     namespace: <hosted_control_plane_namespace> \2
   spec:
     cpuArchitecture: ppc64le
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <path_to_ssh_public_key> 3

   1

   Replace <hosted_cluster_name> with the name of your hosted cluster.

   2

   Replace <hosted_control_plane_namespace> with the name of the hosted control plane namespace, for example, clusters-hosted.

   3

   Replace <path_to_ssh_public_key> with the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

2. Save the file as infraenv-config.yaml.

3. Apply the configuration by entering the following command:

   $ oc apply -f infraenv-config.yaml

4. To fetch the URL to download the live ISO, which allows IBM Power machines to join as agents, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

Procedure

1. Download the live ISO and use it to start a bare metal or a virtual machine (VM) host. You can find the URL for the live ISO in the status.isoDownloadURL field, in the InfraEnv resource. At startup, the host communicates with the Assisted Service and registers as an agent in the same namespace as the InfraEnv resource.

2. To list the agents and some of their properties, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agents

   Example output

   NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   86f7ac75-4fc4-4b36-8130-40fa12602218                        auto-assign
   e57a637f-745b-496e-971d-1abbf03341ba                        auto-assign

3. After each agent is created, you can optionally set the installation_disk_id and hostname for an agent:

   1. To set the installation_disk_id field for an agent, enter the following command:

      $ oc -n <hosted_control_plane_namespace> patch agent <agent_name> -p '{"spec":{"installation_disk_id":"<installation_disk_id>","approved":true}}' --type merge

   2. To set the hostname field for an agent, enter the following command:

      $ oc -n <hosted_control_plane_namespace> patch agent <agent_name> -p '{"spec":{"hostname":"<hostname>","approved":true}}' --type merge

Procedure

1. Run the following command to scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Run the following command to see the status of a specific scaled agent:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
   BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

3. Run the following command to see the transition phases:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER            APPROVED       ROLE          STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true           auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a                      true           auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true           auto-assign

4. Run the following command to generate the kubeconfig file to access the hosted cluster:

   $ hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

   Example output

   NAME                             STATUS   ROLES    AGE      VERSION
   worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
   worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

   Example output

   NAME                                CLUSTER                  NODENAME                           PROVIDERID                                     PHASE     AGE   VERSION
   hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
   hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

7. Run the following command to check the cluster version:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion

   Example output

   NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version   4.15.0        True        False         40h     Cluster version is 4.15.0

8. Run the following command to check the Cluster Operator status:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

   For each component of your cluster, the output shows the following Cluster Operator statuses:

   • NAME
   • VERSION
   • AVAILABLE
   • PROGRESSING
   • DEGRADED
   • SINCE
   • MESSAGE

1. Ensure that your ${HOME}/.docker/config.json file is updated with the registries that you are going to mirror from and with the private registry that you plan to push the images to.

2. By using the following example, create an ImageSetConfiguration object to use for mirroring. Replace values as needed to match your environment:

   apiVersion: mirror.openshift.io/v1alpha2
   kind: ImageSetConfiguration
   storageConfig:
     registry:
       imageURL: registry.<dns.base.domain.name>:5000/openshift/release/metadata:latest 1
   mirror:
     platform:
       channels:
       - name: candidate-4.17
         minVersion: 4.x.y-build  2
         maxVersion: 4.x.y-build 3
         type: ocp
       kubeVirtContainer: true 4
       graph: true
     additionalImages:
     - name: quay.io/karmab/origin-keepalived-ipfailover:latest
     - name: quay.io/karmab/kubectl:latest
     - name: quay.io/karmab/haproxy:latest
     - name: quay.io/karmab/mdns-publisher:latest
     - name: quay.io/karmab/origin-coredns:latest
     - name: quay.io/karmab/curl:latest
     - name: quay.io/karmab/kcli:latest
     - name: quay.io/user-name/trbsht:latest
     - name: quay.io/user-name/hypershift:BMSelfManage-v4.17
     - name: registry.redhat.io/openshift4/ose-kube-rbac-proxy:v4.10
     operators:
     - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17
       packages:
       - name: lvms-operator
       - name: local-storage-operator
       - name: odf-csi-addons-operator
       - name: odf-operator
       - name: mcg-operator
       - name: ocs-operator
       - name: metallb-operator
       - name: kubevirt-hyperconverged 5

   1

   Replace <dns.base.domain.name> with the DNS base domain name.

   2 3

   Replace 4.x.y-build with the supported OpenShift Container Platform version you want to use.

   4

   Set this optional flag to true if you want to also mirror the container disk image for the Red Hat Enterprise Linux CoreOS (RHCOS) boot image for the KubeVirt provider. This flag is available with oc-mirror v2 only.

   5

   For deployments that use the KubeVirt provider, include this line.

3. Start the mirroring process by entering the following command:

   $ oc-mirror --v2 --config imagesetconfig.yaml docker://${REGISTRY}

   After the mirroring process is finished, you have a new folder named oc-mirror-workspace/results-XXXXXX/, which contains the IDMS and the catalog sources to apply on the hosted cluster.

4. Mirror the nightly or CI versions of OpenShift Container Platform by configuring the imagesetconfig.yaml file as follows:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     platform:
       graph: true
       release: registry.ci.openshift.org/ocp/release:4.x.y-build 1
       kubeVirtContainer: true 2
   # ...

   1

   Replace 4.x.y-build with the supported OpenShift Container Platform version you want to use.

   2

   Set this optional flag to true if you want to also mirror the container disk image for the Red Hat Enterprise Linux CoreOS (RHCOS) boot image for the KubeVirt provider. This flag is available with oc-mirror v2 only.

5. Apply the changes to the file by entering the following command:

   $ oc-mirror --v2 --config imagesetconfig.yaml docker://${REGISTRY}

6. Mirror the latest multicluster engine Operator images by following the steps in Install on disconnected networks.