Chapter 5. Managing hosted control planes

Procedure

1. Scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER         APPROVED   ROLE          STAGE
   4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
   d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

3. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
   BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
   BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

   Example output

   NAME           STATUS   ROLES    AGE     VERSION
   ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
   ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machines

   Example output

   NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
   hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
   hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

   The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

7. Enter the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

   Example output

   NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
   
   NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
   clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
   clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb
     labels:
       openshift.io/cluster-monitoring: "true"
     annotations:
       workload.openshift.io/allowed: management
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator-operatorgroup
     namespace: metallb
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: metallb
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

2. Save the file as metallb-operator-config.yaml.

3. Enter the following command to apply the configuration:

   $ oc apply -f metallb-operator-config.yaml

4. After the Operator is running, create the MetalLB instance:

   1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb

   2. Save the file as metallb-instance-config.yaml.

   3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml

5. Configure the MetalLB Operator by creating two resources:

   • An IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

   • A BGPAdvertisement resource to advertise the load balancer IP addresses that the IPAddressPool resource provides through the BGP protocol.

     1. Create a YAML file to contain the configuration:

        apiVersion: metallb.io/v1beta1
        kind: IPAddressPool
        metadata:
          name: <ip_address_pool_name> 1
          namespace: metallb
        spec:
          protocol: layer2
          autoAssign: false
          addresses:
            - <ingress_ip>-<ingress_ip> 2
        ---
        apiVersion: metallb.io/v1beta1
        kind: BGPAdvertisement
        metadata:
          name: <bgp_advertisement_name> 3
          namespace: metallb
        spec:
          ipAddressPools:
            - <ip_address_pool_name> 4

        1 4

        Specify the IPAddressPool resource name.

        2

        Specify the IP address for your environment, for example, 192.168.122.23.

        3

        Specify the BGPAdvertisement resource name.

     2. Save the file as ipaddresspool-bgpadvertisement-config.yaml.

     3. Create the resources by entering the following command:

        $ oc apply -f ipaddresspool-bgpadvertisement-config.yaml

6. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

   1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer

   2. Save the metallb-loadbalancer-service.yaml file.

   3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml

   4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

   5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

1. Verify that the spec.nodeDrainTimeout value in your NodePool resource is greater than 0s. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace and <nodepool_name> with the node pool name. Run the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep nodeDrainTimeout

   Example output

   nodeDrainTimeout: 30s

2. If the spec.nodeDrainTimeout value is not greater than 0s, modify the value by running the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec":{"nodeDrainTimeout": "30m"}}' --type=merge

3. Enable machine health checks by setting the spec.management.autoRepair field to true in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":true}}}' --type=merge

4. Verify that the NodePool resource is updated with the autoRepair: true value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Disable machine health checks by setting the spec.management.autoRepair field to false in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":false}}}' --type=merge

2. Verify that the NodePool resource is updated with the autoRepair: false value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through states in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER         APPROVED   ROLE          STAGE
   4dac1ab2-7dd5-4894-a220-6a3473b67ee6   hypercluster1   true       auto-assign
   d9198891-39f4-4930-a679-65fb142b108b                   true       auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hypercluster1   true       auto-assign

3. Enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: ocp-worker-2 Agent: 4dac1ab2-7dd5-4894-a220-6a3473b67ee6 State: binding
   BMH: ocp-worker-0 Agent: d9198891-39f4-4930-a679-65fb142b108b State: known-unbound
   BMH: ocp-worker-1 Agent: da503cf1-a347-44f2-875c-4960ddb04091 State: insufficient

4. Obtain the kubeconfig for your new hosted cluster by entering the extract command:

   $ oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes in the hosted cluster by entering the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get nodes

   Example output

   NAME           STATUS   ROLES    AGE     VERSION
   ocp-worker-1   Ready    worker   5m41s   v1.24.0+3882f8f
   ocp-worker-2   Ready    worker   6m3s    v1.24.0+3882f8f

   Cluster Operators start to reconcile by adding workloads to the nodes.

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machines

   Example output

   NAME                            CLUSTER               NODENAME       PROVIDERID                                     PHASE     AGE   VERSION
   hypercluster1-c96b6f675-m5vch   hypercluster1-b2qhl   ocp-worker-1   agent://da503cf1-a347-44f2-875c-4960ddb04091   Running   15m   4.x.z
   hypercluster1-c96b6f675-tl42p   hypercluster1-b2qhl   ocp-worker-2   agent://4dac1ab2-7dd5-4894-a220-6a3473b67ee6   Running   15m   4.x.z

   The clusterversion reconcile process eventually reaches a point where only Ingress and Console cluster operators are missing.

7. Enter the following command:

   $ oc --kubeconfig kubeconfig-<hosted_cluster_name> get clusterversion,co

   Example output

   NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version             False       True          40m     Unable to apply 4.x.z: the cluster operator console has not yet successfully rolled out
   
   NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   clusteroperator.config.openshift.io/console                                      4.12z     False       False         False      11m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.hypercluster1.domain.com): Get "https://console-openshift-console.apps.hypercluster1.domain.com": dial tcp 10.19.3.29:443: connect: connection refused
   clusteroperator.config.openshift.io/csi-snapshot-controller                      4.12z     True        False         False      10m
   clusteroperator.config.openshift.io/dns                                          4.12z     True        False         False      9m16s

1. Deploy MetalLB by creating a YAML file that contains the configuration for the MetalLB Operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: metallb
     labels:
       openshift.io/cluster-monitoring: "true"
     annotations:
       workload.openshift.io/allowed: management
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: metallb-operator-operatorgroup
     namespace: metallb
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: metallb-operator
     namespace: metallb
   spec:
     channel: "stable"
     name: metallb-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

2. Save the file as metallb-operator-config.yaml.

3. Enter the following command to apply the configuration:

   $ oc apply -f metallb-operator-config.yaml

4. After the Operator is running, create the MetalLB instance:

   1. Create a YAML file that contains the configuration for the MetalLB instance:

      apiVersion: metallb.io/v1beta1
      kind: MetalLB
      metadata:
        name: metallb
        namespace: metallb

   2. Save the file as metallb-instance-config.yaml.

   3. Create the MetalLB instance by entering this command:

      $ oc apply -f metallb-instance-config.yaml

5. Configure the MetalLB Operator by creating two resources:

   • An IPAddressPool resource with a single IP address. This IP address must be on the same subnet as the network that the cluster nodes use.

   • A BGPAdvertisement resource to advertise the load balancer IP addresses that the IPAddressPool resource provides through the BGP protocol.

     1. Create a YAML file to contain the configuration:

        apiVersion: metallb.io/v1beta1
        kind: IPAddressPool
        metadata:
          name: <ip_address_pool_name> 1
          namespace: metallb
        spec:
          protocol: layer2
          autoAssign: false
          addresses:
            - <ingress_ip>-<ingress_ip> 2
        ---
        apiVersion: metallb.io/v1beta1
        kind: BGPAdvertisement
        metadata:
          name: <bgp_advertisement_name> 3
          namespace: metallb
        spec:
          ipAddressPools:
            - <ip_address_pool_name> 4

        1 4

        Specify the IPAddressPool resource name.

        2

        Specify the IP address for your environment, for example, 192.168.122.23.

        3

        Specify the BGPAdvertisement resource name.

     2. Save the file as ipaddresspool-bgpadvertisement-config.yaml.

     3. Create the resources by entering the following command:

        $ oc apply -f ipaddresspool-bgpadvertisement-config.yaml

6. After creating a service of the LoadBalancer type, MetalLB adds an external IP address for the service.

   1. Configure a new load balancer service that routes ingress traffic to the ingress deployment by creating a YAML file named metallb-loadbalancer-service.yaml:

      kind: Service
      apiVersion: v1
      metadata:
        annotations:
          metallb.universe.tf/address-pool: ingress-public-ip
        name: metallb-ingress
        namespace: openshift-ingress
      spec:
        ports:
          - name: http
            protocol: TCP
            port: 80
            targetPort: 80
          - name: https
            protocol: TCP
            port: 443
            targetPort: 443
        selector:
          ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
        type: LoadBalancer

   2. Save the metallb-loadbalancer-service.yaml file.

   3. Enter the following command to apply the YAML configuration:

      $ oc apply -f metallb-loadbalancer-service.yaml

   4. Enter the following command to reach the OpenShift Container Platform console:

      $ curl -kI https://console-openshift-console.apps.example.krnl.es

      Example output

      HTTP/1.1 200 OK

   5. Check the clusterversion and clusteroperator values to verify that everything is running. Enter the following command:

      $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co

      Example output

      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version   4.x.y      True        False        3m32s   Cluster version is 4.x.y
      
      NAME                                                                             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      clusteroperator.config.openshift.io/console                                      4.x.y     True        False         False      3m50s
      clusteroperator.config.openshift.io/ingress                                      4.x.y     True        False         False      53m

      Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use, for example, 4.17.0-multi.

1. Verify that the spec.nodeDrainTimeout value in your NodePool resource is greater than 0s. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace and <nodepool_name> with the node pool name. Run the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep nodeDrainTimeout

   Example output

   nodeDrainTimeout: 30s

2. If the spec.nodeDrainTimeout value is not greater than 0s, modify the value by running the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec":{"nodeDrainTimeout": "30m"}}' --type=merge

3. Enable machine health checks by setting the spec.management.autoRepair field to true in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":true}}}' --type=merge

4. Verify that the NodePool resource is updated with the autoRepair: true value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Disable machine health checks by setting the spec.management.autoRepair field to false in the NodePool resource. Run the following command:

   $ oc patch nodepool -n <hosted_cluster_namespace> <nodepool_name> -p '{"spec": {"management": {"autoRepair":false}}}' --type=merge

2. Verify that the NodePool resource is updated with the autoRepair: false value by running the following command:

   $ oc get nodepool -n <hosted_cluster_namespace> <nodepool_name> -o yaml | grep autoRepair

Procedure

1. Create a YAML file to configure an InfraEnv resource. See the following example:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: <hosted_cluster_name> \1
     namespace: <hosted_control_plane_namespace> \2
   spec:
     cpuArchitecture: ppc64le
     pullSecretRef:
       name: pull-secret
     sshAuthorizedKey: <path_to_ssh_public_key> 3

   1

   Replace <hosted_cluster_name> with the name of your hosted cluster.

   2

   Replace <hosted_control_plane_namespace> with the name of the hosted control plane namespace, for example, clusters-hosted.

   3

   Replace <path_to_ssh_public_key> with the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub.

2. Save the file as infraenv-config.yaml.

3. Apply the configuration by entering the following command:

   $ oc apply -f infraenv-config.yaml

4. To fetch the URL to download the live ISO, which allows IBM Power machines to join as agents, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json

Procedure

1. Download the live ISO and use it to start a bare metal or a virtual machine (VM) host. You can find the URL for the live ISO in the status.isoDownloadURL field, in the InfraEnv resource. At startup, the host communicates with the Assisted Service and registers as an agent in the same namespace as the InfraEnv resource.

2. To list the agents and some of their properties, enter the following command:

   $ oc -n <hosted_control_plane_namespace> get agents

   Example output

   NAME                                   CLUSTER   APPROVED   ROLE          STAGE
   86f7ac75-4fc4-4b36-8130-40fa12602218                        auto-assign
   e57a637f-745b-496e-971d-1abbf03341ba                        auto-assign

3. After each agent is created, you can optionally set the installation_disk_id and hostname for an agent:

   1. To set the installation_disk_id field for an agent, enter the following command:

      $ oc -n <hosted_control_plane_namespace> patch agent <agent_name> -p '{"spec":{"installation_disk_id":"<installation_disk_id>","approved":true}}' --type merge

   2. To set the hostname field for an agent, enter the following command:

      $ oc -n <hosted_control_plane_namespace> patch agent <agent_name> -p '{"spec":{"hostname":"<hostname>","approved":true}}' --type merge

Procedure

1. Run the following command to scale the NodePool object to two nodes:

   $ oc -n <hosted_cluster_namespace> scale nodepool <nodepool_name> --replicas 2

   The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order:

   • binding
   • discovering
   • insufficient
   • installing
   • installing-in-progress
   • added-to-existing-cluster

2. Run the following command to see the status of a specific scaled agent:

   $ oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'

   Example output

   BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound
   BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient

3. Run the following command to see the transition phases:

   $ oc -n <hosted_control_plane_namespace> get agent

   Example output

   NAME                                   CLUSTER            APPROVED       ROLE          STAGE
   50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   hosted-forwarder   true           auto-assign
   5e498cd3-542c-e54f-0c58-ed43e28b568a                      true           auto-assign
   da503cf1-a347-44f2-875c-4960ddb04091   hosted-forwarder   true           auto-assign

4. Run the following command to generate the kubeconfig file to access the hosted cluster:

   $ hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig

5. After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes

   Example output

   NAME                             STATUS   ROLES    AGE      VERSION
   worker-zvm-0.hostedn.example.com Ready    worker   5m41s    v1.24.0+3882f8f
   worker-zvm-1.hostedn.example.com Ready    worker   6m3s     v1.24.0+3882f8f

6. Enter the following command to verify that two machines were created when you scaled up the NodePool object:

   $ oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io

   Example output

   NAME                                CLUSTER                  NODENAME                           PROVIDERID                                     PHASE     AGE   VERSION
   hosted-forwarder-79558597ff-5tbqp   hosted-forwarder-crqq5   worker-zvm-0.hostedn.example.com   agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d   Running   41h   4.15.0
   hosted-forwarder-79558597ff-lfjfk   hosted-forwarder-crqq5   worker-zvm-1.hostedn.example.com   agent://5e498cd3-542c-e54f-0c58-ed43e28b568a   Running   41h   4.15.0

7. Run the following command to check the cluster version:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion

   Example output

   NAME                                         VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
   clusterversion.config.openshift.io/version   4.15.0        True        False         40h     Cluster version is 4.15.0

8. Run the following command to check the Cluster Operator status:

   $ oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators

   For each component of your cluster, the output shows the following Cluster Operator statuses:

   • NAME
   • VERSION
   • AVAILABLE
   • PROGRESSING
   • DEGRADED
   • SINCE
   • MESSAGE