Postinstallation configuration

Procedure

1. Edit the Image Registry Operator config object and set networkAccess.type to Internal:

   $ oc edit configs.imageregistry/cluster

   # ...
   spec:
     # ...
      storage:
         azure:
           # ...
           networkAccess:
             type: Internal
   # ...

2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

   $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

3. Optional: If the registry is exposed by a route, and you are configuring your storage account to be private, you must disable redirect if you want pulls external to the cluster to continue to work. Enter the following command to disable redirect on the Image Operator configuration:

   $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}'

   Note

   When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

1. Fetch the registry service name by running the following command:

   $ oc get imagestream -n openshift

   Example output

   NAME   IMAGE REPOSITORY                                                 TAGS     UPDATED
   cli    image-registry.openshift-image-registry.svc:5000/openshift/cli   latest   8 hours ago
   ...

2. Enter debug mode by running the following command:

   $ oc debug node/<node_name>

3. Run the suggested chroot command. For example:

   $ chroot /host

4. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

   Example output

   Login Succeeded!

5. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

   Example output

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Procedure

1. Edit the Image Registry Operator config object and configure the private endpoint using your VNet and subnet names:

   $ oc edit configs.imageregistry/cluster

   # ...
   spec:
     # ...
      storage:
         azure:
           # ...
           networkAccess:
             type: Internal
             internal:
               subnetName: <subnet_name>
               vnetName: <vnet_name>
               networkResourceGroupName: <network_resource_group_name>
   # ...

2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

   $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

   Note

   When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

1. Fetch the registry service name by running the following command:

   $ oc get imagestream -n openshift

   Example output

   NAME   IMAGE REPOSITORY                                                 TAGS     UPDATED
   cli    image-registry.openshift-image-registry.svc:5000/openshift/cli   latest   8 hours ago
   ...

2. Enter debug mode by running the following command:

   $ oc debug node/<node_name>

3. Run the suggested chroot command. For example:

   $ chroot /host

4. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

   Example output

   Login Succeeded!

5. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

   Example output

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Verification

1. Fetch the registry service name by running the following command:

   $ oc get imagestream -n openshift

   Example output

   NAME   IMAGE REPOSITORY                                           TAGS     UPDATED
   cli    default-route-openshift-image-registry.<cluster_dns>/cli   latest   8 hours ago
   ...

2. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) default-route-openshift-image-registry.<cluster_dns>

   Example output

   Login Succeeded!

3. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false default-route-openshift-image-registry.<cluster_dns>
   /openshift/tools

   Example output

   Trying to pull default-route-openshift-image-registry.<cluster_dns>/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Log in to your Azure account:

   $ az login

2. Create a storage account and upload the aarch64 virtual hard disk (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS 1

   1

   The westus object is an example region.

3. Create a storage container using the storage account you generated:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

4. You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')

   2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

6. Copy the RHCOS VHD into the storage container:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   You can check the status of the copying process with the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success", 1
    "statusDescription": null
   }

   1

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery using the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

   Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

   $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

10. Your arm64 boot image is now generated. You can access the ID of your image with the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

Procedure

1. Log in to your Azure account by running the following command:

   $ az login

2. Create a storage account and upload the x86_64 virtual hard disk (VHD) to your storage account by running the following command. The OpenShift Container Platform installation program creates a resource group. However, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS 1

   1

   The westus object is an example region.

3. Create a storage container using the storage account you generated by running the following command:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

4. Use the OpenShift Container Platform installation program JSON file to extract the URL and x86_64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.x86_64."rhel-coreos-extensions"."azure-disk".url')

   2. Extract the x86_64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.x86_64."rhel-coreos-extensions"."azure-disk".release')-azure.x86_64.vhd

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container by running the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

6. Copy the RHCOS VHD into the storage container by running the following command:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   You can check the status of the copying process by running the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success", 1
    "statusDescription": null
   }

   1

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery by running the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

8. Use the image gallery to create an image definition by running the following command:

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-x86_64 --publisher RedHat --offer x86_64 --sku x86_64 --os-type linux --architecture x64 --hyper-v-generation V2

   In this example command, rhcos-x86_64 is the name of the image definition.

9. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

10. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version by running the following command:

    $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

    In this example, 1.0.0 is the image version.

11. Optional: Access the ID of the generated x86_64 boot image by running the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-x86_64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-x86_64/versions/1.0.0

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for an Azure 64-bit ARM or 64-bit x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
       machine.openshift.io/cluster-api-machine-role: worker
       machine.openshift.io/cluster-api-machine-type: worker
     name: <infrastructure_id>-machine-set-0
     namespace: openshift-machine-api
   spec:
     replicas: 2
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: worker
           machine.openshift.io/cluster-api-machine-type: worker
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-machine-set-0
       spec:
         lifecycleHooks: {}
         metadata: {}
         providerSpec:
           value:
             acceleratedNetworking: true
             apiVersion: machine.openshift.io/v1beta1
             credentialsSecret:
               name: azure-cloud-credentials
               namespace: openshift-machine-api
             image:
               offer: ""
               publisher: ""
               resourceID: /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0 1
               sku: ""
               version: ""
             kind: AzureMachineProviderSpec
             location: <region>
             managedIdentity: <infrastructure_id>-identity
             networkResourceGroup: <infrastructure_id>-rg
             osDisk:
               diskSettings: {}
               diskSizeGB: 128
               managedDisk:
                 storageAccountType: Premium_LRS
               osType: Linux
             publicIP: false
             publicLoadBalancer: <infrastructure_id>
             resourceGroup: <infrastructure_id>-rg
             subnet: <infrastructure_id>-worker-subnet
             userDataSecret:
               name: worker-user-data
             vmSize: Standard_D4ps_v5 2
             vnet: <infrastructure_id>-vnet
             zone: "<zone>"

   1

   Set the resourceID parameter to either arm64 or amd64 boot image.

   2

   Set the vmSize parameter to the instance type used in your installation. Some example instance types are Standard_D4ps_v5 or D8ps.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 1

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: arm64-machine-set-0.yaml, or amd64-machine-set-0.yaml.

Verification

1. Verify that the new machines are running by running the following command:

   $ oc get machineset -n openshift-machine-api

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-machine-set-0                   2        2      2          2  10m

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for an AWS 64-bit ARM or x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
     name: <infrastructure_id>-aws-machine-set-0 2
     namespace: openshift-machine-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id> 3
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<zone> 4
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: <role> 5
           machine.openshift.io/cluster-api-machine-type: <role> 6
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>-<zone> 7
       spec:
         metadata:
           labels:
             node-role.kubernetes.io/<role>: ""
         providerSpec:
           value:
             ami:
               id: ami-02a574449d4f4d280 8
             apiVersion: awsproviderconfig.openshift.io/v1beta1
             blockDevices:
               - ebs:
                   iops: 0
                   volumeSize: 120
                   volumeType: gp2
             credentialsSecret:
               name: aws-cloud-credentials
             deviceIndex: 0
             iamInstanceProfile:
               id: <infrastructure_id>-worker-profile 9
             instanceType: m6g.xlarge 10
             kind: AWSMachineProviderConfig
             placement:
               availabilityZone: us-east-1a 11
               region: <region> 12
             securityGroups:
               - filters:
                   - name: tag:Name
                     values:
                       - <infrastructure_id>-worker-sg 13
             subnet:
               filters:
                 - name: tag:Name
                   values:
                     - <infrastructure_id>-private-<zone>
             tags:
               - name: kubernetes.io/cluster/<infrastructure_id> 14
                 value: owned
               - name: <custom_tag_name>
                 value: <custom_tag_value>
             userDataSecret:
               name: worker-user-data

   1 2 3 9 13 14

   Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

   $ oc get -o jsonpath=‘{.status.infrastructureName}{“\n”}’ infrastructure cluster

   4 7

   Specify the infrastructure ID, role node label, and zone.

   5 6

   Specify the role node label to add.

   8

   Specify a Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) for your AWS zone for the nodes. The RHCOS AMI must be compatible with the machine architecture.

   $ oc get configmap/coreos-bootimages \
   	  -n openshift-machine-config-operator \
   	  -o jsonpath='{.data.stream}' | jq \
   	  -r '.architectures.<arch>.images.aws.regions."<region>".image'

   10

   Specify a machine type that aligns with the CPU architecture of the chosen AMI. For more information, see "Tested instance types for AWS 64-bit ARM"

   11

   Specify the zone. For example, us-east-1a. Ensure that the zone you select has machines with the required architecture.

   12

   Specify the region. For example, us-east-1. Ensure that the zone you select has machines with the required architecture.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 1

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: aws-arm64-machine-set-0.yaml, or aws-amd64-machine-set-0.yaml.

Verification

1. View the list of compute machine sets by running the following command:

   $ oc get machineset -n openshift-machine-api

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-aws-machine-set-0                   2        2      2          2  10m

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Log in to the OpenShift CLI (oc).

2. Create a YAML file, and add the configuration to create a compute machine set to control the 64-bit ARM or 64-bit x86 compute nodes in your cluster.

   Example MachineSet object for a GCP 64-bit ARM or 64-bit x86 compute node

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
     name: <infrastructure_id>-w-a
     namespace: openshift-machine-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
     template:
       metadata:
         creationTimestamp: null
         labels:
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: <role> 2
           machine.openshift.io/cluster-api-machine-type: <role>
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
       spec:
         metadata:
           labels:
             node-role.kubernetes.io/<role>: ""
         providerSpec:
           value:
             apiVersion: gcpprovider.openshift.io/v1beta1
             canIPForward: false
             credentialsSecret:
               name: gcp-cloud-credentials
             deletionProtection: false
             disks:
             - autoDelete: true
               boot: true
               image: <path_to_image> 3
               labels: null
               sizeGb: 128
               type: pd-ssd
             gcpMetadata: 4
             - key: <custom_metadata_key>
               value: <custom_metadata_value>
             kind: GCPMachineProviderSpec
             machineType: n1-standard-4 5
             metadata:
               creationTimestamp: null
             networkInterfaces:
             - network: <infrastructure_id>-network
               subnetwork: <infrastructure_id>-worker-subnet
             projectID: <project_name> 6
             region: us-central1 7
             serviceAccounts:
             - email: <infrastructure_id>-w@<project_name>.iam.gserviceaccount.com
               scopes:
               - https://www.googleapis.com/auth/cloud-platform
             tags:
               - <infrastructure_id>-worker
             userDataSecret:
               name: worker-user-data
             zone: us-central1-a

   1

   Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:

   $ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

   2

   Specify the role node label to add.

   3

   Specify the path to the image that is used in current compute machine sets. You need the project and image name for your path to image.

   To access the project and image name, run the following command:

   $ oc get configmap/coreos-bootimages \
     -n openshift-machine-config-operator \
     -o jsonpath='{.data.stream}' | jq \
     -r '.architectures.aarch64.images.gcp'

   Example output

     "gcp": {
       "release": "415.92.202309142014-0",
       "project": "rhcos-cloud",
       "name": "rhcos-415-92-202309142014-0-gcp-aarch64"
     }

   Use the project and name parameters from the output to create the path to image field in your machine set. The path to the image should follow the following format:

   $ projects/<project>/global/images/<image_name>

   4

   Optional: Specify custom metadata in the form of a key:value pair. For example use cases, see the GCP documentation for setting custom metadata.

   5

   Specify a machine type that aligns with the CPU architecture of the chosen OS image. For more information, see "Tested instance types for GCP on 64-bit ARM infrastructures".

   6

   Specify the name of the GCP project that you use for your cluster.

   7

   Specify the region. For example, us-central1. Ensure that the zone you select has machines with the required architecture.

3. Create the compute machine set by running the following command:

   $ oc create -f <file_name> 1

   1

   Replace <file_name> with the name of the YAML file with compute machine set configuration. For example: gcp-arm64-machine-set-0.yaml, or gcp-amd64-machine-set-0.yaml.

Verification

1. View the list of compute machine sets by running the following command:

   $ oc get machineset -n openshift-machine-api

   The output must include the machine set that you created.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-gcp-machine-set-0                   2        2      2          2  10m

2. You can check if the nodes are ready and schedulable by running the following command:

   $ oc get nodes

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + aarch64):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on aarch64:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.30.3
   master-1  Ready     master  63m  v1.30.3
   master-2  Ready     master  64m  v1.30.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.30.3
   master-1  Ready     master  73m  v1.30.3
   master-2  Ready     master  74m  v1.30.3
   worker-0  Ready     worker  11m  v1.30.3
   worker-1  Ready     worker  11m  v1.30.3

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Disable UDP aggregation.

   Currently, UDP aggregation is not supported on IBM Z® and is not automatically deactivated on multi-architecture compute clusters with an x86_64 control plane and additional s390x compute machines. To ensure that the addtional compute nodes are added to the cluster correctly, you must manually disable UDP aggregation.

   1. Create a YAML file udp-aggregation-config.yaml with the following content:

      apiVersion: v1
      kind: ConfigMap
      data:
        disable-udp-aggregation: "true"
      metadata:
        name: udp-aggregation-config
        namespace: openshift-network-operator

   2. Create the ConfigMap resource by running the following command:

      $ oc create -f udp-aggregation-config.yaml

2. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

3. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

4. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<http_server>/worker.ign

5. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

6. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server that is accessible from the z/VM guest you want to add.

7. Create a parameter file for the z/VM guest. The following parameters are specific for the virtual machine:

   • Optional: To specify a static IP address, add an ip= parameter with the following entries, with each separated by a colon:

     1. The IP address for the machine.
     2. An empty string.
     3. The gateway.
     4. The netmask.
     5. The machine host and domain name in the form hostname.domainname. Omit this value to let RHCOS decide.
     6. The network interface name. Omit this value to let RHCOS decide.
     7. The value none.
   • For coreos.inst.ignition_url=, specify the URL to the worker.ign file. Only HTTP and HTTPS protocols are supported.
   • For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   • For installations on DASD-type disks, complete the following tasks:

     1. For coreos.inst.install_dev=, specify /dev/dasda.
     2. Use rd.dasd= to specify the DASD where RHCOS is to be installed.

     3. You can adjust further parameters if required.

        The following is an example parameter file, additional-worker-dasd.parm:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/dasda \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        rd.dasd=0.0.3490 \
        zfcp.allow_lun_scan=0

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

   • For installations on FCP-type disks, complete the following tasks:

     1. Use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. For multipathing, repeat this step for each additional path.

        Note

        When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.

     2. Set the install device as: coreos.inst.install_dev=/dev/sda.

        Note

        If additional LUNs are configured with NPIV, FCP requires zfcp.allow_lun_scan=0. If you must enable zfcp.allow_lun_scan=1 because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.

     3. You can adjust further parameters if required.

        Important

        Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on RHCOS" in Postinstallation machine configuration tasks.

        The following is an example parameter file, additional-worker-fcp.parm for a worker node with multipathing:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/sda \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

8. Transfer the initramfs, kernel, parameter files, and RHCOS images to z/VM, for example, by using FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see Installing under Z/VM.

9. Punch the files to the virtual reader of the z/VM guest virtual machine.

   See PUNCH in IBM® Documentation.

   Tip

   You can use the CP PUNCH command or, if you use Linux, the vmur command to transfer files between two z/VM guest virtual machines.

10. Log in to CMS on the bootstrap machine.

11. IPL the bootstrap machine from the reader by running the following command:

    $ ipl c

    See IPL in IBM® Documentation.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.30.3
   master-1  Ready     master  63m  v1.30.3
   master-2  Ready     master  64m  v1.30.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.30.3
   master-1  Ready     master  73m  v1.30.3
   master-2  Ready     master  74m  v1.30.3
   worker-0  Ready     worker  11m  v1.30.3
   worker-1  Ready     worker  11m  v1.30.3

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Disable UDP aggregation.

   Currently, UDP aggregation is not supported on IBM Z® and is not automatically deactivated on multi-architecture compute clusters with an x86_64 control plane and additional s390x compute machines. To ensure that the addtional compute nodes are added to the cluster correctly, you must manually disable UDP aggregation.

   1. Create a YAML file udp-aggregation-config.yaml with the following content:

      apiVersion: v1
      kind: ConfigMap
      data:
        disable-udp-aggregation: "true"
      metadata:
        name: udp-aggregation-config
        namespace: openshift-network-operator

   2. Create the ConfigMap resource by running the following command:

      $ oc create -f udp-aggregation-config.yaml

2. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

3. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

4. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<http_server>/worker.ign

5. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

6. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server that is accessible from the z/VM guest you want to add.

7. Create a parameter file for the z/VM guest. The following parameters are specific for the virtual machine:

   • Optional: To specify a static IP address, add an ip= parameter with the following entries, with each separated by a colon:

     1. The IP address for the machine.
     2. An empty string.
     3. The gateway.
     4. The netmask.
     5. The machine host and domain name in the form hostname.domainname. Omit this value to let RHCOS decide.
     6. The network interface name. Omit this value to let RHCOS decide.
     7. The value none.
   • For coreos.inst.ignition_url=, specify the URL to the worker.ign file. Only HTTP and HTTPS protocols are supported.
   • For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   • For installations on DASD-type disks, complete the following tasks:

     1. For coreos.inst.install_dev=, specify /dev/dasda.
     2. Use rd.dasd= to specify the DASD where RHCOS is to be installed.

     3. You can adjust further parameters if required.

        The following is an example parameter file, additional-worker-dasd.parm:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/dasda \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        rd.dasd=0.0.3490 \
        zfcp.allow_lun_scan=0

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

   • For installations on FCP-type disks, complete the following tasks:

     1. Use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. For multipathing, repeat this step for each additional path.

        Note

        When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.

     2. Set the install device as: coreos.inst.install_dev=/dev/sda.

        Note

        If additional LUNs are configured with NPIV, FCP requires zfcp.allow_lun_scan=0. If you must enable zfcp.allow_lun_scan=1 because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.

     3. You can adjust further parameters if required.

        Important

        Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on RHCOS" in Postinstallation machine configuration tasks.

        The following is an example parameter file, additional-worker-fcp.parm for a worker node with multipathing:

        cio_ignore=all,!condev rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/sda \
        coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \
        coreos.inst.ignition_url=http://<http_server>/worker.ign \
        ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

8. Transfer the initramfs, kernel, parameter files, and RHCOS images to z/VM, for example, by using FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see Installing under Z/VM.

9. Punch the files to the virtual reader of the z/VM guest virtual machine.

   See PUNCH in IBM® Documentation.

   Tip

   You can use the CP PUNCH command or, if you use Linux, the vmur command to transfer files between two z/VM guest virtual machines.

10. Log in to CMS on the bootstrap machine.

11. IPL the bootstrap machine from the reader by running the following command:

    $ ipl c

    See IPL in IBM® Documentation.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.30.3
   master-1  Ready     master  63m  v1.30.3
   master-2  Ready     master  64m  v1.30.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.30.3
   master-1  Ready     master  73m  v1.30.3
   master-2  Ready     master  74m  v1.30.3
   worker-0  Ready     worker  11m  v1.30.3
   worker-1  Ready     worker  11m  v1.30.3

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Disable UDP aggregation.

   Currently, UDP aggregation is not supported on IBM Z® and is not automatically deactivated on multi-architecture compute clusters with an x86_64 control plane and additional s390x compute machines. To ensure that the addtional compute nodes are added to the cluster correctly, you must manually disable UDP aggregation.

   1. Create a YAML file udp-aggregation-config.yaml with the following content:

      apiVersion: v1
      kind: ConfigMap
      data:
        disable-udp-aggregation: "true"
      metadata:
        name: udp-aggregation-config
        namespace: openshift-network-operator

   2. Create the ConfigMap resource by running the following command:

      $ oc create -f udp-aggregation-config.yaml

2. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

3. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

4. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

5. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

    $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

6. Move the downloaded RHEL live kernel, initramfs and rootfs files to an HTTP or HTTPS server before you launch virt-install.

7. Create the new KVM guest nodes using the RHEL kernel, initramfs, and Ignition files; the new disk image; and adjusted parm line arguments.

   $ virt-install \
      --connect qemu:///system \
      --name <vm_name> \
      --autostart \
      --os-variant rhel9.4 \ 1
      --cpu host \
      --vcpus <vcpus> \
      --memory <memory_mb> \
      --disk <vm_name>.qcow2,size=<image_size> \
      --network network=<virt_network_parm> \
      --location <media_location>,kernel=<rhcos_kernel>,initrd=<rhcos_initrd> \ 2
      --extra-args "rd.neednet=1" \
      --extra-args "coreos.inst.install_dev=/dev/vda" \
      --extra-args "coreos.inst.ignition_url=http://<http_server>/worker.ign " \ 3
      --extra-args "coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img" \ 4
      --extra-args "ip=<ip>::<gateway>:<netmask>:<hostname>::none" \ 5
      --extra-args "nameserver=<dns>" \
      --extra-args "console=ttysclp0" \
      --noautoconsole \
      --wait

   1

   For os-variant, specify the RHEL version for the RHCOS compute machine. rhel9.4 is the recommended version. To query the supported RHEL version of your operating system, run the following command:

   $ osinfo-query os -f short-id

   Note

   The os-variant is case sensitive.

   2

   For --location, specify the location of the kernel/initrd on the HTTP or HTTPS server.

   3

   Specify the location of the worker.ign config file. Only HTTP and HTTPS protocols are supported.

   4

   Specify the location of the rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported

   5

   Optional: For hostname, specify the fully qualified hostname of the client machine.

   Note

   If you are using HAProxy as a load balancer, update your HAProxy rules for ingress-router-443 and ingress-router-80 in the /etc/haproxy/haproxy.cfg configuration file.

8. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.30.3
   master-1  Ready     master  63m  v1.30.3
   master-2  Ready     master  64m  v1.30.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.30.3
   master-1  Ready     master  73m  v1.30.3
   master-2  Ready     master  74m  v1.30.3
   worker-0  Ready     worker  11m  v1.30.3
   worker-1  Ready     worker  11m  v1.30.3

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Log in to the OpenShift CLI (oc).

2. You can check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o jsonpath="{ .metadata.metadata}"

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + ppc64le):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on ppc64le architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on ppc64le:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.30.3
   master-1  Ready     master  63m  v1.30.3
   master-2  Ready     master  64m  v1.30.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes -o wide

   Example output

   NAME               STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                  CONTAINER-RUNTIME
   worker-0-ppc64le   Ready    worker                 42d   v1.30.3   192.168.200.21   <none>        Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.ppc64le   cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   worker-1-ppc64le   Ready    worker                 42d   v1.30.3   192.168.200.20   <none>        Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.ppc64le   cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   master-0-x86       Ready    control-plane,master   75d   v1.30.3   10.248.0.38      10.248.0.38   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   master-1-x86       Ready    control-plane,master   75d   v1.30.3   10.248.0.39      10.248.0.39   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   master-2-x86       Ready    control-plane,master   75d   v1.30.3   10.248.0.40      10.248.0.40   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   worker-0-x86       Ready    worker                 75d   v1.30.3   10.248.0.43      10.248.0.43   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9
   worker-1-x86       Ready    worker                 75d   v1.30.3   10.248.0.44      10.248.0.44   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.30.3-3.rhaos4.15.gitb36169e.el9

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Label the nodes where you want to run the 64k page size kernel:

   $ oc label node <node_name> <label>

   Example command

   $ oc label node worker-arm64-01 node-role.kubernetes.io/worker-64k-pages=

2. Create a machine config pool that contains the worker role that uses the ARM64 architecture and the worker-64k-pages role:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-64k-pages
   spec:
     machineConfigSelector:
       matchExpressions:
         - key: machineconfiguration.openshift.io/role
           operator: In
           values:
           - worker
           - worker-64k-pages
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-64k-pages: ""
         kubernetes.io/arch: arm64

3. Create a machine config on your compute node to enable 64k-pages with the 64k-pages parameter.

   $ oc create -f <filename>.yaml

   Example MachineConfig

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: "worker-64k-pages" 1
     name: 99-worker-64kpages
   spec:
     kernelType: 64k-pages 2

   1

   Specify the value of the machineconfiguration.openshift.io/role label in the custom machine config pool. The example MachineConfig uses the worker-64k-pages label to enable 64k pages in the worker-64k-pages pool.

   2

   Specify your desired kernel type. Valid values are 64k-pages and default

   Note

   The 64k-pages type is supported on only 64-bit ARM architecture based compute nodes. The realtime type is supported on only 64-bit x86 architecture based compute nodes.

Procedure

1. Create a new project named openshift-multiarch-tuning-operator by running the following command:

   $ oc create ns openshift-multiarch-tuning-operator

2. Create an OperatorGroup object:

   1. Create a YAML file with the configuration for creating an OperatorGroup object.

      Example YAML configuration for creating an OperatorGroup object

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-multiarch-tuning-operator
        namespace: openshift-multiarch-tuning-operator
      spec: {}

   2. Create the OperatorGroup object by running the following command:

      $ oc create -f <file_name> 1

      1

      Replace <file_name> with the name of the YAML file that contains the OperatorGroup object configuration.

3. Create a Subscription object:

   1. Create a YAML file with the configuration for creating a Subscription object.

      Example YAML configuration for creating a Subscription object

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-multiarch-tuning-operator
        namespace: openshift-multiarch-tuning-operator
      spec:
        channel: stable
        name: multiarch-tuning-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        installPlanApproval: Automatic
        startingCSV: multiarch-tuning-operator.v1.0.0

   2. Create the Subscription object by running the following command:

      $ oc create -f <file_name> 1

      1

      Replace <file_name> with the name of the YAML file that contains the Subscription object configuration.

Verification

1. To verify that the Multiarch Tuning Operator is installed, run the following command:

   $ oc get csv -n openshift-multiarch-tuning-operator

   Example output

   NAME                               DISPLAY                     VERSION   REPLACES                              PHASE
   multiarch-tuning-operator.v1.0.0   Multiarch Tuning Operator   1.0.0     multiarch-tuning-operator.v0.9.0      Succeeded

   The installation is successful if the Operator is in Succeeded phase.

2. Optional: To verify that the OperatorGroup object is created, run the following command:

   $ oc get operatorgroup -n openshift-multiarch-tuning-operator

   Example output

   NAME                                        AGE
   openshift-multiarch-tuning-operator-q8zbb   133m

3. Optional: To verify that the Subscription object is created, run the following command:

   $ oc get subscription -n openshift-multiarch-tuning-operator

   Example output

   NAME                        PACKAGE                     SOURCE                  CHANNEL
   multiarch-tuning-operator   multiarch-tuning-operator   redhat-operators        stable

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → OperatorHub.
3. Enter Multiarch Tuning Operator in the search field.
4. Click Multiarch Tuning Operator.
5. Select the Multiarch Tuning Operator version from the Version list.
6. Click Install

7. Set the following options on the Operator Installation page:

   1. Set Update Channel to stable.
   2. Set Installation Mode to All namespaces on the cluster.

   3. Set Installed Namespace to Operator recommended Namespace or Select a Namespace.

      The recommended Operator namespace is openshift-multiarch-tuning-operator. If the openshift-multiarch-tuning-operator namespace does not exist, it is created during the operator installation.

      If you select Select a namespace, you must select a namespace for the Operator from the Select Project list.

   4. Update approval as Automatic or Manual.

      If you select Automatic updates, Operator Lifecycle Manager (OLM) automatically updates the running instance of the Multiarch Tuning Operator without any intervention.

      If you select Manual updates, OLM creates an update request. As a cluster administrator, you must manually approve the update request to update the Multiarch Tuning Operator to a newer version.

8. Optional: Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
9. Click Install.

Verification

1. Navigate to Operators → Installed Operators.
2. Verify that the Multiarch Tuning Operator is listed with the Status field as Succeeded in the openshift-multiarch-tuning-operator namespace.

Procedure

1. Log in to the OpenShift CLI (oc).

2. Delete the ClusterPodPlacementConfig object by running the following command:

   $ oc delete clusterpodplacementconfig cluster

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → Installed Operators.
3. On the Installed Operators page, click Multiarch Tuning Operator.
4. Click the Cluster Pod Placement Config tab.
5. Select Delete ClusterPodPlacementConfig from the options menu.
6. Click Delete.

Procedure

1. Get the Subscription object name for the Multiarch Tuning Operator by running the following command:

   $ oc get subscription.operators.coreos.com -n <namespace> 1

   1

   Replace <namespace> with the name of the namespace where you want to uninstall the Multiarch Tuning Operator.

   Example output

   NAME                                  PACKAGE                     SOURCE             CHANNEL
   openshift-multiarch-tuning-operator   multiarch-tuning-operator   redhat-operators   stable

2. Get the currentCSV value for the Multiarch Tuning Operator by running the following command:

   $ oc get subscription.operators.coreos.com <subscription_name> -n <namespace> -o yaml | grep currentCSV 1

   1

   Replace <subscription_name> with the Subscription object name. For example: openshift-multiarch-tuning-operator. Replace <namespace> with the name of the namespace where you want to uninstall the Multiarch Tuning Operator.

   Example output

   currentCSV: multiarch-tuning-operator.v1.0.0

3. Delete the Subscription object by running the following command:

   $ oc delete subscription.operators.coreos.com <subscription_name> -n <namespace> 1

   1

   Replace <subscription_name> with the Subscription object name. Replace <namespace> with the name of the namespace where you want to uninstall the Multiarch Tuning Operator.

   Example output

   subscription.operators.coreos.com "openshift-multiarch-tuning-operator" deleted

4. Delete the CSV for the Multiarch Tuning Operator in the target namespace using the currentCSV value by running the following command:

   $ oc delete clusterserviceversion <currentCSV_value> -n <namespace> 1

   1

   Replace <currentCSV> with the currentCSV value for the Multiarch Tuning Operator. For example: multiarch-tuning-operator.v1.0.0. Replace <namespace> with the name of the namespace where you want to uninstall the Multiarch Tuning Operator.

   Example output

   clusterserviceversion.operators.coreos.com "multiarch-tuning-operator.v1.0.0" deleted

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → OperatorHub.
3. Enter Multiarch Tuning Operator in the search field.
4. Click Multiarch Tuning Operator.
5. Click the Details tab.
6. From the Actions menu, select Uninstall Operator.
7. When prompted, click Uninstall.

Verification

1. Navigate to Operators → Installed Operators.
2. On the Installed Operators page, verify that the Multiarch Tuning Operator is not listed.

Procedure

1. View the compute machine sets that are in the cluster by running the following command:

   $ oc get machinesets.machine.openshift.io -n openshift-machine-api

   The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the compute machines that are in the cluster by running the following command:

   $ oc get machines.machine.openshift.io -n openshift-machine-api

3. Set the annotation on the compute machine that you want to delete by running the following command:

   $ oc annotate machines.machine.openshift.io/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"

4. Scale the compute machine set by running one of the following commands:

   $ oc scale --replicas=2 machinesets.machine.openshift.io <machineset> -n openshift-machine-api

   Or:

   $ oc edit machinesets.machine.openshift.io <machineset> -n openshift-machine-api

   Tip

   You can alternatively apply the following YAML to scale the compute machine set:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     name: <machineset>
     namespace: openshift-machine-api
   spec:
     replicas: 2

   You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.

   Important

   By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed. If the drain operation fails, the machine controller cannot proceed removing the machine.

   You can skip draining the node by annotating machine.openshift.io/exclude-node-draining in a specific machine.

1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

   $ oc edit scheduler cluster

   Example Scheduler Operator CR with a node selector

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   spec:
     defaultNodeSelector: type=user-node,region=east 1
     mastersSchedulable: false

   1

   Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a compute machine set or editing the node directly:

   • Use a compute machine set to add labels to nodes managed by the compute machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api 1

        1

        Add a <key>/<value> pair for each label.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

        Tip

        You can alternatively apply the following YAML to add labels to a compute machine set:

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
          name: <machineset>
          namespace: openshift-machine-api
        spec:
          template:
            spec:
              metadata:
                labels:
                  region: "east"
                  type: "user-node"

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
          ...
        spec:
          ...
          template:
            metadata:
          ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
          ...

     3. Redeploy the nodes associated with that compute machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.30.3

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

        Tip

        You can alternatively apply the following YAML to add labels to a node:

        kind: Node
        apiVersion: v1
        metadata:
          name: <node_name>
          labels:
            type: "user-node"
            region: "east"

     2. Verify that the labels are added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.30.3

1. Move to the medium worker latency profile:

   1. Edit the node.config object:

      $ oc edit nodes.config/cluster

   2. Add spec.workerLatencyProfile: MediumUpdateAverageReaction:

      Example node.config object

      apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: MediumUpdateAverageReaction 1
      
      # ...

      1

      Specifies the medium worker latency policy.

      Scheduling on each worker node is disabled as the change is being applied.

2. Optional: Move to the low worker latency profile:

   1. Edit the node.config object:

      $ oc edit nodes.config/cluster

   2. Change the spec.workerLatencyProfile value to LowUpdateSlowReaction:

      Example node.config object

      apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: LowUpdateSlowReaction 1
      
      # ...

      1

      Specifies use of the low worker latency policy.

Procedure

1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

   Ensure that you set the <clusterID> and <role> parameter values.

2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

   1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api

      Example output

      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m

   2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml

      Example output

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
        name: <infrastructure_id>-<role> 2
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: 3
              ...

      1

      The cluster infrastructure ID.

      2

      A default node label.

      Note

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3

      The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

3. Create a MachineSet CR by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Add a label to the worker node that you want to act as application node:

   $ oc label node <node-name> node-role.kubernetes.io/app=""

2. Add a label to the worker nodes that you want to act as infrastructure nodes:

   $ oc label node <node-name> node-role.kubernetes.io/infra=""

3. Check to see if applicable nodes now have the infra role and app roles:

   $ oc get nodes

4. Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.

   Important

   If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied.

   However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as node-role.kubernetes.io/infra="", when a pod’s label is set to a different node role, such as node-role.kubernetes.io/master="", can cause the pod to become unschedulable. For this reason, use caution when setting the default node selector to specific node roles.

   You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts.

   1. Edit the Scheduler object:

      $ oc edit scheduler cluster

   2. Add the defaultNodeSelector field with the appropriate node selector:

      apiVersion: config.openshift.io/v1
      kind: Scheduler
      metadata:
        name: cluster
      spec:
        defaultNodeSelector: node-role.kubernetes.io/infra="" 1
      # ...

      1

      This example node selector deploys pods on infrastructure nodes by default.

   3. Save the file to apply the changes.

Procedure

1. Add a label to the node you want to assign as the infra node with a specific label:

   $ oc label node <node_name> <label>

   $ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=

2. Create a machine config pool that contains both the worker role and your custom role as machine config selector:

   $ cat infra.mcp.yaml

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: infra
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} 1
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/infra: "" 2

   1

   Add the worker role and your custom role.

   2

   Add the label you added to the node as a nodeSelector.

   Note

   Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool.

3. After you have the YAML file, you can create the machine config pool:

   $ oc create -f infra.mcp.yaml

4. Check the machine configs to ensure that the infrastructure configuration rendered successfully:

   $ oc get machineconfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   00-worker                                                   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-master-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-container-runtime                                 365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   01-worker-kubelet                                           365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-master-ssh                                                                                          3.2.0             31d
   99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries   365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             31d
   99-worker-ssh                                                                                          3.2.0             31d
   rendered-infra-4e48906dca84ee702959c71a53ee80e7             365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             23m
   rendered-master-072d4b2da7f88162636902b074e9e28e            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-master-3e88ec72aed3886dec061df60d16d1af            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-master-419bee7de96134963a15fdf9dd473b25            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-master-53f5c91c7661708adce18739cc0f40fb            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d
   rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-master-dc7f874ec77fc4b969674204332da037            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-2640531be11ba43c61d72e82dc634ce6            5b6fb8349a29735e48446d435962dec4547d3090   3.2.0             31d
   rendered-worker-4e48906dca84ee702959c71a53ee80e7            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             7d3h
   rendered-worker-4f110718fe88e5f349987854a1147755            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             17d
   rendered-worker-afc758e194d6188677eb837842d3b379            02c07496ba0417b3e12b78fb32baf6293d314f79   3.2.0             31d
   rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3            365c1cfd14de5b0e3b85e0fc815b0060f36ab955   3.2.0             13d

   You should see a new machine config, with the rendered-infra-* prefix.

5. Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.

   Note

   After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration.

   1. Create a machine config:

      $ cat infra.mc.yaml

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: 51-infra
        labels:
          machineconfiguration.openshift.io/role: infra 1
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
            - path: /etc/infratest
              mode: 0644
              contents:
                source: data:,infra

      1

      Add the label you added to the node as a nodeSelector.

   2. Apply the machine config to the infra-labeled nodes:

      $ oc create -f infra.mc.yaml

6. Confirm that your new machine config pool is available:

   $ oc get mcp

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   infra    rendered-infra-60e35c2e99f42d976e084fa94da4d0fc    True      False      False      1              1                   1                     0                      4m20s
   master   rendered-master-9360fdb895d4c131c7c4bebbae099c90   True      False      False      3              3                   3                     0                      91m
   worker   rendered-worker-60e35c2e99f42d976e084fa94da4d0fc   True      False      False      2              2                   2                     0                      91m

   In this example, a worker node was changed to an infra node.

Procedure

1. Add a taint to the infra node to prevent scheduling user workloads on it:

   1. Determine if the node has the taint:

      $ oc describe nodes <node_name>

      Sample output

      oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Name:               ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
      Roles:              worker
       ...
      Taints:             node-role.kubernetes.io/infra:NoSchedule
       ...

      This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.

   2. If you have not configured a taint to prevent scheduling user workloads on it:

      $ oc adm taint nodes <node_name> <key>=<value>:<effect>

      For example:

      $ oc adm taint nodes node1 node-role.kubernetes.io/infra=reserved:NoSchedule

      Tip

      You can alternatively apply the following YAML to add the taint:

      kind: Node
      apiVersion: v1
      metadata:
        name: <node_name>
        labels:
          ...
      spec:
        taints:
          - key: node-role.kubernetes.io/infra
            effect: NoSchedule
            value: reserved
        ...

      This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.

      Note

      If a descheduler is used, pods violating node taints could be evicted from the cluster.

   3. Add the taint with NoExecute Effect along with the above taint with NoSchedule Effect:

      $ oc adm taint nodes <node_name> <key>=<value>:<effect>

      For example:

      $ oc adm taint nodes node1 node-role.kubernetes.io/infra=reserved:NoExecute

      Tip

      You can alternatively apply the following YAML to add the taint:

      kind: Node
      apiVersion: v1
      metadata:
        name: <node_name>
        labels:
          ...
      spec:
        taints:
          - key: node-role.kubernetes.io/infra
            effect: NoExecute
            value: reserved
        ...

      This example places a taint on node1 that has the key node-role.kubernetes.io/infra and taint effect NoExecute. Nodes with the NoExecute effect schedule only pods that tolerate the taint. The effect will remove any existing pods from the node that do not have a matching toleration.

2. Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:

   tolerations:
     - effect: NoSchedule 1
       key: node-role.kubernetes.io/infra 2
       value: reserved 3
     - effect: NoExecute 4
       key: node-role.kubernetes.io/infra 5
       operator: Exists 6
       value: reserved 7

   1

   Specify the effect that you added to the node.

   2

   Specify the key that you added to the node.

   3

   Specify the value of the key-value pair taint that you added to the node.

   4

   Specify the effect that you added to the node.

   5

   Specify the key that you added to the node.

   6

   Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node.

   7

   Specify the value of the key-value pair taint that you added to the node.

   This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.

   Note

   Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator.

3. Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.

Procedure

1. View the IngressController custom resource for the router Operator:

   $ oc get ingresscontroller default -n openshift-ingress-operator -o yaml

   The command output resembles the following text:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     creationTimestamp: 2019-04-18T12:35:39Z
     finalizers:
     - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
     generation: 1
     name: default
     namespace: openshift-ingress-operator
     resourceVersion: "11341"
     selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a
   spec: {}
   status:
     availableReplicas: 2
     conditions:
     - lastTransitionTime: 2019-04-18T12:36:15Z
       status: "True"
       type: Available
     domain: apps.<cluster>.example.com
     endpointPublishingStrategy:
       type: LoadBalancerService
     selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

2. Edit the ingresscontroller resource and change the nodeSelector to use the infra label:

   $ oc edit ingresscontroller default -n openshift-ingress-operator

     spec:
       nodePlacement:
         nodeSelector: 1
           matchLabels:
             node-role.kubernetes.io/infra: ""
         tolerations:
         - effect: NoSchedule
           key: node-role.kubernetes.io/infra
           value: reserved
         - effect: NoExecute
           key: node-role.kubernetes.io/infra
           value: reserved

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

3. Confirm that the router pod is running on the infra node.

   1. View the list of router pods and note the node name of the running pod:

      $ oc get pod -n openshift-ingress -o wide

      Example output

      NAME                              READY     STATUS        RESTARTS   AGE       IP           NODE                           NOMINATED NODE   READINESS GATES
      router-default-86798b4b5d-bdlvd   1/1      Running       0          28s       10.130.2.4   ip-10-0-217-226.ec2.internal   <none>           <none>
      router-default-955d875f4-255g8    0/1      Terminating   0          19h       10.129.2.4   ip-10-0-148-172.ec2.internal   <none>           <none>

      In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.

   2. View the node status of the running pod:

      $ oc get node <node_name> 1

      1

      Specify the <node_name> that you obtained from the pod list.

      Example output

      NAME                          STATUS  ROLES         AGE   VERSION
      ip-10-0-217-226.ec2.internal  Ready   infra,worker  17h   v1.30.3

      Because the role list includes infra, the pod is running on the correct node.

Procedure

1. View the config/instance object:

   $ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml

   Example output

   apiVersion: imageregistry.operator.openshift.io/v1
   kind: Config
   metadata:
     creationTimestamp: 2019-02-05T13:52:05Z
     finalizers:
     - imageregistry.operator.openshift.io/finalizer
     generation: 1
     name: cluster
     resourceVersion: "56174"
     selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
     uid: 36fd3724-294d-11e9-a524-12ffeee2931b
   spec:
     httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
     logging: 2
     managementState: Managed
     proxy: {}
     replicas: 1
     requests:
       read: {}
       write: {}
     storage:
       s3:
         bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
         region: us-east-1
   status:
   ...

2. Edit the config/instance object:

   $ oc edit configs.imageregistry.operator.openshift.io/cluster

   spec:
     affinity:
       podAntiAffinity:
         preferredDuringSchedulingIgnoredDuringExecution:
         - podAffinityTerm:
             namespaces:
             - openshift-image-registry
             topologyKey: kubernetes.io/hostname
           weight: 100
     logLevel: Normal
     managementState: Managed
     nodeSelector: 1
       node-role.kubernetes.io/infra: ""
     tolerations:
     - effect: NoSchedule
       key: node-role.kubernetes.io/infra
       value: reserved
     - effect: NoExecute
       key: node-role.kubernetes.io/infra
       value: reserved

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration.

3. Verify the registry pod has been moved to the infrastructure node.

   1. Run the following command to identify the node where the registry pod is located:

      $ oc get pods -o wide -n openshift-image-registry

   2. Confirm the node has the label you specified:

      $ oc describe node <node_name>

      Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.

Procedure

1. Edit the cluster-monitoring-config config map and change the nodeSelector to use the infra label:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |+
       alertmanagerMain:
         nodeSelector: 1
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusK8s:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       prometheusOperator:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       metricsServer:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       kubeStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       telemeterClient:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       openshiftStateMetrics:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       thanosQuerier:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute
       monitoringPlugin:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         tolerations:
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoSchedule
         - key: node-role.kubernetes.io/infra
           value: reserved
           effect: NoExecute

   1

   Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration.

2. Watch the monitoring pods move to the new machines:

   $ watch 'oc get pod -n openshift-monitoring -o wide'

3. If a component has not moved to the infra node, delete the pod with this component:

   $ oc delete pod -n openshift-monitoring <pod>

   The component from the deleted pod is re-created on the infra node.

Procedure

1. Create a YAML file for a ClusterAutoscaler resource that contains the custom resource definition.

2. Create the custom resource in the cluster by running the following command:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the custom resource file.

Procedure

1. Create a YAML file for a MachineAutoscaler resource that contains the custom resource definition.

2. Create the custom resource in the cluster by running the following command:

   $ oc create -f <filename>.yaml 1

   1

   <filename> is the name of the custom resource file.

1. In the OpenShift Container Platform web console, switch to the Administration → Custom Resource Definitions page.
2. On the Custom Resource Definitions page, click FeatureGate.
3. On the Custom Resource Definition Details page, click the Instances tab.
4. Click the cluster feature gate, then click the YAML tab.

5. Edit the cluster instance to add specific feature sets:

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample Feature Gate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   # ...
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute → Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

   # ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
   # ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

1. Edit the FeatureGate CR named cluster:

   $ oc edit featuregate cluster

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample FeatureGate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   # ...
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute → Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

   # ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
   # ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

Procedure

1. Modify the APIServer object:

   $ oc edit apiserver

2. Set the spec.encryption.type field to aesgcm or aescbc:

   spec:
     encryption:
       type: aesgcm 1

   1

   Set to aesgcm for AES-GCM encryption or aescbc for AES-CBC encryption.

3. Save the file to apply the changes.

   The encryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of the etcd database.

4. Verify that etcd encryption was successful.

   1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully encrypted:

      $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: routes.route.openshift.io

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

   2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully encrypted:

      $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: secrets, configmaps

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

   3. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully encrypted:

      $ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows EncryptionCompleted upon successful encryption:

      EncryptionCompleted
      All resources encrypted: oauthaccesstokens.oauth.openshift.io, oauthauthorizetokens.oauth.openshift.io

      If the output shows EncryptionInProgress, encryption is still in progress. Wait a few minutes and try again.

Procedure

1. Modify the APIServer object:

   $ oc edit apiserver

2. Set the encryption field type to identity:

   spec:
     encryption:
       type: identity 1

   1

   The identity type is the default value and means that no encryption is performed.

3. Save the file to apply the changes.

   The decryption process starts. It can take 20 minutes or longer for this process to complete, depending on the size of your cluster.

4. Verify that etcd decryption was successful.

   1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully decrypted:

      $ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

   2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully decrypted:

      $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

   3. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully decrypted:

      $ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'

      The output shows DecryptionCompleted upon successful decryption:

      DecryptionCompleted
      Encryption mode set to identity and everything is decrypted

      If the output shows DecryptionInProgress, decryption is still in progress. Wait a few minutes and try again.

Procedure

1. Start a debug session as root for a control plane node:

   $ oc debug --as-root node/<node_name>

2. Change your root directory to /host in the debug shell:

   sh-4.4# chroot /host

3. If the cluster-wide proxy is enabled, export the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables by running the following commands:

   $ export HTTP_PROXY=http://<your_proxy.example.com>:8080

   $ export HTTPS_PROXY=https://<your_proxy.example.com>:8080

   $ export NO_PROXY=<example.com>

4. Run the cluster-backup.sh script in the debug shell and pass in the location to save the backup to.

   Tip

   The cluster-backup.sh script is maintained as a component of the etcd Cluster Operator and is a wrapper around the etcdctl snapshot save command.

   sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

   Example script output

   found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6
   found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-7
   found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6
   found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3
   ede95fe6b88b87ba86a03c15e669fb4aa5bf0991c180d3c6895ce72eaade54a1
   etcdctl version: 3.4.14
   API version: 3.4
   {"level":"info","ts":1624647639.0188997,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db.part"}
   {"level":"info","ts":"2021-06-25T19:00:39.030Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
   {"level":"info","ts":1624647639.0301006,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://10.0.0.5:2379"}
   {"level":"info","ts":"2021-06-25T19:00:40.215Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
   {"level":"info","ts":1624647640.6032252,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://10.0.0.5:2379","size":"114 MB","took":1.584090459}
   {"level":"info","ts":1624647640.6047094,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-06-25_190035.db"}
   Snapshot saved at /home/core/assets/backup/snapshot_2021-06-25_190035.db
   {"hash":3866667823,"revision":31407,"totalKey":12828,"totalSize":114446336}
   snapshot db and kube resources are successfully saved to /home/core/assets/backup

   In this example, two files are created in the /home/core/assets/backup/ directory on the control plane host:

   • snapshot_<datetimestamp>.db: This file is the etcd snapshot. The cluster-backup.sh script confirms its validity.

   • static_kuberesources_<datetimestamp>.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.

     Note

     If etcd encryption is enabled, it is recommended to store this second file separately from the etcd snapshot for security reasons. However, this file is required to restore from the etcd snapshot.

     Keep in mind that etcd encryption only encrypts values, not keys. This means that resource types, namespaces, and object names are unencrypted.

Procedure

1. Select a control plane host to use as the recovery host. This is the host that you will run the restore operation on.

2. Establish SSH connectivity to each of the control plane nodes, including the recovery host.

   kube-apiserver becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.

   Important

   If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state.

3. Copy the etcd backup directory to the recovery control plane host.

   This procedure assumes that you copied the backup directory containing the etcd snapshot and the resources for the static pods to the /home/core/ directory of your recovery control plane host.

4. Stop the static pods on any other control plane nodes.

   Note

   You do not need to stop the static pods on the recovery host.

   1. Access a control plane host that is not the recovery host.

   2. Move the existing etcd pod file out of the kubelet manifest directory by running:

      $ sudo mv -v /etc/kubernetes/manifests/etcd-pod.yaml /tmp

   3. Verify that the etcd pods are stopped by using:

      $ sudo crictl ps | grep etcd | egrep -v "operator|etcd-guard"

      If the output of this command is not empty, wait a few minutes and check again.

   4. Move the existing kube-apiserver file out of the kubelet manifest directory by running:

      $ sudo mv -v /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp

   5. Verify that the kube-apiserver containers are stopped by running:

      $ sudo crictl ps | grep kube-apiserver | egrep -v "operator|guard"

      If the output of this command is not empty, wait a few minutes and check again.

   6. Move the existing kube-controller-manager file out of the kubelet manifest directory by using:

      $ sudo mv -v /etc/kubernetes/manifests/kube-controller-manager-pod.yaml /tmp

   7. Verify that the kube-controller-manager containers are stopped by running:

      $ sudo crictl ps | grep kube-controller-manager | egrep -v "operator|guard"

      If the output of this command is not empty, wait a few minutes and check again.

   8. Move the existing kube-scheduler file out of the kubelet manifest directory by using:

      $ sudo mv -v /etc/kubernetes/manifests/kube-scheduler-pod.yaml /tmp

   9. Verify that the kube-scheduler containers are stopped by using:

      $ sudo crictl ps | grep kube-scheduler | egrep -v "operator|guard"

      If the output of this command is not empty, wait a few minutes and check again.

   10. Move the etcd data directory to a different location with the following example:

       $ sudo mv -v /var/lib/etcd/ /tmp

   11. If the /etc/kubernetes/manifests/keepalived.yaml file exists and the node is deleted, follow these steps:

       1. Move the /etc/kubernetes/manifests/keepalived.yaml file out of the kubelet manifest directory:

          $ sudo mv -v /etc/kubernetes/manifests/keepalived.yaml /tmp

       2. Verify that any containers managed by the keepalived daemon are stopped:

          $ sudo crictl ps --name keepalived

          The output of this command should be empty. If it is not empty, wait a few minutes and check again.

       3. Check if the control plane has any Virtual IPs (VIPs) assigned to it:

          $ ip -o address | egrep '<api_vip>|<ingress_vip>'

       4. For each reported VIP, run the following command to remove it:

          $ sudo ip address del <reported_vip> dev <reported_vip_device>

   12. Repeat this step on each of the other control plane hosts that is not the recovery host.
5. Access the recovery control plane host.

6. If the keepalived daemon is in use, verify that the recovery control plane node owns the VIP:

   $ ip -o address | grep <api_vip>

   The address of the VIP is highlighted in the output if it exists. This command returns an empty string if the VIP is not set or configured incorrectly.

7. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

   Tip

   You can check whether the proxy is enabled by reviewing the output of oc get proxy cluster -o yaml. The proxy is enabled if the httpProxy, httpsProxy, and noProxy fields have values set.

8. Run the restore script on the recovery control plane host and pass in the path to the etcd backup directory:

   $ sudo -E /usr/local/bin/cluster-restore.sh /home/core/assets/backup

   Example script output

   ...stopping kube-scheduler-pod.yaml
   ...stopping kube-controller-manager-pod.yaml
   ...stopping etcd-pod.yaml
   ...stopping kube-apiserver-pod.yaml
   Waiting for container etcd to stop
   .complete
   Waiting for container etcdctl to stop
   .............................complete
   Waiting for container etcd-metrics to stop
   complete
   Waiting for container kube-controller-manager to stop
   complete
   Waiting for container kube-apiserver to stop
   ..........................................................................................complete
   Waiting for container kube-scheduler to stop
   complete
   Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
   starting restore-etcd static pod
   starting kube-apiserver-pod.yaml
   static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
   starting kube-controller-manager-pod.yaml
   static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
   starting kube-scheduler-pod.yaml
   static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml

   The cluster-restore.sh script must show that etcd, kube-apiserver, kube-controller-manager, and kube-scheduler pods are stopped and then started at the end of the restore process.

   Note

   The restore process can cause nodes to enter the NotReady state if the node certificates were updated after the last etcd backup.

9. Check the nodes to ensure they are in the Ready state.

   1. Run the following command:

      $ oc get nodes -w

      Sample output

      NAME                STATUS  ROLES          AGE     VERSION
      host-172-25-75-28   Ready   master         3d20h   v1.30.3
      host-172-25-75-38   Ready   infra,worker   3d20h   v1.30.3
      host-172-25-75-40   Ready   master         3d20h   v1.30.3
      host-172-25-75-65   Ready   master         3d20h   v1.30.3
      host-172-25-75-74   Ready   infra,worker   3d20h   v1.30.3
      host-172-25-75-79   Ready   worker         3d20h   v1.30.3
      host-172-25-75-86   Ready   worker         3d20h   v1.30.3
      host-172-25-75-98   Ready   infra,worker   3d20h   v1.30.3

      It can take several minutes for all nodes to report their state.

   2. If any nodes are in the NotReady state, log in to the nodes and remove all of the PEM files from the /var/lib/kubelet/pki directory on each node. You can SSH into the nodes or use the terminal window in the web console.

      $  ssh -i <ssh-key-path> core@<master-hostname>

      Sample pki directory

      sh-4.4# pwd
      /var/lib/kubelet/pki
      sh-4.4# ls
      kubelet-client-2022-04-28-11-24-09.pem  kubelet-server-2022-04-28-11-24-15.pem
      kubelet-client-current.pem              kubelet-server-current.pem

10. Restart the kubelet service on all control plane hosts.

    1. From the recovery host, run:

       $ sudo systemctl restart kubelet.service

    2. Repeat this step on all other control plane hosts.

11. Approve the pending Certificate Signing Requests (CSRs):

    Note

    Clusters with no worker nodes, such as single-node clusters or clusters consisting of three schedulable control plane nodes, will not have any pending CSRs to approve. You can skip all the commands listed in this step.

    1. Get the list of current CSRs by running:

       $ oc get csr

       Example output

       NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
       csr-2s94x   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending 1
       csr-4bd6t   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending 2
       csr-4hl85   13m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 3
       csr-zhhhp   3m8s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 4
       ...

       1 1 2

       A pending kubelet serving CSR, requested by the node for the kubelet serving endpoint.

       3 4

       A pending kubelet client CSR, requested with the node-bootstrapper node bootstrap credentials.

    2. Review the details of a CSR to verify that it is valid by running:

       $ oc describe csr <csr_name> 1

       1

       <csr_name> is the name of a CSR from the list of current CSRs.

    3. Approve each valid node-bootstrapper CSR by running:

       $ oc adm certificate approve <csr_name>

    4. For user-provisioned installations, approve each valid kubelet service CSR by running:

       $ oc adm certificate approve <csr_name>

12. Verify that the single member control plane has started successfully.

    1. From the recovery host, verify that the etcd container is running by using:

       $ sudo crictl ps | grep etcd | egrep -v "operator|etcd-guard"

       Example output

       3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0

    2. From the recovery host, verify that the etcd pod is running by using:

       $ oc -n openshift-etcd get pods -l k8s-app=etcd

       Example output

       NAME                                             READY   STATUS      RESTARTS   AGE
       etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s

       If the status is Pending, or the output lists more than one running etcd pod, wait a few minutes and check again.

13. If you are using the OVNKubernetes network plugin, you must restart ovnkube-controlplane pods.

    1. Delete all of the ovnkube-controlplane pods by running:

       $ oc -n openshift-ovn-kubernetes delete pod -l app=ovnkube-control-plane

    2. Verify that all of the ovnkube-controlplane pods were redeployed by using:

       $ oc -n openshift-ovn-kubernetes get pod -l app=ovnkube-control-plane

14. If you are using the OVN-Kubernetes network plugin, restart the Open Virtual Network (OVN) Kubernetes pods on all the nodes one by one. Use the following steps to restart OVN-Kubernetes pods on each node:

    Important

    Restart OVN-Kubernetes pods in the following order

    1. The recovery control plane host
    2. The other control plane hosts (if available)
    3. The other nodes
    Note

    Validating and mutating admission webhooks can reject pods. If you add any additional webhooks with the failurePolicy set to Fail, then they can reject pods and the restoration process can fail. You can avoid this by saving and deleting webhooks while restoring the cluster state. After the cluster state is restored successfully, you can enable the webhooks again.

    Alternatively, you can temporarily set the failurePolicy to Ignore while restoring the cluster state. After the cluster state is restored successfully, you can set the failurePolicy to Fail.

    1. Remove the northbound database (nbdb) and southbound database (sbdb). Access the recovery host and the remaining control plane nodes by using Secure Shell (SSH) and run:

       $ sudo rm -f /var/lib/ovn-ic/etc/*.db

    2. Restart the OpenVSwitch services. Access the node by using Secure Shell (SSH) and run the following command:

       $ sudo systemctl restart ovs-vswitchd ovsdb-server

    3. Delete the ovnkube-node pod on the node by running the following command, replacing <node> with the name of the node that you are restarting:

       $ oc -n openshift-ovn-kubernetes delete pod -l app=ovnkube-node --field-selector=spec.nodeName==<node>

    4. Verify that the ovnkube-node pod is running again with:

       $ oc -n openshift-ovn-kubernetes get pod -l app=ovnkube-node --field-selector=spec.nodeName==<node>

       Note

       It might take several minutes for the pods to restart.

15. Delete and re-create other non-recovery, control plane machines, one by one. After the machines are re-created, a new revision is forced and etcd automatically scales up.

    • If you use a user-provisioned bare metal installation, you can re-create a control plane machine by using the same method that you used to originally create it. For more information, see "Installing a user-provisioned cluster on bare metal".

      Warning

      Do not delete and re-create the machine for the recovery host.

    • If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps:

      Warning

      Do not delete and re-create the machine for the recovery host.

      For bare metal installations on installer-provisioned infrastructure, control plane machines are not re-created. For more information, see "Replacing a bare-metal control plane node".

      1. Obtain the machine for one of the lost control plane hosts.

         In a terminal that has access to the cluster as a cluster-admin user, run the following command:

         $ oc get machines -n openshift-machine-api -o wide

         Example output:

         NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
         clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped 1
         clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
         clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
         clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
         clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
         clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

         1

         This is the control plane machine for the lost control plane host, ip-10-0-131-183.ec2.internal.

      2. Delete the machine of the lost control plane host by running:

         $ oc delete machine -n openshift-machine-api clustername-8qw5l-master-0 1

         1

         Specify the name of the control plane machine for the lost control plane host.

         A new machine is automatically provisioned after deleting the machine of the lost control plane host.

      3. Verify that a new machine has been created by running:

         $ oc get machines -n openshift-machine-api -o wide

         Example output:

         NAME                                        PHASE          TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
         clustername-8qw5l-master-1                  Running        m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
         clustername-8qw5l-master-2                  Running        m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
         clustername-8qw5l-master-3                  Provisioning   m4.xlarge   us-east-1   us-east-1a   85s     ip-10-0-173-171.ec2.internal    aws:///us-east-1a/i-015b0888fe17bc2c8  running 1
         clustername-8qw5l-worker-us-east-1a-wbtgd   Running        m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
         clustername-8qw5l-worker-us-east-1b-lrdxb   Running        m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
         clustername-8qw5l-worker-us-east-1c-pkg26   Running        m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

         1

         The new machine, clustername-8qw5l-master-3 is being created and is ready after the phase changes from Provisioning to Running.

         It might take a few minutes for the new machine to be created. The etcd cluster Operator will automatically sync when the machine or node returns to a healthy state.

      4. Repeat these steps for each lost control plane host that is not the recovery host.

16. Turn off the quorum guard by entering:

    $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'

    This command ensures that you can successfully re-create secrets and roll out the static pods.

17. In a separate terminal window within the recovery host, export the recovery kubeconfig file by running:

    $ export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost-recovery.kubeconfig

18. Force etcd redeployment.

    In the same terminal window where you exported the recovery kubeconfig file, run:

    $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 1

    1

    The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

    The etcd redeployment starts.

    When the etcd cluster Operator performs a redeployment, the existing nodes are started with new pods similar to the initial bootstrap scale up.

19. Turn the quorum guard back on by entering:

    $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'

20. You can verify that the unsupportedConfigOverrides section is removed from the object by running:

    $ oc get etcd/cluster -oyaml

21. Verify all nodes are updated to the latest revision.

    In a terminal that has access to the cluster as a cluster-admin user, run:

    $ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

    Review the NodeInstallerProgressing status condition for etcd to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

    AllNodesAtLatestRevision
    3 nodes are at revision 7 1

    1

    In this example, the latest revision number is 7.

    If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

22. After etcd is redeployed, force new rollouts for the control plane. kube-apiserver will reinstall itself on the other nodes because the kubelet is connected to API servers using an internal load balancer.

    In a terminal that has access to the cluster as a cluster-admin user, run:

    1. Force a new rollout for kube-apiserver:

       $ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision.

       $ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    2. Force a new rollout for the Kubernetes controller manager by running the following command:

       $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision by running:

       $ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

    3. Force a new rollout for the kube-scheduler by running:

       $ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

       Verify all nodes are updated to the latest revision by using:

       $ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'

       Review the NodeInstallerProgressing status condition to verify that all nodes are at the latest revision. The output shows AllNodesAtLatestRevision upon successful update:

       AllNodesAtLatestRevision
       3 nodes are at revision 7 1

       1

       In this example, the latest revision number is 7.

       If the output includes multiple revision numbers, such as 2 nodes are at revision 6; 1 nodes are at revision 7, this means that the update is still in progress. Wait a few minutes and try again.

23. Monitor the platform Operators by running:

    $ oc adm wait-for-stable-cluster

    This process can take up to 15 minutes.

24. Verify that all control plane hosts have started and joined the cluster.

    In a terminal that has access to the cluster as a cluster-admin user, run the following command:

    $ oc -n openshift-etcd get pods -l k8s-app=etcd

    Example output

    etcd-ip-10-0-143-125.ec2.internal                2/2     Running     0          9h
    etcd-ip-10-0-154-194.ec2.internal                2/2     Running     0          9h
    etcd-ip-10-0-173-171.ec2.internal                2/2     Running     0          9h