Postinstallation configuration

Procedure

1. Edit the Image Registry Operator config object and set networkAccess.type to Internal:

   $ oc edit configs.imageregistry/cluster

   # ...
   spec:
     # ...
      storage:
         azure:
           # ...
           networkAccess:
             type: Internal
   # ...

2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

   $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

3. Optional: If the registry is exposed by a route, and you are configuring your storage account to be private, you must disable redirect if you want pulls external to the cluster to continue to work. Enter the following command to disable redirect on the Image Operator configuration:

   $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}'

   Note

   When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

1. Fetch the registry service name by running the following command:

   $ oc registry info --internal=true

   Example output

   image-registry.openshift-image-registry.svc:5000

2. Enter debug mode by running the following command:

   $ oc debug node/<node_name>

3. Run the suggested chroot command. For example:

   $ chroot /host

4. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

   Example output

   Login Succeeded!

5. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

   Example output

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Procedure

1. Edit the Image Registry Operator config object and configure the private endpoint using your VNet and subnet names:

   $ oc edit configs.imageregistry/cluster

   # ...
   spec:
     # ...
      storage:
         azure:
           # ...
           networkAccess:
             type: Internal
             internal:
               subnetName: <subnet_name>
               vnetName: <vnet_name>
               networkResourceGroupName: <network_resource_group_name>
   # ...

2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes.

   $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w

   Note

   When redirect is enabled, pulling images from outside of the cluster will not work.

Verification

1. Fetch the registry service name by running the following command:

   $ oc registry info --internal=true

   Example output

   image-registry.openshift-image-registry.svc:5000

2. Enter debug mode by running the following command:

   $ oc debug node/<node_name>

3. Run the suggested chroot command. For example:

   $ chroot /host

4. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

   Example output

   Login Succeeded!

5. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools

   Example output

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Verification

1. Fetch the registry service name by running the following command:

   $ oc registry info

   Example output

   default-route-openshift-image-registry.<cluster_dns>

2. Enter the following command to log in to your container registry:

   $ podman login --tls-verify=false -u unused -p $(oc whoami -t) default-route-openshift-image-registry.<cluster_dns>

   Example output

   Login Succeeded!

3. Enter the following command to verify that you can pull an image from the registry:

   $ podman pull --tls-verify=false default-route-openshift-image-registry.<cluster_dns>
   /openshift/tools

   Example output

   Trying to pull default-route-openshift-image-registry.<cluster_dns>/openshift/tools...
   Getting image source signatures
   Copying blob 6b245f040973 done
   Copying config 22667f5368 done
   Writing manifest to image destination
   Storing signatures
   22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Procedure

1. Log in to your Azure account:

   $ az login

2. Create a storage account and upload the arm64 virtual hard disk (VHD) to your storage account. The OpenShift Container Platform installation program creates a resource group, however, the boot image can also be uploaded to a custom named resource group:

   $ az storage account create -n ${STORAGE_ACCOUNT_NAME} -g ${RESOURCE_GROUP} -l westus --sku Standard_LRS 1

   1

   The westus object is an example region.

3. Create a storage container using the storage account you generated:

   $ az storage container create -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME}

4. You must use the OpenShift Container Platform installation program JSON file to extract the URL and aarch64 VHD name:

   1. Extract the URL field and set it to RHCOS_VHD_ORIGIN_URL as the file name by running the following command:

      $ RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".url')

   2. Extract the aarch64 VHD name and set it to BLOB_NAME as the file name by running the following command:

      $ BLOB_NAME=rhcos-$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64."rhel-coreos-extensions"."azure-disk".release')-azure.aarch64.vhd

5. Generate a shared access signature (SAS) token. Use this token to upload the RHCOS VHD to your storage container with the following commands:

   $ end=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`

   $ sas=`az storage container generate-sas -n ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --https-only --permissions dlrw --expiry $end -o tsv`

6. Copy the RHCOS VHD into the storage container:

   $ az storage blob copy start --account-name ${STORAGE_ACCOUNT_NAME} --sas-token "$sas" \
    --source-uri "${RHCOS_VHD_ORIGIN_URL}" \
    --destination-blob "${BLOB_NAME}" --destination-container ${CONTAINER_NAME}

   You can check the status of the copying process with the following command:

   $ az storage blob show -c ${CONTAINER_NAME} -n ${BLOB_NAME} --account-name ${STORAGE_ACCOUNT_NAME} | jq .properties.copy

   Example output

   {
    "completionTime": null,
    "destinationSnapshot": null,
    "id": "1fd97630-03ca-489a-8c4e-cfe839c9627d",
    "incrementalCopy": null,
    "progress": "17179869696/17179869696",
    "source": "https://rhcos.blob.core.windows.net/imagebucket/rhcos-411.86.202207130959-0-azure.aarch64.vhd",
    "status": "success", 1
    "statusDescription": null
   }

   1

   If the status parameter displays the success object, the copying process is complete.

7. Create an image gallery using the following command:

   $ az sig create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME}

   Use the image gallery to create an image definition. In the following example command, rhcos-arm64 is the name of the image definition.

   $ az sig image-definition create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --publisher RedHat --offer arm --sku arm64 --os-type linux --architecture Arm64 --hyper-v-generation V2

8. To get the URL of the VHD and set it to RHCOS_VHD_URL as the file name, run the following command:

   $ RHCOS_VHD_URL=$(az storage blob url --account-name ${STORAGE_ACCOUNT_NAME} -c ${CONTAINER_NAME} -n "${BLOB_NAME}" -o tsv)

9. Use the RHCOS_VHD_URL file, your storage account, resource group, and image gallery to create an image version. In the following example, 1.0.0 is the image version.

   $ az sig image-version create --resource-group ${RESOURCE_GROUP} --gallery-name ${GALLERY_NAME} --gallery-image-definition rhcos-arm64 --gallery-image-version 1.0.0 --os-vhd-storage-account ${STORAGE_ACCOUNT_NAME} --os-vhd-uri ${RHCOS_VHD_URL}

10. Your arm64 boot image is now generated. You can access the ID of your image with the following command:

    $ az sig image-version show -r $GALLERY_NAME -g $RESOURCE_GROUP -i rhcos-arm64 -e 1.0.0

    The following example image ID is used in the recourseID parameter of the compute machine set:

    Example resourceID

    /resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Compute/galleries/${GALLERY_NAME}/images/rhcos-arm64/versions/1.0.0

Verification

1. Verify that the new ARM64 machines are running by entering the following command:

   $ oc get machineset -n openshift-machine-api

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-arm64-machine-set-0                   2        2      2          2  10m

2. You can check that the nodes are ready and scheduable with the following command:

   $ oc get nodes

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Verification

1. View the list of compute machine sets by entering the following command:

   $ oc get machineset -n openshift-machine-api

   You can then see your created ARM64 machine set.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-aws-arm64-machine-set-0                   2        2      2          2  10m

2. You can check that the nodes are ready and scheduable with the following command:

   $ oc get nodes

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Verification

1. View the list of compute machine sets by entering the following command:

   $ oc get machineset -n openshift-machine-api

   You can then see your created ARM64 machine set.

   Example output

   NAME                                                DESIRED  CURRENT  READY  AVAILABLE  AGE
   <infrastructure_id>-gcp-arm64-machine-set-0                   2        2      2          2  10m

2. You can check that the nodes are ready and scheduable with the following command:

   $ oc get nodes

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + aarch64):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on aarch64:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.28.5
   master-1  Ready     master  63m  v1.28.5
   master-2  Ready     master  64m  v1.28.5

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.28.5
   master-1  Ready     master  73m  v1.28.5
   master-2  Ready     master  74m  v1.28.5
   worker-0  Ready     worker  11m  v1.28.5
   worker-1  Ready     worker  11m  v1.28.5

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Procedure

1. Disable UDP aggregation.

   Currently, UDP aggregation is not supported on IBM Z® and is not automatically deactivated on multi-architecture compute clusters with an x86_64 control plane and additional s390x compute machines. To ensure that the addtional compute nodes are added to the cluster correctly, you must manually disable UDP aggregation.

   1. Create a YAML file udp-aggregation-config.yaml with the following content:

      apiVersion: v1
      kind: ConfigMap
      data:
        disable-udp-aggregation: "true"
      metadata:
        name: udp-aggregation-config
        namespace: openshift-network-operator

   2. Create the ConfigMap resource by running the following command:

      $ oc create -f udp-aggregation-config.yaml

2. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

3. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

4. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

5. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

6. Move the downloaded RHEL live kernel, initramfs, and rootfs files to an HTTP or HTTPS server that is accessible from the z/VM guest you want to add.

7. Create a parameter file for the z/VM guest. The following parameters are specific for the virtual machine:

   • Optional: To specify a static IP address, add an ip= parameter with the following entries, with each separated by a colon:

     1. The IP address for the machine.
     2. An empty string.
     3. The gateway.
     4. The netmask.
     5. The machine host and domain name in the form hostname.domainname. Omit this value to let RHCOS decide.
     6. The network interface name. Omit this value to let RHCOS decide.
     7. The value none.
   • For coreos.inst.ignition_url=, specify the URL to the worker.ign file. Only HTTP and HTTPS protocols are supported.
   • For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   • For installations on DASD-type disks, complete the following tasks:

     1. For coreos.inst.install_dev=, specify /dev/dasda.
     2. Use rd.dasd= to specify the DASD where RHCOS is to be installed.

     3. Leave all other parameters unchanged.

        The following is an example parameter file, additional-worker-dasd.parm:

        rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/dasda \
        coreos.live.rootfs_url=http://cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img \
        coreos.inst.ignition_url=http://cl1.provide.example.com:8080/ignition/worker.ign \
        ip=172.18.78.2::172.18.78.1:255.255.255.0:::none nameserver=172.18.78.1 \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.dasd=0.0.3490

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

   • For installations on FCP-type disks, complete the following tasks:

     1. Use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. For multipathing, repeat this step for each additional path.

        Note

        When you install with multiple paths, you must enable multipathing directly after the installation, not at a later point in time, as this can cause problems.

     2. Set the install device as: coreos.inst.install_dev=/dev/sda.

        Note

        If additional LUNs are configured with NPIV, FCP requires zfcp.allow_lun_scan=0. If you must enable zfcp.allow_lun_scan=1 because you use a CSI driver, for example, you must configure your NPIV so that each node cannot access the boot partition of another node.

     3. Leave all other parameters unchanged.

        Important

        Additional postinstallation steps are required to fully enable multipathing. For more information, see “Enabling multipathing with kernel arguments on RHCOS" in Postinstallation machine configuration tasks.

        The following is an example parameter file, additional-worker-fcp.parm for a worker node with multipathing:

        rd.neednet=1 \
        console=ttysclp0 \
        coreos.inst.install_dev=/dev/sda \
        coreos.live.rootfs_url=http://cl1.provide.example.com:8080/assets/rhcos-live-rootfs.s390x.img \
        coreos.inst.ignition_url=http://cl1.provide.example.com:8080/ignition/worker.ign \
        ip=172.18.78.2::172.18.78.1:255.255.255.0:::none nameserver=172.18.78.1 \
        rd.znet=qeth,0.0.bdf0,0.0.bdf1,0.0.bdf2,layer2=1,portno=0 \
        zfcp.allow_lun_scan=0 \
        rd.zfcp=0.0.1987,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763070bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.1987,0x50050763071bc5e3,0x4008400B00000000 \
        rd.zfcp=0.0.19C7,0x50050763071bc5e3,0x4008400B00000000

        Write all options in the parameter file as a single line and make sure that you have no newline characters.

8. Transfer the initramfs, kernel, parameter files, and RHCOS images to z/VM, for example, by using FTP. For details about how to transfer the files with FTP and boot from the virtual reader, see Installing under Z/VM.

9. Punch the files to the virtual reader of the z/VM guest virtual machine.

   See PUNCH in IBM® Documentation.

   Tip

   You can use the CP PUNCH command or, if you use Linux, the vmur command to transfer files between two z/VM guest virtual machines.

10. Log in to CMS on the bootstrap machine.

11. IPL the bootstrap machine from the reader by running the following command:

    $ ipl c

    See IPL in IBM® Documentation.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.28.5
   master-1  Ready     master  63m  v1.28.5
   master-2  Ready     master  64m  v1.28.5

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.28.5
   master-1  Ready     master  73m  v1.28.5
   master-2  Ready     master  74m  v1.28.5
   worker-0  Ready     worker  11m  v1.28.5
   worker-1  Ready     worker  11m  v1.28.5

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Procedure

1. Disable UDP aggregation.

   Currently, UDP aggregation is not supported on IBM Z® and is not automatically deactivated on multi-architecture compute clusters with an x86_64 control plane and additional s390x compute machines. To ensure that the addtional compute nodes are added to the cluster correctly, you must manually disable UDP aggregation.

   1. Create a YAML file udp-aggregation-config.yaml with the following content:

      apiVersion: v1
      kind: ConfigMap
      data:
        disable-udp-aggregation: "true"
      metadata:
        name: udp-aggregation-config
        namespace: openshift-network-operator

   2. Create the ConfigMap resource by running the following command:

      $ oc create -f udp-aggregation-config.yaml

2. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

3. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URL of this file.

4. You can validate that the Ignition file is available on the URL. The following example gets the Ignition config file for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

5. Download the RHEL live kernel, initramfs, and rootfs files by running the following commands:

    $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.kernel.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.initramfs.location')

   $ curl -LO $(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' \
   | jq -r '.architectures.s390x.artifacts.metal.formats.pxe.rootfs.location')

6. Move the downloaded RHEL live kernel, initramfs and rootfs files to an HTTP or HTTPS server before you launch virt-install.

7. Create the new KVM guest nodes using the RHEL kernel, initramfs, and Ignition files; the new disk image; and adjusted parm line arguments.

   $ virt-install \
      --connect qemu:///system \
      --name <vm_name> \
      --autostart \
      --os-variant rhel9.2 \ 1
      --cpu host \
      --vcpus <vcpus> \
      --memory <memory_mb> \
      --disk <vm_name>.qcow2,size=<image_size> \
      --network network=<virt_network_parm> \
      --location <media_location>,kernel=<rhcos_kernel>,initrd=<rhcos_initrd> \ 2
      --extra-args "rd.neednet=1" \
      --extra-args "coreos.inst.install_dev=/dev/vda" \
      --extra-args "coreos.inst.ignition_url=<worker_ign>" \ 3
      --extra-args "coreos.live.rootfs_url=<rhcos_rootfs>" \ 4
      --extra-args "ip=<ip>::<default_gateway>:<subnet_mask_length>:<hostname>::none:<MTU>" \ 5
      --extra-args "nameserver=<dns>" \
      --extra-args "console=ttysclp0" \
      --noautoconsole \
      --wait

   1

   For os-variant, specify the RHEL version for the RHCOS compute machine. rhel9.2 is the recommended version. To query the supported RHEL version of your operating system, run the following command:

   $ osinfo-query os -f short-id

   Note

   The os-variant is case sensitive.

   2

   For --location, specify the location of the kernel/initrd on the HTTP or HTTPS server.

   3

   For coreos.inst.ignition_url=, specify the worker.ign Ignition file for the machine role. Only HTTP and HTTPS protocols are supported.

   4

   For coreos.live.rootfs_url=, specify the matching rootfs artifact for the kernel and initramfs you are booting. Only HTTP and HTTPS protocols are supported.

   5

   Optional: For hostname, specify the fully qualified hostname of the client machine.

   Note

   If you are using HAProxy as a load balancer, update your HAProxy rules for ingress-router-443 and ingress-router-80 in the /etc/haproxy/haproxy.cfg configuration file.

8. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.28.5
   master-1  Ready     master  63m  v1.28.5
   master-2  Ready     master  64m  v1.28.5

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.28.5
   master-1  Ready     master  73m  v1.28.5
   master-2  Ready     master  74m  v1.28.5
   worker-0  Ready     worker  11m  v1.28.5
   worker-1  Ready     worker  11m  v1.28.5

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Verification

1. If you see the following output, then your cluster is using the multi-architecture payload:

   {
    "release.openshift.io/architecture": "multi",
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   You can then begin adding multi-arch compute nodes to your cluster.

2. If you see the following output, then your cluster is not using the multi-architecture payload:

   {
    "url": "https://access.redhat.com/errata/<errata_version>"
   }

   Important

   To migrate your cluster so the cluster supports multi-architecture compute machines, follow the procedure in Migrating to a cluster with multi-architecture compute machines.

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + ppc64le):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on ppc64le architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on ppc64le:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.28.5
   master-1  Ready     master  63m  v1.28.5
   master-2  Ready     master  64m  v1.28.5

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes -o wide

   Example output

   NAME               STATUS   ROLES                  AGE   VERSION           INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                  CONTAINER-RUNTIME
   worker-0-ppc64le   Ready    worker                 42d   v1.28.2+e3ba6d9   192.168.200.21   <none>        Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.ppc64le   cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   worker-1-ppc64le   Ready    worker                 42d   v1.28.2+e3ba6d9   192.168.200.20   <none>        Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.ppc64le   cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   master-0-x86       Ready    control-plane,master   75d   v1.28.2+e3ba6d9   10.248.0.38      10.248.0.38   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   master-1-x86       Ready    control-plane,master   75d   v1.28.2+e3ba6d9   10.248.0.39      10.248.0.39   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   master-2-x86       Ready    control-plane,master   75d   v1.28.2+e3ba6d9   10.248.0.40      10.248.0.40   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   worker-0-x86       Ready    worker                 75d   v1.28.2+e3ba6d9   10.248.0.43      10.248.0.43   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9
   worker-1-x86       Ready    worker                 75d   v1.28.2+e3ba6d9   10.248.0.44      10.248.0.44   Red Hat Enterprise Linux CoreOS 415.92.202309261919-0 (Plow)   5.14.0-284.34.1.el9_2.x86_64    cri-o://1.28.1-3.rhaos4.15.gitb36169e.el9

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Label the nodes where you want to run the 64k page size kernel:

   $ oc label node <node_name> <label>

   Example command

   $ oc label node worker-arm64-01 node-role.kubernetes.io/worker-64k-pages=

2. Create a machine config pool that contains the worker role that uses the ARM64 architecture and the worker-64k-pages role:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-64k-pages
   spec:
     machineConfigSelector:
       matchExpressions:
         - key: machineconfiguration.openshift.io/role
           operator: In
           values:
           - worker
           - worker-64k-pages
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-64k-pages: ""
         kubernetes.io/arch: arm64

3. Create a machine config on your compute node to enable 64k-pages with the 64k-pages parameter.

   $ oc create -f <filename>.yaml

   Example MachineConfig

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: "worker-64k-pages" 1
     name: 99-worker-64kpages
   spec:
     kernelType: 64k-pages 2

   1

   Specify the value of the machineconfiguration.openshift.io/role label in the custom machine config pool. The example MachineConfig uses the worker-64k-pages label to enable 64k pages in the worker-64k-pages pool.

   2

   Specify your desired kernel type. Valid values are 64k-pages and default

   Note

   The 64k-pages type is supported on only 64-bit ARM architecture based compute nodes. The realtime type is supported on only 64-bit x86 architecture based compute nodes.

1. Cordon. The MCO marks the node as not schedulable for additional workloads.
2. Drain. The MCO terminates all running workloads on the node, causing the workloads to be rescheduled onto other nodes.
3. Apply. The MCO writes the new configuration to the nodes as needed.
4. Reboot. The MCO restarts the node.
5. Uncordon. The MCO marks the node as schedulable for workloads.

Procedure

1. To see the number of MCO-managed nodes available on your cluster for each machine config pool (MCP), run the following command:

   $ oc get machineconfigpool

   Example output

   NAME      CONFIG                    UPDATED  UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT  AGE
   master    rendered-master-06c9c4…   True     False      False     3             3                  3                   0                     4h42m
   worker    rendered-worker-f4b64…    False    True       False     3             2                  2                   0                     4h42m

   where:

   UPDATED

   The True status indicates that the MCO has applied the current machine config to the nodes in that MCP. The current machine config is specified in the STATUS field in the oc get mcp output. The False status indicates a node in the MCP is updating.

   UPDATING

   The True status indicates that the MCO is applying the desired machine config, as specified in the MachineConfigPool custom resource, to at least one of the nodes in that MCP. The desired machine config is the new, edited machine config. Nodes that are updating might not be available for scheduling. The False status indicates that all nodes in the MCP are updated.

   DEGRADED

   A True status indicates the MCO is blocked from applying the current or desired machine config to at least one of the nodes in that MCP, or the configuration is failing. Nodes that are degraded might not be available for scheduling. A False status indicates that all nodes in the MCP are ready.

   MACHINECOUNT

   Indicates the total number of machines in that MCP.

   READYMACHINECOUNT

   Indicates the total number of machines in that MCP that are ready for scheduling.

   UPDATEDMACHINECOUNT

   Indicates the total number of machines in that MCP that have the current machine config.

   DEGRADEDMACHINECOUNT

   Indicates the total number of machines in that MCP that are marked as degraded or unreconcilable.

   In the previous output, there are three control plane (master) nodes and three worker nodes. The control plane MCP and the associated nodes are updated to the current machine config. The nodes in the worker MCP are being updated to the desired machine config. Two of the nodes in the worker MCP are updated and one is still updating, as indicated by the UPDATEDMACHINECOUNT being 2. There are no issues, as indicated by the DEGRADEDMACHINECOUNT being 0 and DEGRADED being False.

   While the nodes in the MCP are updating, the machine config listed under CONFIG is the current machine config, which the MCP is being updated from. When the update is complete, the listed machine config is the desired machine config, which the MCP was updated to.

   Note

   If a node is being cordoned, that node is not included in the READYMACHINECOUNT, but is included in the MACHINECOUNT. Also, the MCP status is set to UPDATING. Because the node has the current machine config, it is counted in the UPDATEDMACHINECOUNT total:

   Example output

   NAME      CONFIG                    UPDATED  UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT  AGE
   master    rendered-master-06c9c4…   True     False      False     3             3                  3                   0                     4h42m
   worker    rendered-worker-c1b41a…   False    True       False     3             2                  3                   0                     4h42m

2. To check the status of the nodes in an MCP by examining the MachineConfigPool custom resource, run the following command: :

   $ oc describe mcp worker

   Example output

   ...
     Degraded Machine Count:     0
     Machine Count:              3
     Observed Generation:        2
     Ready Machine Count:        3
     Unavailable Machine Count:  0
     Updated Machine Count:      3
   Events:                       <none>

   Note

   If a node is being cordoned, the node is not included in the Ready Machine Count. It is included in the Unavailable Machine Count:

   Example output

   ...
     Degraded Machine Count:     0
     Machine Count:              3
     Observed Generation:        2
     Ready Machine Count:        2
     Unavailable Machine Count:  1
     Updated Machine Count:      3

3. To see each existing MachineConfig object, run the following command:

   $ oc get machineconfigs

   Example output

   NAME                             GENERATEDBYCONTROLLER          IGNITIONVERSION  AGE
   00-master                        2c9371fbb673b97a6fe8b1c52...   3.2.0            5h18m
   00-worker                        2c9371fbb673b97a6fe8b1c52...   3.2.0            5h18m
   01-master-container-runtime      2c9371fbb673b97a6fe8b1c52...   3.2.0            5h18m
   01-master-kubelet                2c9371fbb673b97a6fe8b1c52…     3.2.0            5h18m
   ...
   rendered-master-dde...           2c9371fbb673b97a6fe8b1c52...   3.2.0            5h18m
   rendered-worker-fde...           2c9371fbb673b97a6fe8b1c52...   3.2.0            5h18m

   Note that the MachineConfig objects listed as rendered are not meant to be changed or deleted.

4. To view the contents of a particular machine config (in this case, 01-master-kubelet), run the following command:

   $ oc describe machineconfigs 01-master-kubelet

   The output from the command shows that this MachineConfig object contains both configuration files (cloud.conf and kubelet.conf) and a systemd service (Kubernetes Kubelet):

   Example output

   Name:         01-master-kubelet
   ...
   Spec:
     Config:
       Ignition:
         Version:  3.2.0
       Storage:
         Files:
           Contents:
             Source:   data:,
           Mode:       420
           Overwrite:  true
           Path:       /etc/kubernetes/cloud.conf
           Contents:
             Source:   data:,kind%3A%20KubeletConfiguration%0AapiVersion%3A%20kubelet.config.k8s.io%2Fv1beta1%0Aauthentication%3A%0A%20%20x509%3A%0A%20%20%20%20clientCAFile%3A%20%2Fetc%2Fkubernetes%2Fkubelet-ca.crt%0A%20%20anonymous...
           Mode:       420
           Overwrite:  true
           Path:       /etc/kubernetes/kubelet.conf
       Systemd:
         Units:
           Contents:  [Unit]
   Description=Kubernetes Kubelet
   Wants=rpc-statd.service network-online.target crio.service
   After=network-online.target crio.service
   
   ExecStart=/usr/bin/hyperkube \
       kubelet \
         --config=/etc/kubernetes/kubelet.conf \ ...

Procedure

1. Get a summary of update statuses for all nodes in all machine config pools by running the following command:

   $ oc get machineconfignodes

   Example output

   NAME                          UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETED   UPDATECOMPLETED   RESUMED
   ip-10-0-12-194.ec2.internal   True      False             False              False                    False              False
   ip-10-0-17-102.ec2.internal   False     True              False              False                    False              False
   ip-10-0-2-232.ec2.internal    False     False             True               False                    False              False
   ip-10-0-59-251.ec2.internal   False     False             False              True                     False              False
   ip-10-0-59-56.ec2.internal    False     False             False              False                    True               True
   ip-10-0-6-214.ec2.internal    False     False             Unknown            False                    False              False

   where:

   UPDATED

   The True status indicates that the MCO has applied the current machine config to that particular node. The False status indicates that the node is currently updating. The Unknown status means the operation is processing.

   UPDATEPREPARED

   The False status indicates that the MCO has not started reconciling the new machine configs to be distributed. The True status indicates that the MCO has completed this phase of the update. The Unknown status means the operation is processing.

   UPDATEEXECUTED

   The False status indicates that the MCO has not started cordoning and draining the node. It also indicates that the disk state and operating system have not started updating. The True status indicates that the MCO has completed this phase of the update. The Unknown status means the operation is processing.

   UPDATEPOSTACTIONCOMPLETED

   The False status indicates that the MCO has not started rebooting the node or closing the daemon. The True status indicates that the MCO has completed reboot and updating the node status. The Unknown status indicates either that an error has occurred during the update process at this phase, or that the MCO is currently applying the update.

   UPDATECOMPLETED

   The False status indicates that the MCO has not started uncordoning the node and updating the node state and metrics. The True status indicates that the MCO has finished updating the node state and available metrics.

   RESUMED

   The False status indicates that the MCO has not started the config drift monitor. The True status indicates that the node has resumed operation. The Unknown status means the operation is processing.

   Note

   Within the primary phases previously described, you can have secondary phases which you can use to see the update progression in more detail. You can get more information that includes secondary phases of updates by using the -o wide option of the preceding command. This provides the additional UPDATECOMPATIBLE, UPDATEFILESANDOS, DRAINEDNODE, CORDONEDNODE, REBOOTNODE, RELOADEDCRIO and UNCORDONED columns. These secondary phases do not always occur and depend on the type of update you want to apply.

2. Check the update status of nodes in a specific machine config pool by running the following command:

   $ oc get machineconfignodes $(oc get machineconfignodes -o json | jq -r '.items[]|select(.spec.pool.name=="<pool_name>")|.metadata.name') 1

   1

   The name of the pool is the MachineConfigPool object name.

   Example output

   NAME                          UPDATED   UPDATEPREPARED   UPDATEEXECUTED   UPDATEPOSTACTIONCOMPLETE   UPDATECOMPLETE   RESUMED
   ip-10-0-48-226.ec2.internal   True      False            False            False                      False            False
   ip-10-0-5-241.ec2.internal    True      False            False            False                      False            False
   ip-10-0-74-108.ec2.internal   True      False            False            False                      False            False

3. Check the update status of an individual node by running the following command:

   $ oc describe machineconfignode/<node_name> 1

   1

   The name of the node is the MachineConfigNode object name.

   Example output

   Name:         <node_name>
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1alpha1
   Kind:         MachineConfigNode
   Metadata:
     Creation Timestamp:  2023-10-17T13:08:58Z
     Generation:          1
     Resource Version:    49443
     UID:                 4bd758ab-2187-413c-ac42-882e61761b1d
   Spec:
     Node Ref:
       Name:         <node_name>
     Pool:
       Name:         master
     ConfigVersion:
       Desired: rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b 1
   Status:
     Conditions:
       Last Transition Time:  2023-10-17T13:09:02Z
       Message:               Node has completed update to config rendered-master-cf99e619747ab19165f11e3546c71f1e
       Reason:                NodeUpgradeComplete
       Status:                True
       Type:                  Updated
       Last Transition Time:  2023-10-17T13:09:02Z
       Message:               This node has not yet entered the UpdatePreparing phase
       Reason:                NotYetOccured
       Status:                False
     Config Version:
       Current:            rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b
       Desired:            rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b 2
     Health:               Healthy
     Most Recent Error:
     Observed Generation:  3

   1

   The desired configuration specified in the spec.configversion.desired field updates immediately when a new configuration is detected on the node.

   2

   The desired configuration specified in the status.configversion.desired field updates only when the new configuration is validated by the Machine Config Daemon (MCD). The MCD performs validation by checking the current phase of the update. If the update successfully passes the UPDATEPREPARED phase, then the status adds the new configuration.

Procedure

1. Create a Butane config including the contents of the chrony.conf file. For example, to configure chrony on worker nodes, create a 99-worker-chrony.bu file.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.15.0
   metadata:
     name: 99-worker-chrony 1
     labels:
       machineconfiguration.openshift.io/role: worker 2
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644 3
       overwrite: true
       contents:
         inline: |
           pool 0.rhel.pool.ntp.org iburst 4
           driftfile /var/lib/chrony/drift
           makestep 1.0 3
           rtcsync
           logdir /var/log/chrony

   1 2

   On control plane nodes, substitute master for worker in both of these locations.

   3

   Specify an octal value mode for the mode field in the machine config file. After creating the file and applying the changes, the mode is converted to a decimal value. You can check the YAML file with the command oc get mc <mc-name> -o yaml.

   4

   Specify any valid, reachable time source, such as the one provided by your DHCP server. Alternately, you can specify any of the following NTP servers: 1.rhel.pool.ntp.org, 2.rhel.pool.ntp.org, or 3.rhel.pool.ntp.org.

2. Use Butane to generate a MachineConfig object file, 99-worker-chrony.yaml, containing the configuration to be delivered to the nodes:

   $ butane 99-worker-chrony.bu -o 99-worker-chrony.yaml

3. Apply the configurations in one of two ways:

   • If the cluster is not running yet, after you generate manifest files, add the MachineConfig object file to the <installation_directory>/openshift directory, and then continue to create the cluster.

   • If the cluster is already running, apply the file:

     $ oc apply -f ./99-worker-chrony.yaml

Procedure

1. Create the MachineConfig CR that disables chronyd for the specified node role.

   1. Save the following YAML in the disable-chronyd.yaml file:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: <node_role> 1
        name: disable-chronyd
      spec:
        config:
          ignition:
            version: 3.2.0
          systemd:
            units:
              - contents: |
                  [Unit]
                  Description=NTP client/server
                  Documentation=man:chronyd(8) man:chrony.conf(5)
                  After=ntpdate.service sntp.service ntpd.service
                  Conflicts=ntpd.service systemd-timesyncd.service
                  ConditionCapability=CAP_SYS_TIME
                  [Service]
                  Type=forking
                  PIDFile=/run/chrony/chronyd.pid
                  EnvironmentFile=-/etc/sysconfig/chronyd
                  ExecStart=/usr/sbin/chronyd $OPTIONS
                  ExecStartPost=/usr/libexec/chrony-helper update-daemon
                  PrivateTmp=yes
                  ProtectHome=yes
                  ProtectSystem=full
                  [Install]
                  WantedBy=multi-user.target
                enabled: false
                name: "chronyd.service"

      1

      Node role where you want to disable chronyd, for example, master.

   2. Create the MachineConfig CR by running the following command:

      $ oc create -f disable-chronyd.yaml

Procedure

1. List existing MachineConfig objects for your OpenShift Container Platform cluster to determine how to label your machine config:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

2. Create a MachineConfig object file that identifies the kernel argument (for example, 05-worker-kernelarg-selinuxpermissive.yaml)

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker1
     name: 05-worker-kernelarg-selinuxpermissive2
   spec:
     kernelArguments:
       - enforcing=03

   1

   Applies the new kernel argument only to worker nodes.

   2

   Named to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode).

   3

   Identifies the exact kernel argument as enforcing=0.

3. Create the new machine config:

   $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml

4. Check the machine configs to see that the new one was added:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   05-worker-kernelarg-selinuxpermissive                                                         3.2.0             105s
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

5. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.28.5
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.28.5
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.28.5
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.28.5
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.28.5
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.28.5

   You can see that scheduling on each worker node is disabled as the change is being applied.

6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   BOOT_IMAGE=/ostree/rhcos-... console=tty0 console=ttyS0,115200n8
   rootflags=defaults,prjquota rw root=UUID=fd0... ostree=/ostree/boot.0/rhcos/16...
   coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2 enforcing=0
   
   sh-4.2# exit

   You should see the enforcing=0 argument added to the other kernel arguments.

Procedure

1. To enable multipathing postinstallation on control plane nodes:

   • Create a machine config file, such as 99-master-kargs-mpath.yaml, that instructs the cluster to add the master label and that identifies the multipath kernel argument, for example:

     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: "master"
       name: 99-master-kargs-mpath
     spec:
       kernelArguments:
         - 'rd.multipath=default'
         - 'root=/dev/disk/by-label/dm-mpath-root'

2. To enable multipathing postinstallation on worker nodes:

   • Create a machine config file, such as 99-worker-kargs-mpath.yaml, that instructs the cluster to add the worker label and that identifies the multipath kernel argument, for example:

     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: "worker"
       name: 99-worker-kargs-mpath
     spec:
       kernelArguments:
         - 'rd.multipath=default'
         - 'root=/dev/disk/by-label/dm-mpath-root'

3. Create the new machine config by using either the master or worker YAML file you previously created:

   $ oc create -f ./99-worker-kargs-mpath.yaml

4. Check the machine configs to see that the new one was added:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-kargs-mpath                              52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             105s
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

5. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.28.5
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.28.5
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.28.5
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.28.5
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.28.5
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.28.5

   You can see that scheduling on each worker node is disabled as the change is being applied.

6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   ...
   rd.multipath=default root=/dev/disk/by-label/dm-mpath-root
   ...
   
   sh-4.2# exit

   You should see the added kernel arguments.

Procedure

1. Create a machine config for the real-time kernel: Create a YAML file (for example, 99-worker-realtime.yaml) that contains a MachineConfig object for the realtime kernel type. This example tells the cluster to use a real-time kernel for all worker nodes:

   $ cat << EOF > 99-worker-realtime.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: "worker"
     name: 99-worker-realtime
   spec:
     kernelType: realtime
   EOF

2. Add the machine config to the cluster. Type the following to add the machine config to the cluster:

   $ oc create -f 99-worker-realtime.yaml

3. Check the real-time kernel: Once each impacted node reboots, log in to the cluster and run the following commands to make sure that the real-time kernel has replaced the regular kernel for the set of nodes you configured:

   $ oc get nodes

   Example output

   NAME                                        STATUS  ROLES    AGE   VERSION
   ip-10-0-143-147.us-east-2.compute.internal  Ready   worker   103m  v1.28.5
   ip-10-0-146-92.us-east-2.compute.internal   Ready   worker   101m  v1.28.5
   ip-10-0-169-2.us-east-2.compute.internal    Ready   worker   102m  v1.28.5

   $ oc debug node/ip-10-0-143-147.us-east-2.compute.internal

   Example output

   Starting pod/ip-10-0-143-147us-east-2computeinternal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.4# uname -a
   Linux <worker_node> 4.18.0-147.3.1.rt24.96.el8_1.x86_64 #1 SMP PREEMPT RT
           Wed Nov 27 18:29:55 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

   The kernel name contains rt and text “PREEMPT RT” indicates that this is a real-time kernel.

4. To go back to the regular kernel, delete the MachineConfig object:

   $ oc delete -f 99-worker-realtime.yaml

Procedure

1. Create a Butane config file, 40-worker-custom-journald.bu, that includes an /etc/systemd/journald.conf file with the required settings.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.15.0
   metadata:
     name: 40-worker-custom-journald
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
     - path: /etc/systemd/journald.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           # Disable rate limiting
           RateLimitInterval=1s
           RateLimitBurst=10000
           Storage=volatile
           Compress=no
           MaxRetentionSec=30s

2. Use Butane to generate a MachineConfig object file, 40-worker-custom-journald.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml

3. Apply the machine config to the pool:

   $ oc apply -f 40-worker-custom-journald.yaml

4. Check that the new machine config is applied and that the nodes are not in a degraded state. It might take a few minutes. The worker pool will show the updates in progress, as each node successfully has the new machine config applied:

   $ oc get machineconfigpool
   NAME   CONFIG             UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
   master rendered-master-35 True    False    False    3            3                 3                   0                    34m
   worker rendered-worker-d8 False   True     False    3            1                 1                   0                    34m

5. To check that the change was applied, you can log in to a worker node:

   $ oc get node | grep worker
   ip-10-0-0-1.us-east-2.compute.internal   Ready    worker   39m   v0.0.0-master+$Format:%h$
   $ oc debug node/ip-10-0-0-1.us-east-2.compute.internal
   Starting pod/ip-10-0-141-142us-east-2computeinternal-debug ...
   ...
   sh-4.2# chroot /host
   sh-4.4# cat /etc/systemd/journald.conf
   # Disable rate limiting
   RateLimitInterval=1s
   RateLimitBurst=10000
   Storage=volatile
   Compress=no
   MaxRetentionSec=30s
   sh-4.4# exit

Procedure

1. Create a machine config for extensions: Create a YAML file (for example, 80-extensions.yaml) that contains a MachineConfig extensions object. This example tells the cluster to add the usbguard extension.

   $ cat << EOF > 80-extensions.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 80-worker-extensions
   spec:
     config:
       ignition:
         version: 3.2.0
     extensions:
       - usbguard
   EOF

2. Add the machine config to the cluster. Type the following to add the machine config to the cluster:

   $ oc create -f 80-extensions.yaml

   This sets all worker nodes to have rpm packages for usbguard installed.

3. Check that the extensions were applied:

   $ oc get machineconfig 80-worker-extensions

   Example output

   NAME                 GENERATEDBYCONTROLLER IGNITIONVERSION AGE
   80-worker-extensions                       3.2.0           57s

4. Check that the new machine config is now applied and that the nodes are not in a degraded state. It may take a few minutes. The worker pool will show the updates in progress, as each machine successfully has the new machine config applied:

   $ oc get machineconfigpool

   Example output

   NAME   CONFIG             UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
   master rendered-master-35 True    False    False    3            3                 3                   0                    34m
   worker rendered-worker-d8 False   True     False    3            1                 1                   0                    34m

5. Check the extensions. To check that the extension was applied, run:

   $ oc get node | grep worker

   Example output

   NAME                                        STATUS  ROLES    AGE   VERSION
   ip-10-0-169-2.us-east-2.compute.internal    Ready   worker   102m  v1.28.5

   $ oc debug node/ip-10-0-169-2.us-east-2.compute.internal

   Example output

   ...
   To use host binaries, run `chroot /host`
   sh-4.4# chroot /host
   sh-4.4# rpm -q usbguard
   usbguard-0.7.4-4.el8.x86_64.rpm

Procedure

1. Create a Butane config file, 98-worker-firmware-blob.bu, that updates the search path so that it is root-owned and writable to local storage. The following example places the custom blob file from your local workstation onto nodes under /var/lib/firmware.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config file for custom firmware blob

   variant: openshift
   version: 4.15.0
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 98-worker-firmware-blob
   storage:
     files:
     - path: /var/lib/firmware/<package_name> 1
       contents:
         local: <package_name> 2
       mode: 0644 3
   openshift:
     kernel_arguments:
       - 'firmware_class.path=/var/lib/firmware' 4

   1

   Sets the path on the node where the firmware package is copied to.

   2

   Specifies a file with contents that are read from a local file directory on the system running Butane. The path of the local file is relative to a files-dir directory, which must be specified by using the --files-dir option with Butane in the following step.

   3

   Sets the permissions for the file on the RHCOS node. It is recommended to set 0644 permissions.

   4

   The firmware_class.path parameter customizes the kernel search path of where to look for the custom firmware blob that was copied from your local workstation onto the root file system of the node. This example uses /var/lib/firmware as the customized path.

2. Run Butane to generate a MachineConfig object file that uses a copy of the firmware blob on your local workstation named 98-worker-firmware-blob.yaml. The firmware blob contains the configuration to be delivered to the nodes. The following example uses the --files-dir option to specify the directory on your workstation where the local file or files are located:

   $ butane 98-worker-firmware-blob.bu -o 98-worker-firmware-blob.yaml --files-dir <directory_including_package_name>

3. Apply the configurations to the nodes in one of two ways:

   • If the cluster is not running yet, after you generate manifest files, add the MachineConfig object file to the <installation_directory>/openshift directory, and then continue to create the cluster.

   • If the cluster is already running, apply the file:

     $ oc apply -f 98-worker-firmware-blob.yaml

     A MachineConfig object YAML file is created for you to finish configuring your machines.

4. Save the Butane config in case you need to update the MachineConfig object in the future.

Procedure

1. Using a tool that is supported by your operating system, create a hashed password. For example, create a hashed password using mkpasswd by running the following command:

   $ mkpasswd -m SHA-512 testpass

   Example output

   $ $6$CBZwA6s6AVFOtiZe$aUKDWpthhJEyR3nnhM02NM1sKCpHn9XN.NPrJNQ3HYewioaorpwL3mKGLxvW0AOb4pJxqoqP4nFX77y0p00.8.

2. Create a machine config file that contains the core username and the hashed password:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: set-core-user-password
   spec:
     config:
       ignition:
         version: 3.2.0
       passwd:
         users:
         - name: core 1
           passwordHash: <password> 2

   1

   This must be core.

   2

   The hashed password to use with the core account.

3. Create the machine config by running the following command:

   $ oc create -f <file-name>.yaml

   The nodes do not reboot and should become available in a few moments. You can use the oc get mcp to watch for the machine config pools to be updated, as shown in the following example:

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-d686a3ffc8fdec47280afec446fce8dd   True      False      False      3              3                   3                     0                      64m
   worker   rendered-worker-4605605a5b1f9de1d061e9d350f251e5   False     True       False      3              0                   0                     0                      64m

Verification

1. After the nodes return to the UPDATED=True state, start a debug session for a node by running the following command:

   $ oc debug node/<node_name>

2. Set /host as the root directory within the debug shell by running the following command:

   sh-4.4# chroot /host

3. Check the contents of the /etc/shadow file:

   Example output

   ...
   core:$6$2sE/010goDuRSxxv$o18K52wor.wIwZp:19418:0:99999:7:::
   ...

   The hashed password is assigned to the core user.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CR for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: set-max-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

Procedure

1. View the available machine configuration objects that you can select:

   $ oc get machineconfig

   By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. Check the current value for the maximum pods per node:

   $ oc describe node <node_name>

   For example:

   $ oc describe node ci-ln-5grqprb-f76d1-ncnqq-worker-a-mdv94

   Look for value: pods: <value> in the Allocatable stanza:

   Example output

   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

3. Set the maximum pods per node on the worker nodes by creating a custom resource file that contains the kubelet configuration:

   Important

   Kubelet configurations that target a specific machine config pool also affect any dependent pools. For example, creating a kubelet configuration for the pool containing worker nodes will also apply to any subset pools, including the pool containing infrastructure nodes. To avoid this, you must create a new machine config pool with a selection expression that only includes worker nodes, and have your kubelet configuration target this new pool.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods 1
     kubeletConfig:
       maxPods: 500 2

   1

   Enter the label from the machine config pool.

   2

   Add the kubelet configuration. In this example, use maxPods to set the maximum pods per node.

   Note

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 50 for kubeAPIQPS and 100 for kubeAPIBurst, are sufficient if there are limited pods running on each node. It is recommended to update the kubelet QPS and burst rates if there are enough CPU and memory resources on the node.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
     kubeletConfig:
       maxPods: <pod_count>
       kubeAPIBurst: <burst_rate>
       kubeAPIQPS: <QPS>

   1. Update the machine config pool for workers with the label:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

   2. Create the KubeletConfig object:

      $ oc create -f change-maxPods-cr.yaml

   3. Verify that the KubeletConfig object is created:

      $ oc get kubeletconfig

      Example output

      NAME                AGE
      set-max-pods        15m

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Verify that the changes are applied to the node:

   1. Check on a worker node that the maxPods value changed:

      $ oc describe node <node_name>

   2. Locate the Allocatable stanza:

       ...
      Allocatable:
        attachable-volumes-gce-pd:  127
        cpu:                        3500m
        ephemeral-storage:          123201474766
        hugepages-1Gi:              0
        hugepages-2Mi:              0
        memory:                     14225400Ki
        pods:                       500 1
       ...

      1

      In this example, the pods parameter should report the value you set in the KubeletConfig object.

5. Verify the change in the KubeletConfig object:

   $ oc get kubeletconfigs set-max-pods -o yaml

   This should show a status of True and type:Success, as shown in the following example:

   spec:
     kubeletConfig:
       maxPods: 500
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
   status:
     conditions:
     - lastTransitionTime: "2021-06-30T17:04:07Z"
       message: Success
       status: "True"
       type: Success

1. Create a YAML file for the ContainerRuntimeConfig CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: ContainerRuntimeConfig
   metadata:
    name: overlay-size
   spec:
    machineConfigPoolSelector:
      matchLabels:
        pools.operator.machineconfiguration.openshift.io/worker: '' 1
    containerRuntimeConfig: 2
      logLevel: debug
      overlaySize: 8G

   1

   Specify a label for the machine config pool that you want you want to modify.

   2

   Set the parameters as needed.

2. Create the ContainerRuntimeConfig CR:

   $ oc create -f <file_name>.yaml

3. Verify that the CR is created:

   $ oc get ContainerRuntimeConfig

   Example output

   NAME           AGE
   overlay-size   3m19s

4. Check that a new containerruntime machine config is created:

   $ oc get machineconfigs | grep containerrun

   Example output

   99-worker-generated-containerruntime   2c9371fbb673b97a6fe8b1c52691999ed3a1bfc2  3.2.0  31s

5. Monitor the machine config pool until all are shown as ready:

   $ oc get mcp worker

   Example output

   NAME    CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
   worker  rendered-worker-169  False    True      False     3             1                  1                    0                     9h

6. Verify that the settings were applied in CRI-O:

   1. Open an oc debug session to a node in the machine config pool and run chroot /host.

      $ oc debug node/<node_name>

      sh-4.4# chroot /host

   2. Verify the changes in the crio.conf file:

      sh-4.4# crio config | grep 'log_level'

      Example output

      log_level = "debug"

   3. Verify the changes in the `storage.conf`file:

      sh-4.4# head -n 7 /etc/containers/storage.conf

      Example output

      [storage]
        driver = "overlay"
        runroot = "/var/run/containers/storage"
        graphroot = "/var/lib/containers/storage"
        [storage.options]
          additionalimagestores = []
          size = "8G"

Procedure

1. Create the configuration object:

   $ oc apply -f overlaysize.yml

2. To apply the new CRI-O configuration to your worker nodes, edit the worker machine config pool:

   $ oc edit machineconfigpool worker

3. Add the custom-crio label based on the matchLabels name you set in the ContainerRuntimeConfig CRD:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2020-07-09T15:46:34Z"
     generation: 3
     labels:
       custom-crio: overlay-size
       machineconfiguration.openshift.io/mco-built-in: ""

4. Save the changes, then view the machine configs:

   $ oc get machineconfigs

   New 99-worker-generated-containerruntime and rendered-worker-xyz objects are created:

   Example output

   99-worker-generated-containerruntime  4173030d89fbf4a7a0976d1665491a4d9a6e54f1   3.2.0             7m42s
   rendered-worker-xyz                   4173030d89fbf4a7a0976d1665491a4d9a6e54f1   3.2.0             7m36s

5. After those objects are created, monitor the machine config pool for the changes to be applied:

   $ oc get mcp worker

   The worker nodes show UPDATING as True, as well as the number of machines, the number updated, and other details:

   Example output

   NAME   CONFIG              UPDATED   UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker rendered-worker-xyz False True False     3             2                   2                    0                      20h

   When complete, the worker nodes transition back to UPDATING as False, and the UPDATEDMACHINECOUNT number matches the MACHINECOUNT:

   Example output

   NAME   CONFIG              UPDATED   UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker   rendered-worker-xyz   True      False      False      3         3            3             0           20h

   Looking at a worker machine, you see that the new 8 GB max size configuration is applied to all of the workers:

   Example output

   head -n 7 /etc/containers/storage.conf
   [storage]
     driver = "overlay"
     runroot = "/var/run/containers/storage"
     graphroot = "/var/lib/containers/storage"
     [storage.options]
       additionalimagestores = []
       size = "8G"

   Looking inside a container, you see that the root partition is now 8 GB:

   Example output

   ~ $ df -h
   Filesystem                Size      Used Available Use% Mounted on
   overlay                   8.0G      8.0K      8.0G   0% /

Procedure

1. View the compute machine sets that are in the cluster by running the following command:

   $ oc get machinesets -n openshift-machine-api

   The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the compute machines that are in the cluster by running the following command:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the compute machine that you want to delete by running the following command:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"

4. Scale the compute machine set by running one of the following commands:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   Tip

   You can alternatively apply the following YAML to scale the compute machine set:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     name: <machineset>
     namespace: openshift-machine-api
   spec:
     replicas: 2

   You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.

   Important

   By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed. If the drain operation fails, the machine controller cannot proceed removing the machine.

   You can skip draining the node by annotating machine.openshift.io/exclude-node-draining in a specific machine.

1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

   $ oc edit scheduler cluster

   Example Scheduler Operator CR with a node selector

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   spec:
     defaultNodeSelector: type=user-node,region=east 1
     mastersSchedulable: false

   1

   Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a compute machine set or editing the node directly:

   • Use a compute machine set to add labels to nodes managed by the compute machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api 1

        1

        Add a <key>/<value> pair for each label.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

        Tip

        You can alternatively apply the following YAML to add labels to a compute machine set:

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
          name: <machineset>
          namespace: openshift-machine-api
        spec:
          template:
            spec:
              metadata:
                labels:
                  region: "east"
                  type: "user-node"

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
          ...
        spec:
          ...
          template:
            metadata:
          ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
          ...

     3. Redeploy the nodes associated with that compute machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.28.5

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

        Tip

        You can alternatively apply the following YAML to add labels to a node:

        kind: Node
        apiVersion: v1
        metadata:
          name: <node_name>
          labels:
            type: "user-node"
            region: "east"

     2. Verify that the labels are added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.28.5