Chapter 7. Postinstallation node tasks

Procedure

1. Ensure that the kubeconfig file for the cluster and the installation program that you used to install the cluster are on the RHEL 8 machine. One way to accomplish this is to use the same machine that you used to install the cluster.
2. Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.

3. Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.

   Important

   If you use SSH key-based authentication, you must manage the key with an SSH agent.

4. If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift subscription to it:

   1. Register the machine with RHSM:

      # subscription-manager register --username=<user_name> --password=<password>

   2. Pull the latest subscription data from RHSM:

      # subscription-manager refresh

   3. List the available subscriptions:

      # subscription-manager list --available --matches '*OpenShift*'

   4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

      # subscription-manager attach --pool=<pool_id>

5. Enable the repositories required by OpenShift Container Platform 4.14:

   # subscription-manager repos \
       --enable="rhel-8-for-x86_64-baseos-rpms" \
       --enable="rhel-8-for-x86_64-appstream-rpms" \
       --enable="rhocp-4.14-for-rhel-8-x86_64-rpms"

6. Install the required packages, including openshift-ansible:

   # yum install openshift-ansible openshift-clients jq

   The openshift-ansible package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients provides the oc CLI, and the jq package improves the display of JSON output on your command line.

1. On each host, register with RHSM:

   # subscription-manager register --username=<user_name> --password=<password>

2. Pull the latest subscription data from RHSM:

   # subscription-manager refresh

3. List the available subscriptions:

   # subscription-manager list --available --matches '*OpenShift*'

4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

   # subscription-manager attach --pool=<pool_id>

5. Disable all yum repositories:

   1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"

   2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist

   3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

      # yum-config-manager --disable \*

      Note that this might take a few minutes if you have a large number of available repositories

6. Enable only the repositories required by OpenShift Container Platform 4.14:

   # subscription-manager repos \
       --enable="rhel-8-for-x86_64-baseos-rpms" \
       --enable="rhel-8-for-x86_64-appstream-rpms" \
       --enable="rhocp-4.14-for-rhel-8-x86_64-rpms" \
       --enable="fast-datapath-for-rhel-8-x86_64-rpms"

7. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

1. Create an Ansible inventory file that is named /<path>/inventory/hosts that defines your compute machine hosts and required variables:

   [all:vars]
   ansible_user=root 1
   #ansible_become=True 2
   
   openshift_kubeconfig_path="~/.kube/config" 3
   
   [new_workers] 4
   mycluster-rhel8-0.example.com
   mycluster-rhel8-1.example.com

   1

   Specify the user name that runs the Ansible tasks on the remote compute machines.

   2

   If you do not specify root for the ansible_user, you must set ansible_become to True and assign the user sudo permissions.

   3

   Specify the path and file name of the kubeconfig file for your cluster.

   4

   List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the hostname that the cluster uses to access the machine, so set the correct public or private name to access the machine.

2. Navigate to the Ansible playbook directory:

   $ cd /usr/share/ansible/openshift-ansible

3. Run the playbook:

   $ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

   1

   For <path>, specify the path to the Ansible inventory file that you created.

Procedure

1. View the list of machines and record the node names of the RHCOS compute machines:

   $ oc get nodes -o wide

2. For each RHCOS compute machine, delete the node:

   1. Mark the node as unschedulable by running the oc adm cordon command:

      $ oc adm cordon <node_name> 1

      1

      Specify the node name of one of the RHCOS compute machines.

   2. Drain all the pods from the node:

      $ oc adm drain <node_name> --force --delete-emptydir-data --ignore-daemonsets 1

      1

      Specify the node name of the RHCOS compute machine that you isolated.

   3. Delete the node:

      $ oc delete nodes <node_name> 1

      1

      Specify the node name of the RHCOS compute machine that you drained.

3. Review the list of compute machines to ensure that only the RHEL nodes remain:

   $ oc get nodes -o wide

4. Remove the RHCOS machines from the load balancer for your cluster’s compute machines. You can delete the virtual machines or reimage the physical hardware for the RHCOS compute machines.

Procedure

1. Extract the Ignition config file from the cluster by running the following command:

   $ oc extract -n openshift-machine-api secret/worker-user-data-managed --keys=userData --to=- > worker.ign

2. Upload the worker.ign Ignition config file you exported from your cluster to your HTTP server. Note the URLs of these files.

3. You can validate that the ignition files are available on the URLs. The following example gets the Ignition config files for the compute node:

   $ curl -k http://<HTTP_server>/worker.ign

4. You can access the ISO image for booting your new machine by running to following command:

   RHCOS_VHD_ORIGIN_URL=$(oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.<architecture>.artifacts.metal.formats.iso.disk.location')

5. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.

6. Boot the RHCOS ISO image without specifying any options, or interrupting the live boot sequence. Wait for the installer to boot into a shell prompt in the RHCOS live environment.

   Note

   You can interrupt the RHCOS installation boot process to add kernel arguments. However, for this ISO procedure you must use the coreos-installer command as outlined in the following steps, instead of adding kernel arguments.

7. Run the coreos-installer command and specify the options that meet your installation requirements. At a minimum, you must specify the URL that points to the Ignition config file for the node type, and the device that you are installing to:

   $ sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest> 12

   1

   You must run the coreos-installer command by using sudo, because the core user does not have the required root privileges to perform the installation.

   2

   The --ignition-hash option is required when the Ignition config file is obtained through an HTTP URL to validate the authenticity of the Ignition config file on the cluster node. <digest> is the Ignition config file SHA512 digest obtained in a preceding step.

   Note

   If you want to provide your Ignition config files through an HTTPS server that uses TLS, you can add the internal certificate authority (CA) to the system trust store before running coreos-installer.

   The following example initializes a bootstrap node installation to the /dev/sda device. The Ignition config file for the bootstrap node is obtained from an HTTP web server with the IP address 192.168.1.2:

   $ sudo coreos-installer install --ignition-url=http://192.168.1.2:80/installation_directory/bootstrap.ign /dev/sda --ignition-hash=sha512-a5a2d43879223273c9b60af66b44202a1d1248fc01cf156c46d4a79f552b6bad47bc8cc78ddf0116e80c59d2ea9e32ba53bc807afbca581aa059311def2c3e3b

8. Monitor the progress of the RHCOS installation on the console of the machine.

   Important

   Ensure that the installation is successful on each node before commencing with the OpenShift Container Platform installation. Observing the installation process can also help to determine the cause of RHCOS installation issues that might arise.

9. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

     Note

     This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

   • For iPXE (x86_64 + aarch64):

     kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
     initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img 3
     boot

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your HTTP server.

     Note

     This configuration does not enable serial console access on machines with a graphical console To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux? and "Enabling the serial console for PXE and ISO installation" in the "Advanced RHCOS installation configuration" section.

     Note

     To network boot the CoreOS kernel on aarch64 architecture, you need to use a version of iPXE build with the IMAGE_GZIP option enabled. See IMAGE_GZIP option in iPXE.

   • For PXE (with UEFI and GRUB as second stage) on aarch64:

     menuentry 'Install CoreOS' {
         linux rhcos-<version>-live-kernel-<architecture>  coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign 1 2
         initrd rhcos-<version>-live-initramfs.<architecture>.img 3
     }

     1

     Specify the locations of the RHCOS files that you uploaded to your HTTP/TFTP server. The kernel parameter value is the location of the kernel file on your TFTP server. The coreos.live.rootfs_url parameter value is the location of the rootfs file, and the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file on your HTTP Server.

     2

     If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC that is named eno1, set ip=eno1:dhcp.

     3

     Specify the location of the initramfs file that you uploaded to your TFTP server.

2. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.27.3
   master-1  Ready     master  63m  v1.27.3
   master-2  Ready     master  64m  v1.27.3

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.27.3
   master-1  Ready     master  73m  v1.27.3
   master-2  Ready     master  74m  v1.27.3
   worker-0  Ready     worker  11m  v1.27.3
   worker-1  Ready     worker  11m  v1.27.3

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. On a command line, change to the openshift-machine-api namespace:

   $ oc project openshift-machine-api

2. Create a new secret from the worker-user-data secret:

   1. Export the userData section of the secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt

   2. Edit the text file to add the storage, filesystems, and systemd stanzas for the partitions you want to use for the new node. You can specify any Ignition configuration parameters as needed.

      Note

      Do not change the values in the ignition stanza.

      {
        "ignition": {
          "config": {
            "merge": [
              {
                "source": "https:...."
              }
            ]
          },
          "security": {
            "tls": {
              "certificateAuthorities": [
                {
                  "source": "data:text/plain;charset=utf-8;base64,.....=="
                }
              ]
            }
          },
          "version": "3.2.0"
        },
        "storage": {
          "disks": [
            {
              "device": "/dev/nvme1n1", 1
              "partitions": [
                {
                  "label": "var",
                  "sizeMiB": 50000, 2
                  "startMiB": 0 3
                }
              ]
            }
          ],
          "filesystems": [
            {
              "device": "/dev/disk/by-partlabel/var", 4
              "format": "xfs", 5
              "path": "/var" 6
            }
          ]
        },
        "systemd": {
          "units": [ 7
            {
              "contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
              "enabled": true,
              "name": "var.mount"
            }
          ]
        }
      }

      1

      Specifies an absolute path to the AWS block device.

      2

      Specifies the size of the data partition in Mebibytes.

      3

      Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition.

      4

      Specifies an absolute path to the /var partition.

      5

      Specifies the filesystem format.

      6

      Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.

      7

      Defines a systemd mount unit that mounts the /dev/disk/by-partlabel/var device to the /var partition.

   3. Extract the disableTemplating section from the work-user-data secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt

   4. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the userData.txt file to the newly created node.

      $ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt

3. Create a new compute machine set for the new node:

   1. Create a new compute machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:

      Tip

      Use an existing compute machine set as a template and change the parameters as needed for the new node.

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: auto-52-92tf4
        name: worker-us-east-2-nvme1n1 1
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: auto-52-92tf4
            machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: auto-52-92tf4
              machine.openshift.io/cluster-api-machine-role: worker
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
          spec:
            metadata: {}
            providerSpec:
              value:
                ami:
                  id: ami-0c2dbd95931a
                apiVersion: awsproviderconfig.openshift.io/v1beta1
                blockDevices:
                - DeviceName: /dev/nvme1n1 2
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 120
                    volumeType: gp2
                - DeviceName: /dev/nvme1n2 3
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 50
                    volumeType: gp2
                credentialsSecret:
                  name: aws-cloud-credentials
                deviceIndex: 0
                iamInstanceProfile:
                  id: auto-52-92tf4-worker-profile
                instanceType: m6i.large
                kind: AWSMachineProviderConfig
                metadata:
                  creationTimestamp: null
                placement:
                  availabilityZone: us-east-2b
                  region: us-east-2
                securityGroups:
                - filters:
                  - name: tag:Name
                    values:
                    - auto-52-92tf4-worker-sg
                subnet:
                  id: subnet-07a90e5db1
                tags:
                - name: kubernetes.io/cluster/auto-52-92tf4
                  value: owned
                userDataSecret:
                  name: worker-user-data-x5 4

      1

      Specifies a name for the new node.

      2

      Specifies an absolute path to the AWS block device, here an encrypted EBS volume.

      3

      Optional. Specifies an additional EBS volume.

      4

      Specifies the user data secret file.

   2. Create the compute machine set:

      $ oc create -f <file-name>.yaml

      The machines might take a few moments to become available.

4. Verify that the new partition and nodes are created:

   1. Verify that the compute machine set is created:

      $ oc get machineset

      Example output

      NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a        1         1         1       1           124m
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b        2         2         2       2           124m
      worker-us-east-2-nvme1n1                           1         1         1       1           2m35s 1

      1

      This is the new compute machine set.

   2. Verify that the new node is created:

      $ oc get nodes

      Example output

      NAME                           STATUS   ROLES    AGE     VERSION
      ip-10-0-128-78.ec2.internal    Ready    worker   117m    v1.27.3
      ip-10-0-146-113.ec2.internal   Ready    master   127m    v1.27.3
      ip-10-0-153-35.ec2.internal    Ready    worker   118m    v1.27.3
      ip-10-0-176-58.ec2.internal    Ready    master   126m    v1.27.3
      ip-10-0-217-135.ec2.internal   Ready    worker   2m57s   v1.27.3 1
      ip-10-0-225-248.ec2.internal   Ready    master   127m    v1.27.3
      ip-10-0-245-59.ec2.internal    Ready    worker   116m    v1.27.3

      1

      This is new new node.

   3. Verify that the custom /var partition is created on the new node:

      $ oc debug node/<node-name> -- chroot /host lsblk

      For example:

      $ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk

      Example output

      NAME        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
      nvme0n1     202:0    0   120G  0 disk
      |-nvme0n1p1 202:1    0     1M  0 part
      |-nvme0n1p2 202:2    0   127M  0 part
      |-nvme0n1p3 202:3    0   384M  0 part /boot
      `-nvme0n1p4 202:4    0 119.5G  0 part /sysroot
      nvme1n1     202:16   0    50G  0 disk
      `-nvme1n1p1 202:17   0  48.8G  0 part /var 1

      1

      The nvme1n1 device is mounted to the /var partition.

Procedure

1. Create a healthcheck.yml file that contains the definition of your machine health check.

2. Apply the healthcheck.yml file to your cluster:

   $ oc apply -f healthcheck.yml

Procedure

1. View the compute machine sets that are in the cluster by running the following command:

   $ oc get machinesets -n openshift-machine-api

   The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the compute machines that are in the cluster by running the following command:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the compute machine that you want to delete by running the following command:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"

4. Scale the compute machine set by running one of the following commands:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   Tip

   You can alternatively apply the following YAML to scale the compute machine set:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     name: <machineset>
     namespace: openshift-machine-api
   spec:
     replicas: 2

   You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.

   Important

   By default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed. If the drain operation fails, the machine controller cannot proceed removing the machine.

   You can skip draining the node by annotating machine.openshift.io/exclude-node-draining in a specific machine.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CR for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: set-max-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

Procedure

1. View the available machine configuration objects that you can select:

   $ oc get machineconfig

   By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. Check the current value for the maximum pods per node:

   $ oc describe node <node_name>

   For example:

   $ oc describe node ci-ln-5grqprb-f76d1-ncnqq-worker-a-mdv94

   Look for value: pods: <value> in the Allocatable stanza:

   Example output

   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

3. Set the maximum pods per node on the worker nodes by creating a custom resource file that contains the kubelet configuration:

   Important

   Kubelet configurations that target a specific machine config pool also affect any dependent pools. For example, creating a kubelet configuration for the pool containing worker nodes will also apply to any subset pools, including the pool containing infrastructure nodes. To avoid this, you must create a new machine config pool with a selection expression that only includes worker nodes, and have your kubelet configuration target this new pool.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods 1
     kubeletConfig:
       maxPods: 500 2

   1

   Enter the label from the machine config pool.

   2

   Add the kubelet configuration. In this example, use maxPods to set the maximum pods per node.

   Note

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 50 for kubeAPIQPS and 100 for kubeAPIBurst, are sufficient if there are limited pods running on each node. It is recommended to update the kubelet QPS and burst rates if there are enough CPU and memory resources on the node.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
     kubeletConfig:
       maxPods: <pod_count>
       kubeAPIBurst: <burst_rate>
       kubeAPIQPS: <QPS>

   1. Update the machine config pool for workers with the label:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

   2. Create the KubeletConfig object:

      $ oc create -f change-maxPods-cr.yaml

   3. Verify that the KubeletConfig object is created:

      $ oc get kubeletconfig

      Example output

      NAME                AGE
      set-max-pods        15m

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Verify that the changes are applied to the node:

   1. Check on a worker node that the maxPods value changed:

      $ oc describe node <node_name>

   2. Locate the Allocatable stanza:

       ...
      Allocatable:
        attachable-volumes-gce-pd:  127
        cpu:                        3500m
        ephemeral-storage:          123201474766
        hugepages-1Gi:              0
        hugepages-2Mi:              0
        memory:                     14225400Ki
        pods:                       500 1
       ...

      1

      In this example, the pods parameter should report the value you set in the KubeletConfig object.

5. Verify the change in the KubeletConfig object:

   $ oc get kubeletconfigs set-max-pods -o yaml

   This should show a status of True and type:Success, as shown in the following example:

   spec:
     kubeletConfig:
       maxPods: 500
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
   status:
     conditions:
     - lastTransitionTime: "2021-06-30T17:04:07Z"
       message: Success
       status: "True"
       type: Success

Procedure

1. Edit the worker machine config pool:

   $ oc edit machineconfigpool worker

2. Add the maxUnavailable field and set the value:

   spec:
     maxUnavailable: <node_count>

   Important

   When setting the value, consider the number of worker nodes that can be unavailable without affecting the applications running on the cluster.

Procedure

1. Optional: Label a node:

   # oc label node perf-node.example.com cpumanager=true

2. Edit the MachineConfigPool of the nodes where CPU Manager should be enabled. In this example, all workers have CPU Manager enabled:

   # oc edit machineconfigpool worker

3. Add a label to the worker machine config pool:

   metadata:
     creationTimestamp: 2020-xx-xxx
     generation: 3
     labels:
       custom-kubelet: cpumanager-enabled

4. Create a KubeletConfig, cpumanager-kubeletconfig.yaml, custom resource (CR). Refer to the label created in the previous step to have the correct nodes updated with the new kubelet config. See the machineConfigPoolSelector section:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static 1
        cpuManagerReconcilePeriod: 5s 2

   1

   Specify a policy:

   • none. This policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the scheduler does automatically. This is the default policy.
   • static. This policy allows containers in guaranteed pods with integer CPU requests. It also limits access to exclusive CPUs on the node. If static, you must use a lowercase s.

   2

   Optional. Specify the CPU Manager reconcile frequency. The default is 5s.

5. Create the dynamic kubelet config:

   # oc create -f cpumanager-kubeletconfig.yaml

   This adds the CPU Manager feature to the kubelet config and, if needed, the Machine Config Operator (MCO) reboots the node. To enable CPU Manager, a reboot is not needed.

6. Check for the merged kubelet config:

   # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7

   Example output

          "ownerReferences": [
               {
                   "apiVersion": "machineconfiguration.openshift.io/v1",
                   "kind": "KubeletConfig",
                   "name": "cpumanager-enabled",
                   "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878"
               }
           ]

7. Check the worker for the updated kubelet.conf:

   # oc debug node/perf-node.example.com
   sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager

   Example output

   cpuManagerPolicy: static        1
   cpuManagerReconcilePeriod: 5s   2

   1

   cpuManagerPolicy is defined when you create the KubeletConfig CR.

   2

   cpuManagerReconcilePeriod is defined when you create the KubeletConfig CR.

8. Create a pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this pod:

   # cat cpumanager-pod.yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: cpumanager-
   spec:
     containers:
     - name: cpumanager
       image: gcr.io/google_containers/pause:3.2
       resources:
         requests:
           cpu: 1
           memory: "1G"
         limits:
           cpu: 1
           memory: "1G"
     nodeSelector:
       cpumanager: "true"

9. Create the pod:

   # oc create -f cpumanager-pod.yaml

10. Verify that the pod is scheduled to the node that you labeled:

    # oc describe pod cpumanager

    Example output

    Name:               cpumanager-6cqz7
    Namespace:          default
    Priority:           0
    PriorityClassName:  <none>
    Node:  perf-node.example.com/xxx.xx.xx.xxx
    ...
     Limits:
          cpu:     1
          memory:  1G
        Requests:
          cpu:        1
          memory:     1G
    ...
    QoS Class:       Guaranteed
    Node-Selectors:  cpumanager=true

11. Verify that the cgroups are set up correctly. Get the process ID (PID) of the pause process:

    # ├─init.scope
    │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
    └─kubepods.slice
      ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice
      │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope
      │ └─32706 /pause

    Pods of quality of service (QoS) tier Guaranteed are placed within the kubepods.slice. Pods of other QoS tiers end up in child cgroups of kubepods:

    # cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope
    # for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done

    Example output

    cpuset.cpus 1
    tasks 32706

12. Check the allowed CPU list for the task:

    # grep ^Cpus_allowed_list /proc/32706/status

    Example output

     Cpus_allowed_list:    1

13. Verify that another pod (in this case, the pod in the burstable QoS tier) on the system cannot run on the core allocated for the Guaranteed pod:

    # cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus
    0
    # oc describe node perf-node.example.com

    Example output

    ...
    Capacity:
     attachable-volumes-aws-ebs:  39
     cpu:                         2
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      8162900Ki
     pods:                        250
    Allocatable:
     attachable-volumes-aws-ebs:  39
     cpu:                         1500m
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      7548500Ki
     pods:                        250
    -------                               ----                           ------------  ----------  ---------------  -------------  ---
      default                                 cpumanager-6cqz7               1 (66%)       1 (66%)     1G (12%)         1G (12%)       29m
    
    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      Resource                    Requests          Limits
      --------                    --------          ------
      cpu                         1440m (96%)       1 (66%)

    This VM has two CPU cores. The system-reserved setting reserves 500 millicores, meaning that half of one core is subtracted from the total capacity of the node to arrive at the Node Allocatable amount. You can see that Allocatable CPU is 1500 millicores. This means you can run one of the CPU Manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores. If you try to schedule a second pod, the system will accept the pod, but it will never be scheduled:

    NAME                    READY   STATUS    RESTARTS   AGE
    cpumanager-6cqz7        1/1     Running   0          33m
    cpumanager-7qc2t        0/1     Pending   0          11s

1. Label all nodes that need the same huge pages setting by a label.

   $ oc label node <node_using_hugepages> node-role.kubernetes.io/worker-hp=

2. Create a file with the following content and name it hugepages-tuned-boottime.yaml:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: hugepages 1
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile: 2
     - data: |
         [main]
         summary=Boot time configuration for hugepages
         include=openshift-node
         [bootloader]
         cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50 3
       name: openshift-node-hugepages
   
     recommend:
     - machineConfigLabels: 4
         machineconfiguration.openshift.io/role: "worker-hp"
       priority: 30
       profile: openshift-node-hugepages

   1

   Set the name of the Tuned resource to hugepages.

   2

   Set the profile section to allocate huge pages.

   3

   Note the order of parameters is important as some platforms support huge pages of various sizes.

   4

   Enable machine config pool based matching.

3. Create the Tuned hugepages object

   $ oc create -f hugepages-tuned-boottime.yaml

4. Create a file with the following content and name it hugepages-mcp.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-hp
     labels:
       worker-hp: ""
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-hp: ""

5. Create the machine config pool:

   $ oc create -f hugepages-mcp.yaml

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command. Perform one of the following steps:

   1. View the machine config:

      # oc describe machineconfig <name>

      For example:

      # oc describe machineconfig 00-worker

      Example output

      Name:         00-worker
      Namespace:
      Labels:       machineconfiguration.openshift.io/role=worker 1

      1

      Label required for the Device Manager.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a Device Manager CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: devicemgr 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
          machineconfiguration.openshift.io: devicemgr 2
     kubeletConfig:
       feature-gates:
         - DevicePlugins=true 3

   1

   Assign a name to CR.

   2

   Enter the label from the Machine Config Pool.

   3

   Set DevicePlugins to 'true`.

2. Create the Device Manager:

   $ oc create -f devicemgr.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/devicemgr created

3. Ensure that Device Manager was actually enabled by confirming that /var/lib/kubelet/device-plugins/kubelet.sock is created on the node. This is the UNIX domain socket on which the Device Manager gRPC server listens for new plugin registrations. This sock file is created when the Kubelet is started only if Device Manager is enabled.

1. If you add a NoSchedule taint to a control plane node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
     name: my-node
   #...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   #...

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with an Equal operator

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
     tolerations:
     - key: "key1" 1
       value: "value1"
       operator: "Equal"
       effect: "NoExecute"
       tolerationSeconds: 3600 2
   #...

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod can remain bound to a node before being evicted.

   For example:

   Sample pod configuration file with an Exists operator

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
      tolerations:
       - key: "key1"
         operator: "Exists" 1
         effect: "NoExecute"
         tolerationSeconds: 3600
   #...

   1

   The Exists operator does not take a value.

   This example places a taint on node1 that has key key1, value value1, and taint effect NoExecute.

2. Add a taint to a node by using the following command with the parameters described in the Taint and toleration components table:

   $ oc adm taint nodes <node_name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 key1=value1:NoExecute

   This command places a taint on node1 that has key key1, value value1, and effect NoExecute.

   Note

   If you add a NoSchedule taint to a control plane node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
     name: my-node
   #...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   #...

   The tolerations on the pod match the taint on the node. A pod with either toleration can be scheduled onto node1.

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with Equal operator

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
     tolerations:
     - key: "key1" 1
       value: "value1"
       operator: "Equal"
       effect: "NoExecute"
       tolerationSeconds: 3600 2
   #...

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod is bound to a node before being evicted.

   For example:

   Sample pod configuration file with Exists operator

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
     tolerations:
     - key: "key1"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600
   #...

2. Add the taint to the MachineSet object:

   1. Edit the MachineSet YAML for the nodes you want to taint or you can create a new MachineSet object:

      $ oc edit machineset <machineset>

   2. Add the taint to the spec.template.spec section:

      Example taint in a compute machine set specification

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        name: my-machineset
      #...
      spec:
      #...
        template:
      #...
          spec:
            taints:
            - effect: NoExecute
              key: key1
              value: value1
      #...

      This example places a taint that has the key key1, value value1, and taint effect NoExecute on the nodes.

   3. Scale down the compute machine set to 0:

      $ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api

      Tip

      You can alternatively apply the following YAML to scale the compute machine set:

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        name: <machineset>
        namespace: openshift-machine-api
      spec:
        replicas: 0

      Wait for the machines to be removed.

   4. Scale up the compute machine set as needed:

      $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

      Or:

      $ oc edit machineset <machineset> -n openshift-machine-api

      Wait for the machines to start. The taint is added to the nodes associated with the MachineSet object.

1. Add a corresponding taint to those nodes:

   For example:

   $ oc adm taint nodes node1 dedicated=groupName:NoSchedule

   Tip

   You can alternatively apply the following YAML to add the taint:

   kind: Node
   apiVersion: v1
   metadata:
     name: my-node
   #...
   spec:
     taints:
       - key: dedicated
         value: groupName
         effect: NoSchedule
   #...

2. Add a toleration to the pods by writing a custom admission controller.

1. Add a toleration to pods that need the special hardware.

   For example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
     tolerations:
       - key: "disktype"
         value: "ssd"
         operator: "Equal"
         effect: "NoSchedule"
         tolerationSeconds: 3600
   #...

2. Taint the nodes that have the specialized hardware using one of the following commands:

   $ oc adm taint nodes <node-name> disktype=ssd:NoSchedule

   Or:

   $ oc adm taint nodes <node-name> disktype=ssd:PreferNoSchedule

   Tip

   You can alternatively apply the following YAML to add the taint:

   kind: Node
   apiVersion: v1
   metadata:
     name: my_node
   #...
   spec:
     taints:
       - key: disktype
         value: ssd
         effect: PreferNoSchedule
   #...

1. To remove a taint from a node:

   $ oc adm taint nodes <node-name> <key>-

   For example:

   $ oc adm taint nodes ip-10-0-132-248.ec2.internal key1-

   Example output

   node/ip-10-0-132-248.ec2.internal untainted

2. To remove a toleration from a pod, edit the Pod spec to remove the toleration:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
   #...
   spec:
     tolerations:
     - key: "key2"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600
   #...

1. Configure the Topology Manager allocation policy in the custom resource.

   $ oc edit KubeletConfig cpumanager-enabled

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static 1
        cpuManagerReconcilePeriod: 5s
        topologyManagerPolicy: single-numa-node 2

   1

   This parameter must be static with a lowercase s.

   2

   Specify your selected Topology Manager allocation policy. Here, the policy is single-numa-node. Acceptable values are: default, best-effort, restricted, single-numa-node.

1. In the OpenShift Container Platform web console, navigate to Home Projects

   1. Click Create Project.
   2. Specify clusterresourceoverride-operator as the name of the project.
   3. Click Create.

2. Navigate to Operators OperatorHub.

   1. Choose ClusterResourceOverride Operator from the list of available Operators and click Install.
   2. On the Install Operator page, make sure A specific Namespace on the cluster is selected for Installation Mode.
   3. Make sure clusterresourceoverride-operator is selected for Installed Namespace.
   4. Select an Update Channel and Approval Strategy.
   5. Click Install.

3. On the Installed Operators page, click ClusterResourceOverride.

   1. On the ClusterResourceOverride Operator details page, click Create ClusterResourceOverride.

   2. On the Create ClusterResourceOverride page, click YAML view and edit the YAML template to set the overcommit values as needed:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4
      # ...

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Click Create.

4. Check the current state of the admission webhook by checking the status of the cluster custom resource:

   1. On the ClusterResourceOverride Operator page, click cluster.

   2. On the ClusterResourceOverride Details page, click YAML. The mutatingWebhookConfigurationRef section appears when the webhook is called.

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
        creationTimestamp: "2019-12-18T22:35:02Z"
        generation: 1
        name: cluster
        resourceVersion: "127622"
        selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
        uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
      spec:
        podResourceOverride:
          spec:
            cpuRequestToLimitPercent: 25
            limitCPUToMemoryPercent: 200
            memoryRequestToLimitPercent: 50
      status:
      
      # ...
      
          mutatingWebhookConfigurationRef: 1
            apiVersion: admissionregistration.k8s.io/v1
            kind: MutatingWebhookConfiguration
            name: clusterresourceoverrides.admission.autoscaling.openshift.io
            resourceVersion: "127621"
            uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
      
      # ...

      1

      Reference to the ClusterResourceOverride admission webhook.

1. Create a namespace for the Cluster Resource Override Operator:

   1. Create a Namespace object YAML file (for example, cro-namespace.yaml) for the Cluster Resource Override Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: clusterresourceoverride-operator

   2. Create the namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-namespace.yaml

2. Create an Operator group:

   1. Create an OperatorGroup object YAML file (for example, cro-og.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: clusterresourceoverride-operator
        namespace: clusterresourceoverride-operator
      spec:
        targetNamespaces:
          - clusterresourceoverride-operator

   2. Create the Operator Group:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-og.yaml

3. Create a subscription:

   1. Create a Subscription object YAML file (for example, cro-sub.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: clusterresourceoverride
        namespace: clusterresourceoverride-operator
      spec:
        channel: "4.14"
        name: clusterresourceoverride
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the subscription:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-sub.yaml

4. Create a ClusterResourceOverride custom resource (CR) object in the clusterresourceoverride-operator namespace:

   1. Change to the clusterresourceoverride-operator namespace.

      $ oc project clusterresourceoverride-operator

   2. Create a ClusterResourceOverride object YAML file (for example, cro-cr.yaml) for the Cluster Resource Override Operator:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
          name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Create the ClusterResourceOverride object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-cr.yaml

5. Verify the current state of the admission webhook by checking the status of the cluster custom resource.

   $ oc get clusterresourceoverride cluster -n clusterresourceoverride-operator -o yaml

   The mutatingWebhookConfigurationRef section appears when the webhook is called.

   Example output

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
     creationTimestamp: "2019-12-18T22:35:02Z"
     generation: 1
     name: cluster
     resourceVersion: "127622"
     selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
     uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
   spec:
     podResourceOverride:
       spec:
         cpuRequestToLimitPercent: 25
         limitCPUToMemoryPercent: 200
         memoryRequestToLimitPercent: 50
   status:
   
   # ...
   
       mutatingWebhookConfigurationRef: 1
         apiVersion: admissionregistration.k8s.io/v1
         kind: MutatingWebhookConfiguration
         name: clusterresourceoverrides.admission.autoscaling.openshift.io
         resourceVersion: "127621"
         uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
   
   # ...

   1

   Reference to the ClusterResourceOverride admission webhook.

1. Edit the ClusterResourceOverride CR:

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
       name: cluster
   spec:
     podResourceOverride:
       spec:
         memoryRequestToLimitPercent: 50 1
         cpuRequestToLimitPercent: 25 2
         limitCPUToMemoryPercent: 200 3
   # ...

   1

   Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

   2

   Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

   3

   Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

2. Ensure the following label has been added to the Namespace object for each project where you want the Cluster Resource Override Operator to control overcommit:

   apiVersion: v1
   kind: Namespace
   metadata:
   
   # ...
   
     labels:
       clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true" 1
   
   # ...

   1

   Add this label to each project.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a disabling CPU limits

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: disable-cpu-units 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 2
     kubeletConfig:
       cpuCfsQuota: false 3

   1

   Assign a name to CR.

   2

   Specify the label from the machine config pool.

   3

   Set the cpuCfsQuota parameter to false.

2. Run the following command to create the CR:

   $ oc create -f <file_name>.yaml

1. Create or edit the namespace object file.

2. Add the following annotation:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       quota.openshift.io/cluster-resource-override-enabled: "false" 1
   # ...

   1

   Setting this annotation to false disables overcommit for this namespace.

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:

   $ oc edit machineconfigpool <name>

   For example:

   $ oc edit machineconfigpool worker

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2022-11-16T15:34:25Z"
     generation: 4
     labels:
       pools.operator.machineconfiguration.openshift.io/worker: "" 1
     name: worker
   #...

   1

   The label appears under Labels.

   Tip

   If the label is not present, add a key/value pair such as:

   $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Important

   If there is one file system, or if /var/lib/kubelet and /var/lib/containers/ are in the same file system, the settings with the highest values trigger evictions, as those are met first. The file system triggers the eviction.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 2
     kubeletConfig:
       evictionSoft: 3
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard: 6
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 7
       imageMinimumGCAge: 5m 8
       imageGCHighThresholdPercent: 80 9
       imageGCLowThresholdPercent: 75 10
   #...

   1

   Name for the object.

   2

   Specify the label from the machine config pool.

   3

   For container garbage collection: Type of eviction: evictionSoft or evictionHard.

   4

   For container garbage collection: Eviction thresholds based on a specific eviction trigger signal.

   5

   For container garbage collection: Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   For container garbage collection: Eviction thresholds based on a specific eviction trigger signal. For evictionHard you must specify all of these parameters. If you do not specify all parameters, only the specified parameters are applied and the garbage collection will not function properly.

   7

   For container garbage collection: The duration to wait before transitioning out of an eviction pressure condition.

   8

   For image garbage collection: The minimum age for an unused image before the image is removed by garbage collection.

   9

   For image garbage collection: The percent of disk usage (expressed as an integer) that triggers image garbage collection.

   10

   For image garbage collection: The percent of disk usage (expressed as an integer) that image garbage collection attempts to free.

2. Run the following command to create the CR:

   $ oc create -f <file_name>.yaml

   For example:

   $ oc create -f gc-container.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/gc-container created

Verification

1. Verify that garbage collection is active by entering the following command. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Procedure

1. Create a machine resource YAML file and define static IP address network information in the network parameter.

   Example of a machine resource YAML file with static IP address information defined in the network parameter.

   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     creationTimestamp: null
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
       machine.openshift.io/cluster-api-machine-role: <role>
       machine.openshift.io/cluster-api-machine-type: <role>
       machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
     name: <infrastructure_id>-<role>
     namespace: openshift-machine-api
   spec:
     lifecycleHooks: {}
     metadata: {}
     providerSpec:
       value:
         apiVersion: machine.openshift.io/v1beta1
         credentialsSecret:
           name: vsphere-cloud-credentials
         diskGiB: 120
         kind: VSphereMachineProviderSpec
         memoryMiB: 8192
         metadata:
           creationTimestamp: null
         network:
           devices:
           - gateway: 192.168.204.1 1
             ipAddrs:
             - 192.168.204.8/24 2
             nameservers: 3
             - 192.168.204.1
             networkName: qe-segment-204
         numCPUs: 4
         numCoresPerSocket: 2
         snapshot: ""
         template: <vm_template_name>
         userDataSecret:
           name: worker-user-data
         workspace:
           datacenter: <vcenter_datacenter_name>
           datastore: <vcenter_datastore_name>
           folder: <vcenter_vm_folder_path>
           resourcepool: <vsphere_resource_pool>
           server: <vcenter_server_ip>
   status: {}

   1

   The IP address for the default gateway for the network interface.

   2

   Lists IPv4, IPv6, or both IP addresses that installation program passes to the network interface. Both IP families must use the same network interface for the default network.

   3

   Lists a DNS nameserver. You can define up to 3 DNS nameservers. Consider defining more than one DNS nameserver to take advantage of DNS resolution if that one DNS nameserver becomes unreachable.

   • Create a machine custom resource (CR) by entering the following command in your terminal:

     $ oc create -f <file_name>.yaml

Procedure

1. Configure a machine set by specifying IP pool information in the network.devices.addressesFromPools schema of the machine set’s YAML file:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     annotations:
       machine.openshift.io/memoryMb: "8192"
       machine.openshift.io/vCPU: "4"
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
     name: <infrastructure_id>-<role>
     namespace: openshift-machine-api
   spec:
     replicas: 0
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: <infrastructure_id>
         machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
     template:
       metadata:
         labels:
           ipam: "true"
           machine.openshift.io/cluster-api-cluster: <infrastructure_id>
           machine.openshift.io/cluster-api-machine-role: worker
           machine.openshift.io/cluster-api-machine-type: worker
           machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
       spec:
         lifecycleHooks: {}
         metadata: {}
         providerSpec:
           value:
             apiVersion: machine.openshift.io/v1beta1
             credentialsSecret:
               name: vsphere-cloud-credentials
             diskGiB: 120
             kind: VSphereMachineProviderSpec
             memoryMiB: 8192
             metadata: {}
             network:
               devices:
               - addressesFromPools: 1
                 - group: ipamcontroller.example.io
                   name: static-ci-pool
                   resource: IPPool
                 nameservers:
                 - "192.168.204.1" 2
                 networkName: qe-segment-204
             numCPUs: 4
             numCoresPerSocket: 2
             snapshot: ""
             template: rvanderp4-dev-9n5wg-rhcos-generated-region-generated-zone
             userDataSecret:
               name: worker-user-data
             workspace:
               datacenter: IBMCdatacenter
               datastore: /IBMCdatacenter/datastore/vsanDatastore
               folder: /IBMCdatacenter/vm/rvanderp4-dev-9n5wg
               resourcePool: /IBMCdatacenter/host/IBMCcluster//Resources
               server: vcenter.ibmc.devcluster.openshift.com

   1

   Specifies an IP pool, which lists a static IP address or a range of static IP addresses. The IP Pool can either be a reference to a custom resource definition (CRD) or a resource supported by the IPAddressClaims resource handler. The machine controller accesses static IP addresses listed in the machine set’s configuration and then allocates each address to each machine.

   2

   Lists a nameserver. You must specify a nameserver for nodes that receive static IP address, because the Dynamic Host Configuration Protocol (DHCP) network configuration does not support static IP addresses.

2. Scale the machine set by entering the following commands in your oc CLI:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   After each machine is scaled up, the machine controller creates an IPAddresssClaim resource.

3. Optional: Check that the IPAddressClaim resource exists in the openshift-machine-api namespace by entering the following command:

   $ oc get ipaddressclaims.ipam.cluster.x-k8s.io -n openshift-machine-api

   Example oc CLI output that lists two IP pools listed in the openshift-machine-api namespace

   NAME                                         POOL NAME        POOL KIND
   cluster-dev-9n5wg-worker-0-m7529-claim-0-0   static-ci-pool   IPPool
   cluster-dev-9n5wg-worker-0-wdqkt-claim-0-0   static-ci-pool   IPPool

4. Create an IPAddress resource by entering the following command:

   $ oc create -f ipaddress.yaml

   The following example shows an IPAddress resource with defined network configuration information and one defined static IP address:

   apiVersion: ipam.cluster.x-k8s.io/v1alpha1
   kind: IPAddress
   metadata:
     name: cluster-dev-9n5wg-worker-0-m7529-ipaddress-0-0
     namespace: openshift-machine-api
   spec:
     address: 192.168.204.129
     claimRef: 1
       name: cluster-dev-9n5wg-worker-0-m7529-claim-0-0
     gateway: 192.168.204.1
     poolRef: 2
       apiGroup: ipamcontroller.example.io
       kind: IPPool
       name: static-ci-pool
     prefix: 23

   1

   The name of the target IPAddressClaim resource.

   2

   Details information about the static IP address or addresses from your nodes.

   Note

   By default, the external controller automatically scans any resources in the machine set for recognizable address pool types. When the external controller finds kind: IPPool defined in the IPAddress resource, the controller binds any static IP addresses to the IPAddressClaim resource.

5. Update the IPAddressClaim status with a reference to the IPAddress resource:

   $ oc --type=merge patch IPAddressClaim cluster-dev-9n5wg-worker-0-m7529-claim-0-0 -p='{"status":{"addressRef": {"name": "cluster-dev-9n5wg-worker-0-m7529-ipaddress-0-0"}}}' -n openshift-machine-api --subresource=status