Chapter 4. Configuring MCO-related custom resources

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CR for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: set-max-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

Procedure

1. View the available machine configuration objects that you can select:

   $ oc get machineconfig

   By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. Check the current value for the maximum pods per node:

   $ oc describe node <node_name>

   For example:

   $ oc describe node ci-ln-5grqprb-f76d1-ncnqq-worker-a-mdv94

   Look for value: pods: <value> in the Allocatable stanza:

   Example output

   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

3. Set the maximum pods per node on the worker nodes by creating a custom resource file that contains the kubelet configuration:

   Important

   Kubelet configurations that target a specific machine config pool also affect any dependent pools. For example, creating a kubelet configuration for the pool containing worker nodes will also apply to any subset pools, including the pool containing infrastructure nodes. To avoid this, you must create a new machine config pool with a selection expression that only includes worker nodes, and have your kubelet configuration target this new pool.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods 1
     kubeletConfig:
       maxPods: 500 2

   1

   Enter the label from the machine config pool.

   2

   Add the kubelet configuration. In this example, use maxPods to set the maximum pods per node.

   Note

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 50 for kubeAPIQPS and 100 for kubeAPIBurst, are sufficient if there are limited pods running on each node. It is recommended to update the kubelet QPS and burst rates if there are enough CPU and memory resources on the node.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
     kubeletConfig:
       maxPods: <pod_count>
       kubeAPIBurst: <burst_rate>
       kubeAPIQPS: <QPS>

   1. Update the machine config pool for workers with the label:

      $ oc label machineconfigpool worker custom-kubelet=set-max-pods

   2. Create the KubeletConfig object:

      $ oc create -f change-maxPods-cr.yaml

   3. Verify that the KubeletConfig object is created:

      $ oc get kubeletconfig

      Example output

      NAME                AGE
      set-max-pods        15m

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Verify that the changes are applied to the node:

   1. Check on a worker node that the maxPods value changed:

      $ oc describe node <node_name>

   2. Locate the Allocatable stanza:

       ...
      Allocatable:
        attachable-volumes-gce-pd:  127
        cpu:                        3500m
        ephemeral-storage:          123201474766
        hugepages-1Gi:              0
        hugepages-2Mi:              0
        memory:                     14225400Ki
        pods:                       500 1
       ...

      1

      In this example, the pods parameter should report the value you set in the KubeletConfig object.

5. Verify the change in the KubeletConfig object:

   $ oc get kubeletconfigs set-max-pods -o yaml

   This should show a status of True and type:Success, as shown in the following example:

   spec:
     kubeletConfig:
       maxPods: 500
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: set-max-pods
   status:
     conditions:
     - lastTransitionTime: "2021-06-30T17:04:07Z"
       message: Success
       status: "True"
       type: Success

1. Create a YAML file for the ContainerRuntimeConfig CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: ContainerRuntimeConfig
   metadata:
    name: overlay-size
   spec:
    machineConfigPoolSelector:
      matchLabels:
        pools.operator.machineconfiguration.openshift.io/worker: '' 1
    containerRuntimeConfig: 2
      logLevel: debug
      overlaySize: 8G

   1

   Specify a label for the machine config pool that you want you want to modify.

   2

   Set the parameters as needed.

2. Create the ContainerRuntimeConfig CR:

   $ oc create -f <file_name>.yaml

3. Verify that the CR is created:

   $ oc get ContainerRuntimeConfig

   Example output

   NAME           AGE
   overlay-size   3m19s

4. Check that a new containerruntime machine config is created:

   $ oc get machineconfigs | grep containerrun

   Example output

   99-worker-generated-containerruntime   2c9371fbb673b97a6fe8b1c52691999ed3a1bfc2  3.2.0  31s

5. Monitor the machine config pool until all are shown as ready:

   $ oc get mcp worker

   Example output

   NAME    CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
   worker  rendered-worker-169  False    True      False     3             1                  1                    0                     9h

6. Verify that the settings were applied in CRI-O:

   1. Open an oc debug session to a node in the machine config pool and run chroot /host.

      $ oc debug node/<node_name>

      sh-4.4# chroot /host

   2. Verify the changes in the crio.conf file:

      sh-4.4# crio config | grep 'log_level'

      Example output

      log_level = "debug"

   3. Verify the changes in the `storage.conf`file:

      sh-4.4# head -n 7 /etc/containers/storage.conf

      Example output

      [storage]
        driver = "overlay"
        runroot = "/var/run/containers/storage"
        graphroot = "/var/lib/containers/storage"
        [storage.options]
          additionalimagestores = []
          size = "8G"

Procedure

1. Create the configuration object:

   $ oc apply -f overlaysize.yml

2. To apply the new CRI-O configuration to your worker nodes, edit the worker machine config pool:

   $ oc edit machineconfigpool worker

3. Add the custom-crio label based on the matchLabels name you set in the ContainerRuntimeConfig CRD:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2020-07-09T15:46:34Z"
     generation: 3
     labels:
       custom-crio: overlay-size
       machineconfiguration.openshift.io/mco-built-in: ""

4. Save the changes, then view the machine configs:

   $ oc get machineconfigs

   New 99-worker-generated-containerruntime and rendered-worker-xyz objects are created:

   Example output

   99-worker-generated-containerruntime  4173030d89fbf4a7a0976d1665491a4d9a6e54f1   3.2.0             7m42s
   rendered-worker-xyz                   4173030d89fbf4a7a0976d1665491a4d9a6e54f1   3.2.0             7m36s

5. After those objects are created, monitor the machine config pool for the changes to be applied:

   $ oc get mcp worker

   The worker nodes show UPDATING as True, as well as the number of machines, the number updated, and other details:

   Example output

   NAME   CONFIG              UPDATED   UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker rendered-worker-xyz False True False     3             2                   2                    0                      20h

   When complete, the worker nodes transition back to UPDATING as False, and the UPDATEDMACHINECOUNT number matches the MACHINECOUNT:

   Example output

   NAME   CONFIG              UPDATED   UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker   rendered-worker-xyz   True      False      False      3         3            3             0           20h

   Looking at a worker machine, you see that the new 8 GB max size configuration is applied to all of the workers:

   Example output

   head -n 7 /etc/containers/storage.conf
   [storage]
     driver = "overlay"
     runroot = "/var/run/containers/storage"
     graphroot = "/var/lib/containers/storage"
     [storage.options]
       additionalimagestores = []
       size = "8G"

   Looking inside a container, you see that the root partition is now 8 GB:

   Example output

   ~ $ df -h
   Filesystem                Size      Used Available Use% Mounted on
   overlay                   8.0G      8.0K      8.0G   0% /