Chapter 6. Working with nodes

Procedure

1. Mark the nodes unschedulable before performing the pod evacuation.

   1. Mark the node as unschedulable:

      $ oc adm cordon <node1>

      Example output

      node/<node1> cordoned

   2. Check that the node status is Ready,SchedulingDisabled:

      $ oc get node <node1>

      Example output

      NAME        STATUS                     ROLES     AGE       VERSION
      <node1>     Ready,SchedulingDisabled   worker    1d        v1.24.0

2. Evacuate the pods using one of the following methods:

   • Evacuate all or selected pods on one or more nodes:

     $ oc adm drain <node1> <node2> [--pod-selector=<pod_selector>]

   • Force the deletion of bare pods using the --force option. When set to true, deletion continues even if there are pods not managed by a replication controller, replica set, job, daemon set, or stateful set:

     $ oc adm drain <node1> <node2> --force=true

   • Set a period of time in seconds for each pod to terminate gracefully, use --grace-period. If negative, the default value specified in the pod will be used:

     $ oc adm drain <node1> <node2> --grace-period=-1

   • Ignore pods managed by daemon sets using the --ignore-daemonsets flag set to true:

     $ oc adm drain <node1> <node2> --ignore-daemonsets=true

   • Set the length of time to wait before giving up using the --timeout flag. A value of 0 sets an infinite length of time:

     $ oc adm drain <node1> <node2> --timeout=5s

   • Delete pods even if there are pods using emptyDir volumes by setting the --delete-emptydir-data flag to true. Local data is deleted when the node is drained:

     $ oc adm drain <node1> <node2> --delete-emptydir-data=true

   • List objects that will be migrated without actually performing the evacuation, using the --dry-run option set to true:

     $ oc adm drain <node1> <node2>  --dry-run=true

     Instead of specifying specific node names (for example, <node1> <node2>), you can use the --selector=<node_selector> option to evacuate pods on selected nodes.

3. Mark the node as schedulable when done.

   $ oc adm uncordon <node1>

Procedure

1. Obtain the label associated with the static CRD, Machine Config Pool, for the type of node you want to configure. Perform one of the following steps:

   1. Check current labels of the desired machine config pool.

      For example:

      $  oc get machineconfigpool  --show-labels

      Example output

      NAME      CONFIG                                             UPDATED   UPDATING   DEGRADED   LABELS
      master    rendered-master-e05b81f5ca4db1d249a1bf32f9ec24fd   True      False      False      operator.machineconfiguration.openshift.io/required-for-upgrade=
      worker    rendered-worker-f50e78e1bc06d8e82327763145bfcf62   True      False      False

   2. Add a custom label to the desired machine config pool.

      For example:

      $ oc label machineconfigpool worker custom-kubelet=enabled

2. Create a kubeletconfig custom resource (CR) for your configuration change.

   For example:

   Sample configuration for a custom-config CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-config 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: enabled 2
     kubeletConfig: 3
       podsPerCore: 10
       maxPods: 250
       systemReserved:
         cpu: 2000m
         memory: 1Gi
   #...

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change, this is the label you added to the machine config pool.

   3

   Specify the new value(s) you want to change.

3. Create the CR object.

   $ oc create -f <file-name>

   For example:

   $ oc create -f master-kube-config.yaml

Procedure

1. Edit the schedulers.config.openshift.io resource.

   $ oc edit schedulers.config.openshift.io cluster

2. Configure the mastersSchedulable field.

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     creationTimestamp: "2019-09-10T03:04:05Z"
     generation: 1
     name: cluster
     resourceVersion: "433"
     selfLink: /apis/config.openshift.io/v1/schedulers/cluster
     uid: a636d30a-d377-11e9-88d4-0a60097bee62
   spec:
     mastersSchedulable: false 1
   status: {}
   #...

   1

   Set to true to allow control plane nodes to be schedulable, or false to disallow control plane nodes to be schedulable.

3. Save the file to apply the changes.

Procedure

1. Create a new YAML file with a MachineConfig object, displayed in the following example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-setsebool
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
         - contents: |
             [Unit]
             Description=Set SELinux booleans
             Before=kubelet.service
   
             [Service]
             Type=oneshot
             ExecStart=/sbin/setsebool container_manage_cgroup=on
             RemainAfterExit=true
   
             [Install]
             WantedBy=multi-user.target graphical.target
           enabled: true
           name: setsebool.service
   #...

2. Create the new MachineConfig object by running the following command:

   $ oc create -f 99-worker-setsebool.yaml

Procedure

1. List existing MachineConfig objects for your OpenShift Container Platform cluster to determine how to label your machine config:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

2. Create a MachineConfig object file that identifies the kernel argument (for example, 05-worker-kernelarg-selinuxpermissive.yaml)

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker1
     name: 05-worker-kernelarg-selinuxpermissive2
   spec:
     kernelArguments:
       - enforcing=03

   1

   Applies the new kernel argument only to worker nodes.

   2

   Named to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode).

   3

   Identifies the exact kernel argument as enforcing=0.

3. Create the new machine config:

   $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml

4. Check the machine configs to see that the new one was added:

   $ oc get MachineConfig

   Example output

   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   05-worker-kernelarg-selinuxpermissive                                                         3.2.0             105s
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m

5. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.24.0
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.24.0
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.24.0
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.24.0
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.24.0
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.24.0

   You can see that scheduling on each worker node is disabled as the change is being applied.

6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   BOOT_IMAGE=/ostree/rhcos-... console=tty0 console=ttyS0,115200n8
   rootflags=defaults,prjquota rw root=UUID=fd0... ostree=/ostree/boot.0/rhcos/16...
   coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2 enforcing=0
   
   sh-4.2# exit

   You should see the enforcing=0 argument added to the other kernel arguments.

Procedure

1. Apply a custom label to the machine config pool where you want to allow swap memory.

   $ oc label machineconfigpool worker kubelet-swap=enabled

2. Create a custom resource (CR) to enable and configure swap settings.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: swap-config
   spec:
     machineConfigPoolSelector:
       matchLabels:
         kubelet-swap: enabled
     kubeletConfig:
       failSwapOn: false 1
       memorySwap:
         swapBehavior: LimitedSwap 2
   #...

   1

   Set to false to enable swap memory use on the associated nodes. Set to true to disable swap memory use.

   2

   Specify the swap memory behavior. If unspecified, the default is LimitedSwap.

3. Enable swap memory on the machines.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:

   $ oc edit machineconfigpool <name>

   For example:

   $ oc edit machineconfigpool worker

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2022-11-16T15:34:25Z"
     generation: 4
     labels:
       pools.operator.machineconfiguration.openshift.io/worker: "" 1
     name: worker
   #...

   1

   The label appears under Labels.

   Tip

   If the label is not present, add a key/value pair such as:

   $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a max-pods CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 2
     kubeletConfig:
       podsPerCore: 10 3
       maxPods: 250 4
   #...

   1

   Assign a name to CR.

   2

   Specify the label from the machine config pool.

   3

   Specify the number of pods the node can run based on the number of processor cores on the node.

   4

   Specify the number of pods the node can run to a fixed value, regardless of the properties of the node.

   Note

   Setting podsPerCore to 0 disables this limit.

   In the above example, the default value for podsPerCore is 10 and the default value for maxPods is 250. This means that unless the node has 25 cores or more, by default, podsPerCore will be the limiting factor.

2. Run the following command to create the CR:

   $ oc create -f <file_name>.yaml

Verification

1. List the MachineConfigPool CRDs to see if the change is applied. The UPDATING column reports True if the change is picked up by the Machine Config Controller:

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     False      False
   worker   worker-8cecd1236b33ee3f8a5e   False     True       False

   Once the change is complete, the UPDATED column reports True.

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     True       False
   worker   worker-8cecd1236b33ee3f8a5e   True      False      False

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators OperatorHub.
2. Search for the Self Node Remediation Operator from the list of available Operators, and then click Install.
3. Keep the default selection of Installation mode and namespace to ensure that the Operator is installed to the openshift-operators namespace.
4. Click Install.

1. Navigate to the Operators Installed Operators page.
2. Check that the Operator is installed in the openshift-operators namespace and its status is Succeeded.

1. Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures.
2. Navigate to the Workloads Pods page and check the logs in any pods in the self-node-remediation-controller-manager project that are reporting issues.

Procedure

1. Create a Namespace custom resource (CR) for the Self Node Remediation Operator:

   1. Define the Namespace CR and save the YAML file, for example, self-node-remediation-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: self-node-remediation

   2. To create the Namespace CR, run the following command:

      $ oc create -f self-node-remediation-namespace.yaml

2. Create an OperatorGroup CR:

   1. Define the OperatorGroup CR and save the YAML file, for example, self-node-remediation-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: self-node-remediation-operator
        namespace: self-node-remediation

   2. To create the OperatorGroup CR, run the following command:

      $ oc create -f self-node-remediation-operator-group.yaml

3. Create a Subscription CR:

   1. Define the Subscription CR and save the YAML file, for example, self-node-remediation-subscription.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: self-node-remediation-operator
          namespace: self-node-remediation 1
      spec:
          channel: stable
          installPlanApproval: Manual 2
          name: self-node-remediation-operator
          source: redhat-operators
          sourceNamespace: openshift-marketplace
          package: self-node-remediation

      1

      Specify the Namespace where you want to install the Self Node Remediation Operator. To install the Self Node Remediation Operator in the openshift-operators namespace, specify openshift-operators in the Subscription CR.

      2

      Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2. To create the Subscription CR, run the following command:

      $ oc create -f self-node-remediation-subscription.yaml

Verification

1. Verify that the installation succeeded by inspecting the CSV resource:

   $ oc get csv -n self-node-remediation

   Example output

   NAME                               DISPLAY                          VERSION   REPLACES   PHASE
   self-node-remediation.v.0.4.0      Self Node Remediation Operator   v.0.4.0              Succeeded

2. Verify that the Self Node Remediation Operator is up and running:

   $ oc get deploy -n self-node-remediation

   Example output

   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   self-node-remediation-controller-manager    1/1     1            1           28h

3. Verify that the Self Node Remediation Operator created the SelfNodeRemediationConfig CR:

   $ oc get selfnoderemediationconfig -n self-node-remediation

   Example output

   NAME                           AGE
   self-node-remediation-config   28h

4. Verify that each self node remediation pod is scheduled and running on each worker node:

   $ oc get daemonset -n self-node-remediation

   Example output

   NAME                      DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
   self-node-remediation-ds  3        3        3      3           3          <none>         28h

   Note

   This command is unsupported for the control plane nodes.

Procedure

1. Create a SelfNodeRemediationTemplate CR:

   1. Define the SelfNodeRemediationTemplate CR:

      apiVersion: self-node-remediation.medik8s.io/v1alpha1
      kind: SelfNodeRemediationTemplate
      metadata:
        namespace: openshift-machine-api
        name: selfnoderemediationtemplate-sample
      spec:
        template:
          spec:
            remediationStrategy: ResourceDeletion 1

      1

      Specifies the remediation strategy. The default strategy is ResourceDeletion.

   2. To create the SelfNodeRemediationTemplate CR, run the following command:

      $ oc create -f <snr-name>.yaml

2. Create or update the MachineHealthCheck CR to point to the SelfNodeRemediationTemplate CR:

   1. Define or update the MachineHealthCheck CR:

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineHealthCheck
      metadata:
        name: machine-health-check
        namespace: openshift-machine-api
      spec:
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-machine-role: "worker"
            machine.openshift.io/cluster-api-machine-type: "worker"
        unhealthyConditions:
        - type:    "Ready"
          timeout: "300s"
          status: "False"
        - type:    "Ready"
          timeout: "300s"
          status: "Unknown"
        maxUnhealthy: "40%"
        nodeStartupTimeout: "10m"
        remediationTemplate: 1
          kind: SelfNodeRemediationTemplate
          apiVersion: self-node-remediation.medik8s.io/v1alpha1
          name: selfnoderemediationtemplate-sample

      1

      Specifies the details for the remediation template.

   2. To create a MachineHealthCheck CR, run the following command:

      $ oc create -f <file-name>.yaml

   3. To update a MachineHealthCheck CR, run the following command:

      $ oc apply -f <file-name>.yaml

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators OperatorHub.
2. Search for the Node Health Check Operator, then click Install.
3. Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-operators namespace.
4. Click Install.

1. Navigate to the Operators Installed Operators page.
2. Check that the Operator is installed in the openshift-operators namespace and that its status is Succeeded.

1. Navigate to the Operators Installed Operators page and inspect the Status column for any errors or failures.
2. Navigate to the Workloads Pods page and check the logs in any pods in the openshift-operators project that are reporting issues.

Procedure

1. Create a Namespace custom resource (CR) for the Node Health Check Operator:

   1. Define the Namespace CR and save the YAML file, for example, node-health-check-namespace.yaml:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: node-health-check

   2. To create the Namespace CR, run the following command:

      $ oc create -f node-health-check-namespace.yaml

2. Create an OperatorGroup CR:

   1. Define the OperatorGroup CR and save the YAML file, for example, node-health-check-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: node-health-check-operator
        namespace: node-health-check

   2. To create the OperatorGroup CR, run the following command:

      $ oc create -f node-health-check-operator-group.yaml

3. Create a Subscription CR:

   1. Define the Subscription CR and save the YAML file, for example, node-health-check-subscription.yaml:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: node-health-check-operator
          namespace: node-health-check 1
      spec:
          channel: stable 2
          installPlanApproval: Manual 3
          name: node-healthcheck-operator
          source: redhat-operators
          sourceNamespace: openshift-marketplace
          package: node-healthcheck-operator

      1

      Specify the Namespace where you want to install the Node Health Check Operator. To install the Node Health Check Operator in the openshift-operators namespace, specify openshift-operators in the Subscription CR.

      2

      Specify the channel name for your subscription. To upgrade to the latest version of the Node Health Check Operator, you must manually change the channel name for your subscription from candidate to stable.

      3

      Set the approval strategy to Manual in case your specified version is superseded by a later version in the catalog. This plan prevents an automatic upgrade to a later version and requires manual approval before the starting CSV can complete the installation.

   2. To create the Subscription CR, run the following command:

      $ oc create -f node-health-check-subscription.yaml

Verification

1. Verify that the installation succeeded by inspecting the CSV resource:

   $ oc get csv -n openshift-operators

   Example output

   NAME                              DISPLAY                     VERSION  REPLACES PHASE
   node-healthcheck-operator.v0.2.0. Node Health Check Operator  0.2.0             Succeeded

2. Verify that the Node Health Check Operator is up and running:

   $ oc get deploy -n openshift-operators

   Example output

   NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
   node-health-check-operator-controller-manager  1/1     1            1           10d

1. Edit the node specification to configure pod anti-affinity:

   apiVersion: v1
   kind: Pod
   metadata:
     name: with-pod-antiaffinity
   spec:
     affinity:
       podAntiAffinity: 1
         preferredDuringSchedulingIgnoredDuringExecution: 2
         - weight: 100 3
           podAffinityTerm:
             labelSelector:
               matchExpressions:
               - key: registry 4
                 operator: In 5
                 values:
                 - default
             topologyKey: kubernetes.io/hostname
   #...

   1

   Stanza to configure pod anti-affinity.

   2

   Defines a preferred rule.

   3

   Specifies a weight for a preferred rule. The node with the highest weight is preferred.

   4

   Description of the pod label that determines when the anti-affinity rule applies. Specify a key and value for the label.

   5

   The operator represents the relationship between the label on the existing pod and the set of values in the matchExpression parameters in the specification for the new pod. Can be In, NotIn, Exists, or DoesNotExist.

   This example assumes the container image registry pod has a label of registry=default. Pod anti-affinity can use any Kubernetes match expression.

2. Enable the MatchInterPodAffinity scheduler predicate in the scheduling policy file.
3. Perform a graceful restart of the node.

1. Mark the node as unschedulable:

   $ oc adm cordon <node1>

2. Drain the node to remove all the running pods:

   $ oc adm drain <node1> --ignore-daemonsets --delete-emptydir-data --force

   You might receive errors that pods associated with custom pod disruption budgets (PDB) cannot be evicted.

   Example error

   error when evicting pods/"rails-postgresql-example-1-72v2w" -n "rails" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

   In this case, run the drain command again, adding the disable-eviction flag, which bypasses the PDB checks:

   $ oc adm drain <node1> --ignore-daemonsets --delete-emptydir-data --force --disable-eviction

3. Access the node in debug mode:

   $ oc debug node/<node1>

4. Change your root directory to /host:

   $ chroot /host

5. Restart the node:

   $ systemctl reboot

   In a moment, the node enters the NotReady state.

   Note

   With some single-node OpenShift clusters, the oc commands might not be available after you cordon and drain the node because the openshift-oauth-apiserver pod is not running. You can use SSH to connect to the node and perform the reboot.

   $ ssh core@<master-node>.<cluster_name>.<base_domain>

   $ sudo systemctl reboot

6. After the reboot is complete, mark the node as schedulable by running the following command:

   $ oc adm uncordon <node1>

   Note

   With some single-node OpenShift clusters, the oc commands might not be available after you cordon and drain the node because the openshift-oauth-apiserver pod is not running. You can use SSH to connect to the node and uncordon it.

   $ ssh core@<target_node>

   $ sudo oc adm uncordon <node> --kubeconfig /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig

7. Verify that the node is ready:

   $ oc get node <node1>

   Example output

   NAME    STATUS  ROLES    AGE     VERSION
   <node1> Ready   worker   6d22h   v1.18.3+b0068a8

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:

   $ oc edit machineconfigpool <name>

   For example:

   $ oc edit machineconfigpool worker

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2022-11-16T15:34:25Z"
     generation: 4
     labels:
       pools.operator.machineconfiguration.openshift.io/worker: "" 1
     name: worker
   #...

   1

   The label appears under Labels.

   Tip

   If the label is not present, add a key/value pair such as:

   $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Important

   If there is one file system, or if /var/lib/kubelet and /var/lib/containers/ are in the same file system, the settings with the highest values trigger evictions, as those are met first. The file system triggers the eviction.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 2
     kubeletConfig:
       evictionSoft: 3
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard: 6
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 7
       imageMinimumGCAge: 5m 8
       imageGCHighThresholdPercent: 80 9
       imageGCLowThresholdPercent: 75 10
   #...

   1

   Name for the object.

   2

   Specify the label from the machine config pool.

   3

   For container garbage collection: Type of eviction: evictionSoft or evictionHard.

   4

   For container garbage collection: Eviction thresholds based on a specific eviction trigger signal.

   5

   For container garbage collection: Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   For container garbage collection: Eviction thresholds based on a specific eviction trigger signal. For evictionHard you must specify all of these parameters. If you do not specify all parameters, only the specified parameters are applied and the garbage collection will not function properly.

   7

   For container garbage collection: The duration to wait before transitioning out of an eviction pressure condition.

   8

   For image garbage collection: The minimum age for an unused image before the image is removed by garbage collection.

   9

   For image garbage collection: The percent of disk usage (expressed as an integer) that triggers image garbage collection.

   10

   For image garbage collection: The percent of disk usage (expressed as an integer) that image garbage collection attempts to free.

2. Run the following command to create the CR:

   $ oc create -f <file_name>.yaml

   For example:

   $ oc create -f gc-container.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/gc-container created

Verification

1. Verify that garbage collection is active by entering the following command. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Prerequisites

1. Obtain the label associated with the static MachineConfigPool object for the type of node you want to configure by entering the following command:

   $ oc edit machineconfigpool <name>

   For example:

   $ oc edit machineconfigpool worker

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2022-11-16T15:34:25Z"
     generation: 4
     labels:
       pools.operator.machineconfiguration.openshift.io/worker: "" 1
     name: worker
   #...

   1

   The label appears under Labels.

   Tip

   If an appropriate label is not present, add a key/value pair such as:

   $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change:

   Sample configuration for a resource allocation CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: dynamic-node 1
   spec:
     autoSizingReserved: true 2
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 3
   #...

   1

   Assign a name to CR.

   2

   Add the autoSizingReserved parameter set to true to allow OpenShift Container Platform to automatically determine and allocate the system-reserved resources on the nodes associated with the specified label. To disable automatic allocation on those nodes, set this parameter to false.

   3

   Specify the label from the machine config pool that you configured in the "Prerequisites" section. You can choose any desired labels for the machine config pool, such as custom-kubelet: small-pods, or the default label, pools.operator.machineconfiguration.openshift.io/worker: "".

   The previous example enables automatic resource allocation on all worker nodes. OpenShift Container Platform drains the nodes, applies the kubelet config, and restarts the nodes.

2. Create the CR by entering the following command:

   $ oc create -f <file_name>.yaml

Verification

1. Log in to a node you configured by entering the following command:

   $ oc debug node/<node_name>

2. Set /host as the root directory within the debug shell:

   # chroot /host

3. View the /etc/node-sizing.env file:

   Example output

   SYSTEM_RESERVED_MEMORY=3Gi
   SYSTEM_RESERVED_CPU=0.08

   The kubelet uses the system-reserved values in the /etc/node-sizing.env file. In the previous example, the worker nodes are allocated 0.08 CPU and 3 Gi of memory. It can take several minutes for the optimal values to appear.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:

   $ oc edit machineconfigpool <name>

   For example:

   $ oc edit machineconfigpool worker

   Example output

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     creationTimestamp: "2022-11-16T15:34:25Z"
     generation: 4
     labels:
       pools.operator.machineconfiguration.openshift.io/worker: "" 1
     name: worker
   #...

   1

   The label appears under Labels.

   Tip

   If the label is not present, add a key/value pair such as:

   $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a resource allocation CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-allocatable 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 2
     kubeletConfig:
       systemReserved: 3
         cpu: 1000m
         memory: 1Gi
   #...

   1

   Assign a name to CR.

   2

   Specify the label from the machine config pool.

   3

   Specify the resources to reserve for the node components and system components.

2. Run the following command to create the CR:

   $ oc create -f <file_name>.yaml

Procedure

1. Obtain the label associated with the machine config pool (MCP) for the type of node you want to configure:

   $ oc describe machineconfigpool <name>

   For example:

   $ oc describe machineconfigpool worker

   Example output

   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
                 pools.operator.machineconfiguration.openshift.io/worker= 1
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   #...

   1

   Get the MCP label.

2. Create a YAML file for the KubeletConfig CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-reserved-cpus 1
   spec:
     kubeletConfig:
       reservedSystemCPUs: "0,1,2,3" 2
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 3
   #...

   1

   Specify a name for the CR.

   2

   Specify the core IDs of the CPUs you want to reserve for the nodes associated with the MCP.

   3

   Specify the label from the MCP.

3. Create the CR object:

   $ oc create -f <file_name>.yaml

Procedure

1. Create a KubeletConfig CR to configure the TLS security profile:

   Sample KubeletConfig CR for a Custom profile

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-kubelet-tls-security-profile
   spec:
     tlsSecurityProfile:
       type: Custom 1
       custom: 2
         ciphers: 3
         - ECDHE-ECDSA-CHACHA20-POLY1305
         - ECDHE-RSA-CHACHA20-POLY1305
         - ECDHE-RSA-AES128-GCM-SHA256
         - ECDHE-ECDSA-AES128-GCM-SHA256
         minTLSVersion: VersionTLS11
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 4
   #...

   1

   Specify the TLS security profile type (Old, Intermediate, or Custom). The default is Intermediate.

   2

   Specify the appropriate field for the selected type:

   • old: {}
   • intermediate: {}
   • custom:

   3

   For the custom type, specify a list of TLS ciphers and minimum accepted TLS version.

   4

   Optional: Specify the machine config pool label for the nodes you want to apply the TLS security profile.

2. Create the KubeletConfig object:

   $ oc create -f <filename>

   Depending on the number of worker nodes in the cluster, wait for the configured nodes to be rebooted one by one.

1. Start a debug session for a configured node:

   $ oc debug node/<node_name>

2. Set /host as the root directory within the debug shell:

   sh-4.4# chroot /host

3. View the kubelet.conf file:

   sh-4.4# cat /etc/kubernetes/kubelet.conf

   Example output

     "kind": "KubeletConfiguration",
     "apiVersion": "kubelet.config.k8s.io/v1beta1",
   #...
     "tlsCipherSuites": [
       "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
       "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
       "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
       "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
       "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
     ],
     "tlsMinVersion": "VersionTLS12",
   #...