Chapter 4. Working with nodes

Procedure

1. Mark the nodes unschedulable before performing the pod evacuation.

   1. Mark the node as unschedulable:

      $ oc adm cordon <node1>

      Example output

      node/<node1> cordoned

   2. Check that the node status is NotReady,SchedulingDisabled:

      $ oc get node <node1>

      Example output

      NAME        STATUS                        ROLES     AGE       VERSION
      <node1>     NotReady,SchedulingDisabled   worker    1d        v1.18.3

2. Evacuate the pods using one of the following methods:

   • Evacuate all or selected pods on one or more nodes:

     $ oc adm drain <node1> <node2> [--pod-selector=<pod_selector>]

   • Force the deletion of bare pods using the --force option. When set to true, deletion continues even if there are pods not managed by a replication controller, replica set, job, daemon set, or stateful set:

     $ oc adm drain <node1> <node2> --force=true

   • Set a period of time in seconds for each Pod to terminate gracefully, use --grace-period. If negative, the default value specified in the Pod will be used:

     $ oc adm drain <node1> <node2> --grace-period=-1

   • Ignore pods managed by daemon sets using the --ignore-daemonsets flag set to true:

     $ oc adm drain <node1> <node2> --ignore-daemonsets=true

   • Set the length of time to wait before giving up using the --timeout flag. A value of 0 sets an infinite length of time:

     $ oc adm drain <node1> <node2> --timeout=5s

   • Delete pods even if there are pods using emptyDir using the --delete-local-data flag set to true. Local data is deleted when the node is drained:

     $ oc adm drain <node1> <node2> --delete-local-data=true

   • List objects that will be migrated without actually performing the evacuation, using the --dry-run option set to true:

     $ oc adm drain <node1> <node2>  --dry-run=true

     Instead of specifying specific node names (for example, <node1> <node2>), you can use the --selector=<node_selector> option to evacuate pods on selected nodes.

3. Mark the node as schedulable when done.

   $ oc adm uncordon <node1>

Procedure

1. Edit the schedulers.config.openshift.io resource.

   $ oc edit schedulers.config.openshift.io cluster

2. Configure the mastersSchedulable field.

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     creationTimestamp: "2019-09-10T03:04:05Z"
     generation: 1
     name: cluster
     resourceVersion: "433"
     selfLink: /apis/config.openshift.io/v1/schedulers/cluster
     uid: a636d30a-d377-11e9-88d4-0a60097bee62
   spec:
     mastersSchedulable: false 1
     policy:
       name: ""
   status: {}

   1

   Set to true to allow master nodes to be schedulable, or false to disallow master nodes to be schedulable.

3. Save the file to apply the changes.

Procedure

1. List existing MachineConfig objects for your OpenShift Container Platform cluster to determine how to label your machine config:

   $ oc get MachineConfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   00-worker                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-master-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-master-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-worker-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-worker-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-master-1131169f-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-master-ssh                                                                                          2.2.0             30m
   99-worker-114e8ac7-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-worker-ssh                                                                                          2.2.0             30m
   rendered-master-b3729e5f6124ca3678188071343115d0            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   rendered-worker-18ff9506c718be1e8bd0a066850065b7            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m

2. Create a MachineConfig object file that identifies the kernel argument (for example, 05-worker-kernelarg-selinuxpermissive.yaml)

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker1
     name: 05-worker-kernelarg-selinuxpermissive2
   spec:
     config:
       ignition:
         version: 2.2.0
     kernelArguments:
       - enforcing=03

   1

   Applies the new kernel argument only to worker nodes.

   2

   Named to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode).

   3

   Identifies the exact kernel argument as enforcing=0.

3. Create the new machine config:

   $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml

4. Check the machine configs to see that the new one was added:

   $ oc get MachineConfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   00-worker                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-master-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-master-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-worker-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-worker-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   
   05-worker-kernelarg-selinuxpermissive                                                                  3.1.0             105s
   
   99-master-1131169f-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   99-master-ssh                                                                                          2.2.0             30m
   99-worker-114e8ac7-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   99-worker-ssh                                                                                          2.2.0             31m
   rendered-master-b3729e5f6124ca3678188071343115d0            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   rendered-worker-18ff9506c718be1e8bd0a066850065b7            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m

5. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.18.3
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.18.3
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.18.3
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.18.3
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.18.3
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.18.3

   You can see that scheduling on each worker node is disabled as the change is being applied.

6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   BOOT_IMAGE=/ostree/rhcos-... console=tty0 console=ttyS0,115200n8
   rootflags=defaults,prjquota rw root=UUID=fd0... ostree=/ostree/boot.0/rhcos/16...
   coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2 enforcing=0
   
   sh-4.2# exit

   You should see the enforcing=0 argument added to the other kernel arguments.

Procedure

1. Obtain the label associated with the static CRD, Machine Config Pool, for the type of node you want to configure. Perform one of the following steps:

   1. Check current labels of the desired machine config pool.

      For example:

      $  oc get machineconfigpool  --show-labels

      Example output

      NAME      CONFIG                                             UPDATED   UPDATING   DEGRADED   LABELS
      master    rendered-master-e05b81f5ca4db1d249a1bf32f9ec24fd   True      False      False      operator.machineconfiguration.openshift.io/required-for-upgrade=
      worker    rendered-worker-f50e78e1bc06d8e82327763145bfcf62   True      False      False

   2. Add a custom label to the desired machine config pool.

      For example:

      $ oc label machineconfigpool worker custom-kubelet=enabled

2. Create a kubeletconfig custom resource (CR) for your configuration change.

   For example:

   Sample configuration for a custom-config CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-config 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: enabled 2
     kubeletConfig: 3
       podsPerCore: 10
       maxPods: 250
       systemReserved:
         cpu: 2000m
         memory: 1Gi

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change, this is the label you added to the machine config pool.

   3

   Specify the new value(s) you want to change.

3. Create the CR object.

   $ oc create -f <file-name>

   For example:

   $ oc create -f master-kube-config.yaml

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a max-pods CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       podsPerCore: 10 3
       maxPods: 250 4

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change.

   3

   Specify the number of pods the node can run based on the number of processor cores on the node.

   4

   Specify the number of pods the node can run to a fixed value, regardless of the properties of the node.

   Note

   Setting podsPerCore to 0 disables this limit.

   In the above example, the default value for podsPerCore is 10 and the default value for maxPods is 250. This means that unless the node has 25 cores or more, by default, podsPerCore will be the limiting factor.

2. List the MachineConfigPool CRDs to see if the change is applied. The UPDATING column reports True if the change is picked up by the Machine Config Controller:

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     False      False
   worker   worker-8cecd1236b33ee3f8a5e   False     True       False

   Once the change is complete, the UPDATED column reports True.

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     True       False
   worker   worker-8cecd1236b33ee3f8a5e   True      False      False

Procedure

1. Run:

   $ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator

1. Edit the node specification to configure pod anti-affinity:

   apiVersion: v1
   kind: Pod
   metadata:
     name: with-pod-antiaffinity
   spec:
     affinity:
       podAntiAffinity: 1
         preferredDuringSchedulingIgnoredDuringExecution: 2
         - weight: 100 3
           podAffinityTerm:
             labelSelector:
               matchExpressions:
               - key: registry 4
                 operator: In 5
                 values:
                 - default
             topologyKey: kubernetes.io/hostname

   1

   Stanza to configure pod anti-affinity.

   2

   Defines a preferred rule.

   3

   Specifies a weight for a preferred rule. The node with the highest weight is preferred.

   4

   Description of the pod label that determines when the anti-affinity rule applies. Specify a key and value for the label.

   5

   The operator represents the relationship between the label on the existing pod and the set of values in the matchExpression parameters in the specification for the new pod. Can be In, NotIn, Exists, or DoesNotExist.

   This example assumes the container image registry pod has a label of registry=default. Pod anti-affinity can use any Kubernetes match expression.

2. Enable the MatchInterPodAffinity scheduler predicate in the scheduling policy file.

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      Name:         worker
      Namespace:
      Labels:       custom-kubelet=small-pods 1

      1

      If a label has been added it appears under Labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       evictionSoft: 3
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard:
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 6
       imageMinimumGCAge: 5m 7
       imageGCHighThresholdPercent: 80 8
       imageGCLowThresholdPercent: 75 9

   1

   Name for the object.

   2

   Selector label.

   3

   Type of eviction: EvictionSoft and EvictionHard.

   4

   Eviction thresholds based on a specific eviction trigger signal.

   5

   Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   The duration to wait before transitioning out of an eviction pressure condition

   7

   The minimum age for an unused image before the image is removed by garbage collection.

   8

   The percent of disk usage (expressed as an integer) which triggers image garbage collection.

   9

   The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free.

2. Create the object:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f gc-container.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/gc-container created

3. Verify that garbage collection is active. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Prerequisites

1. To help you determine values for the system-reserved setting, you can introspect the resource use for a node by using the node summary API. Enter the following command for your node:

   $ oc get --raw /api/v1/nodes/<node>/proxy/stats/summary

   For example, to access the resources from cluster.node22 node, you can enter:

   $ oc get --raw /api/v1/nodes/cluster.node22/proxy/stats/summary

   Example output

   {
       "node": {
           "nodeName": "cluster.node22",
           "systemContainers": [
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 929684480915,
                       "usageNanoCores": 190998084
                   },
                   "memory": {
                       "rssBytes": 176726016,
                       "usageBytes": 1397895168,
                       "workingSetBytes": 1050509312
                   },
                   "name": "kubelet"
               },
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 128521955903,
                       "usageNanoCores": 5928600
                   },
                   "memory": {
                       "rssBytes": 35958784,
                       "usageBytes": 129671168,
                       "workingSetBytes": 102416384
                   },
                   "name": "runtime"
               }
           ]
       }
   }

2. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the Machine Config Pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a resource allocation CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-allocatable 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       systemReserved:
         cpu: 1000m
         memory: 1Gi

   1

   Assign a name to CR.

   2

   Specify the label from the Machine Config Pool.

Procedure

1. Obtain the label associated with the machine config pool (MCP) for the type of node you want to configure:

   $ oc describe machineconfigpool <name>

   For example:

   $ oc describe machineconfigpool worker

   Example output

   Name:         worker
   Namespace:
   Labels:       machineconfiguration.openshift.io/mco-built-in=
                 pools.operator.machineconfiguration.openshift.io/worker= 1
   Annotations:  <none>
   API Version:  machineconfiguration.openshift.io/v1
   Kind:         MachineConfigPool
   ...

   1

   Get the MCP label.

2. Create a YAML file for the KubeletConfig CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-reserved-cpus 1
   spec:
     kubeletConfig:
       reservedSystemCPUs: "0,1,2,3" 2
     machineConfigPoolSelector:
       matchLabels:
         pools.operator.machineconfiguration.openshift.io/worker: "" 3

   1

   Specify a name for the CR.

   2

   Specify the core IDs of the CPUs you want to reserve for the nodes associated with the MCP.

   3

   Specify the label from the MCP.

3. Create the CR object:

   $ oc create -f <file_name>.yaml