Chapter 4. Post-installation node tasks

Procedure

1. Ensure that the kubeconfig file for the cluster and the installation program that you used to install the cluster are on the machine. One way to accomplish this is to use the same machine that you used to install the cluster.
2. Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.

3. Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.

   Important

   If you use SSH key-based authentication, you must manage the key with an SSH agent.

4. If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift subscription to it:

   1. Register the machine with RHSM:

      # subscription-manager register --username=<user_name> --password=<password>

   2. Pull the latest subscription data from RHSM:

      # subscription-manager refresh

   3. List the available subscriptions:

      # subscription-manager list --available --matches '*OpenShift*'

   4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

      # subscription-manager attach --pool=<pool_id>

5. Enable the repositories required by OpenShift Container Platform 4.6:

   # subscription-manager repos \
       --enable="rhel-7-server-rpms" \
       --enable="rhel-7-server-extras-rpms" \
       --enable="rhel-7-server-ansible-2.9-rpms" \
       --enable="rhel-7-server-ose-4.6-rpms"

6. Install the required packages, including openshift-ansible:

   # yum install openshift-ansible openshift-clients jq

   The openshift-ansible package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients provides the oc CLI, and the jq package improves the display of JSON output on your command line.

1. On each host, register with RHSM:

   # subscription-manager register --username=<user_name> --password=<password>

2. Pull the latest subscription data from RHSM:

   # subscription-manager refresh

3. List the available subscriptions:

   # subscription-manager list --available --matches '*OpenShift*'

4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

   # subscription-manager attach --pool=<pool_id>

5. Disable all yum repositories:

   1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"

   2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist

   3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

      # yum-config-manager --disable \*

      Note that this might take a few minutes if you have a large number of available repositories

6. Enable only the repositories required by OpenShift Container Platform 4.6:

   # subscription-manager repos \
       --enable="rhel-7-server-rpms" \
       --enable="rhel-7-fast-datapath-rpms" \
       --enable="rhel-7-server-extras-rpms" \
       --enable="rhel-7-server-optional-rpms" \
       --enable="rhel-7-server-ose-4.6-rpms"

7. Stop and disable firewalld on the host:

   # systemctl disable --now firewalld.service

   Note

   You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

1. Create an Ansible inventory file that is named /<path>/inventory/hosts that defines your compute machine hosts and required variables:

   [all:vars]
   ansible_user=root 1
   #ansible_become=True 2
   
   openshift_kubeconfig_path="~/.kube/config" 3
   
   [new_workers] 4
   mycluster-rhel7-0.example.com
   mycluster-rhel7-1.example.com

   1

   Specify the user name that runs the Ansible tasks on the remote compute machines.

   2

   If you do not specify root for the ansible_user, you must set ansible_become to True and assign the user sudo permissions.

   3

   Specify the path and file name of the kubeconfig file for your cluster.

   4

   List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the hostname that the cluster uses to access the machine, so set the correct public or private name to access the machine.

2. Navigate to the Ansible playbook directory:

   $ cd /usr/share/ansible/openshift-ansible

3. Run the playbook:

   $ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml 1

   1

   For <path>, specify the path to the Ansible inventory file that you created.

Procedure

1. View the list of machines and record the node names of the RHCOS compute machines:

   $ oc get nodes -o wide

2. For each RHCOS compute machine, delete the node:

   1. Mark the node as unschedulable by running the oc adm cordon command:

      $ oc adm cordon <node_name> 1

      1

      Specify the node name of one of the RHCOS compute machines.

   2. Drain all the pods from the node:

      $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets 1

      1

      Specify the node name of the RHCOS compute machine that you isolated.

   3. Delete the node:

      $ oc delete nodes <node_name> 1

      1

      Specify the node name of the RHCOS compute machine that you drained.

3. Review the list of compute machines to ensure that only the RHEL nodes remain:

   $ oc get nodes -o wide

4. Remove the RHCOS machines from the load balancer for your cluster’s compute machines. You can delete the virtual machines or reimage the physical hardware for the RHCOS compute machines.

Procedure

1. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection with a LOM interface.
2. After the instance boots, press the TAB or E key to edit the kernel command line.

3. Add the parameters to the kernel command line:

   coreos.inst.install_dev=sda 1
   coreos.inst.ignition_url=http://example.com/worker.ign 2

   1

   Specify the block device of the system to install to.

   2

   Specify the URL of the compute Ignition config file. Only HTTP and HTTPS protocols are supported.

4. Press Enter to complete the installation. After RHCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.
5. Continue to create more compute machines for your cluster.

Procedure

1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> 1
         APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img 2

     1

     Specify the location of the live kernel file that you uploaded to your HTTP server.

     2

     Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

1. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.19.0
   master-1  Ready     master  63m  v1.19.0
   master-2  Ready     master  64m  v1.19.0

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. Once the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.20.0
   master-1  Ready     master  73m  v1.20.0
   master-2  Ready     master  74m  v1.20.0
   worker-0  Ready     worker  11m  v1.20.0
   worker-1  Ready     worker  11m  v1.20.0

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Create a healthcheck.yml file that contains the definition of your machine health check.

2. Apply the healthcheck.yml file to your cluster:

   $ oc apply -f healthcheck.yml

Procedure

1. View the machine sets that are in the cluster:

   $ oc get machinesets -n openshift-machine-api

   The machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.

2. View the machines that are in the cluster:

   $ oc get machine -n openshift-machine-api

3. Set the annotation on the machine that you want to delete:

   $ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/cluster-api-delete-machine="true"

4. Cordon and drain the node that you want to delete:

   $ oc adm cordon <node_name>
   $ oc adm drain <node_name>

5. Scale the machine set:

   $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

   Or:

   $ oc edit machineset <machineset> -n openshift-machine-api

   You can scale the machine set up or down. It takes several minutes for the new machines to be available.

Procedure

1. View the available machine configuration objects that you can select:

   $ oc get machineconfig

   By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. To check the current value of max pods per node, run:

   # oc describe node <node-ip> | grep Allocatable -A6

   Look for value: pods: <value>.

   For example:

   # oc describe node ip-172-31-128-158.us-east-2.compute.internal | grep Allocatable -A6

   Example output

   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

3. To set the max pods per node on the worker nodes, create a custom resource file that contains the kubelet configuration. For example, change-maxPods-cr.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: large-pods
     kubeletConfig:
       maxPods: 500

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 50 for kubeAPIQPS and 100 for kubeAPIBurst, are good enough if there are limited pods running on each node. Updating the kubelet QPS and burst rates is recommended if there are enough CPU and memory resources on the node:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: large-pods
     kubeletConfig:
       maxPods: <pod_count>
       kubeAPIBurst: <burst_rate>
       kubeAPIQPS: <QPS>

   1. Update the machine config pool for workers with the label:

      $ oc label machineconfigpool worker custom-kubelet=large-pods

   2. Create the KubeletConfig object:

      $ oc create -f change-maxPods-cr.yaml

   3. Verify that the KubeletConfig object is created:

      $ oc get kubeletconfig

      This should return set-max-pods.

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Check for maxPods changing for the worker nodes:

   $ oc describe node

   1. Verify the change by running:

      $ oc get kubeletconfigs set-max-pods -o yaml

      This should show a status of True and type:Success

1. Edit the worker machine config pool:

   $ oc edit machineconfigpool worker

2. Set maxUnavailable to the desired value.

   spec:
     maxUnavailable: <node_count>

   Important

   When setting the value, consider the number of worker nodes that can be unavailable without affecting the applications running on the cluster.

Procedure

1. Optional: Label a node:

   # oc label node perf-node.example.com cpumanager=true

2. Edit the MachineConfigPool of the nodes where CPU Manager should be enabled. In this example, all workers have CPU Manager enabled:

   # oc edit machineconfigpool worker

3. Add a label to the worker machine config pool:

   metadata:
     creationTimestamp: 2020-xx-xxx
     generation: 3
     labels:
       custom-kubelet: cpumanager-enabled

4. Create a KubeletConfig, cpumanager-kubeletconfig.yaml, custom resource (CR). Refer to the label created in the previous step to have the correct nodes updated with the new kubelet config. See the machineConfigPoolSelector section:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static 1
        cpuManagerReconcilePeriod: 5s 2

   1

   Specify a policy:

   • none. This policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the scheduler does automatically.
   • static. This policy allows pods with certain resource characteristics to be granted increased CPU affinity and exclusivity on the node.

   2

   Optional. Specify the CPU Manager reconcile frequency. The default is 5s.

5. Create the dynamic kubelet config:

   # oc create -f cpumanager-kubeletconfig.yaml

   This adds the CPU Manager feature to the kubelet config and, if needed, the Machine Config Operator (MCO) reboots the node. To enable CPU Manager, a reboot is not needed.

6. Check for the merged kubelet config:

   # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7

   Example output

          "ownerReferences": [
               {
                   "apiVersion": "machineconfiguration.openshift.io/v1",
                   "kind": "KubeletConfig",
                   "name": "cpumanager-enabled",
                   "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878"
               }
           ]

7. Check the worker for the updated kubelet.conf:

   # oc debug node/perf-node.example.com
   sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager

   Example output

   cpuManagerPolicy: static        1
   cpuManagerReconcilePeriod: 5s   2

   1 2

   These settings were defined when you created the KubeletConfig CR.

8. Create a pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this pod:

   # cat cpumanager-pod.yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: cpumanager-
   spec:
     containers:
     - name: cpumanager
       image: gcr.io/google_containers/pause-amd64:3.0
       resources:
         requests:
           cpu: 1
           memory: "1G"
         limits:
           cpu: 1
           memory: "1G"
     nodeSelector:
       cpumanager: "true"

9. Create the pod:

   # oc create -f cpumanager-pod.yaml

10. Verify that the pod is scheduled to the node that you labeled:

    # oc describe pod cpumanager

    Example output

    Name:               cpumanager-6cqz7
    Namespace:          default
    Priority:           0
    PriorityClassName:  <none>
    Node:  perf-node.example.com/xxx.xx.xx.xxx
    ...
     Limits:
          cpu:     1
          memory:  1G
        Requests:
          cpu:        1
          memory:     1G
    ...
    QoS Class:       Guaranteed
    Node-Selectors:  cpumanager=true

11. Verify that the cgroups are set up correctly. Get the process ID (PID) of the pause process:

    # ├─init.scope
    │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
    └─kubepods.slice
      ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice
      │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope
      │ └─32706 /pause

    Pods of quality of service (QoS) tier Guaranteed are placed within the kubepods.slice. Pods of other QoS tiers end up in child cgroups of kubepods:

    # cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope
    # for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done

    Example output

    cpuset.cpus 1
    tasks 32706

12. Check the allowed CPU list for the task:

    # grep ^Cpus_allowed_list /proc/32706/status

    Example output

     Cpus_allowed_list:    1

13. Verify that another pod (in this case, the pod in the burstable QoS tier) on the system cannot run on the core allocated for the Guaranteed pod:

    # cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus
    0
    # oc describe node perf-node.example.com

    Example output

    ...
    Capacity:
     attachable-volumes-aws-ebs:  39
     cpu:                         2
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      8162900Ki
     pods:                        250
    Allocatable:
     attachable-volumes-aws-ebs:  39
     cpu:                         1500m
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      7548500Ki
     pods:                        250
    -------                               ----                           ------------  ----------  ---------------  -------------  ---
      default                                 cpumanager-6cqz7               1 (66%)       1 (66%)     1G (12%)         1G (12%)       29m
    
    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      Resource                    Requests          Limits
      --------                    --------          ------
      cpu                         1440m (96%)       1 (66%)

    This VM has two CPU cores. The system-reserved setting reserves 500 millicores, meaning that half of one core is subtracted from the total capacity of the node to arrive at the Node Allocatable amount. You can see that Allocatable CPU is 1500 millicores. This means you can run one of the CPU Manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores. If you try to schedule a second pod, the system will accept the pod, but it will never be scheduled:

    NAME                    READY   STATUS    RESTARTS   AGE
    cpumanager-6cqz7        1/1     Running   0          33m
    cpumanager-7qc2t        0/1     Pending   0          11s

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config:

      # oc describe machineconfig <name>

      For example:

      # oc describe machineconfig 00-worker

      Example output

      Name:         00-worker
      Namespace:
      Labels:       machineconfiguration.openshift.io/role=worker 1

      1

      Label required for the Device Manager.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a Device Manager CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: devicemgr 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
          machineconfiguration.openshift.io: devicemgr 2
     kubeletConfig:
       feature-gates:
         - DevicePlugins=true 3

   1

   Assign a name to CR.

   2

   Enter the label from the Machine Config Pool.

   3

   Set DevicePlugins to 'true`.

2. Create the Device Manager:

   $ oc create -f devicemgr.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/devicemgr created

3. Ensure that Device Manager was actually enabled by confirming that /var/lib/kubelet/device-plugins/kubelet.sock is created on the node. This is the UNIX domain socket on which the Device Manager gRPC server listens for new plug-in registrations. This sock file is created when the Kubelet is started only if Device Manager is enabled.

1. If you add a NoSchedule taint to a control plane node (also known as the master node) the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with an Equal operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1" 1
           value: "value1"
           operator: "Equal"
           effect: "NoExecute"
           tolerationSeconds: 3600 2

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod can remain bound to a node before being evicted.

   For example:

   Sample pod configuration file with an Exists operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1"
           operator: "Exists" 1
           effect: "NoExecute"
           tolerationSeconds: 3600

   1

   The Exists operator does not take a value.

   This example places a taint on node1 that has key key1, value value1, and taint effect NoExecute.

2. Add a taint to a node by using the following command with the parameters described in the Taint and toleration components table:

   $ oc adm taint nodes <node_name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 key1=value1:NoExecute

   This command places a taint on node1 that has key key1, value value1, and effect NoExecute.

   Note

   If you add a NoSchedule taint to a control plane node (also known as the master node) the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

   The tolerations on the pod match the taint on the node. A pod with either toleration can be scheduled onto node1.

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with Equal operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1" 1
           value: "value1"
           operator: "Equal"
           effect: "NoExecute"
           tolerationSeconds: 3600 2

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod is bound to a node before being evicted.

   For example:

   Sample pod configuration file with Exists operator

   spec:
     tolerations:
     - key: "key1"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600

2. Add the taint to the MachineSet object:

   1. Edit the MachineSet YAML for the nodes you want to taint or you can create a new MachineSet object:

      $ oc edit machineset <machineset>

   2. Add the taint to the spec.template.spec section:

      Example taint in a machine set specification

      spec:
      ....
        template:
      ....
          spec:
            taints:
            - effect: NoExecute
              key: key1
              value: value1
      ....

      This example places a taint that has the key key1, value value1, and taint effect NoExecute on the nodes.

   3. Scale down the machine set to 0:

      $ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api

      Wait for the machines to be removed.

   4. Scale up the machine set as needed:

      $ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

      Wait for the machines to start. The taint is added to the nodes associated with the MachineSet object.

1. Add a corresponding taint to those nodes:

   For example:

   $ oc adm taint nodes node1 dedicated=groupName:NoSchedule

2. Add a toleration to the pods by writing a custom admission controller.

1. Add a toleration to pods that need the special hardware.

   For example:

   spec:
     tolerations:
       - key: "disktype"
         value: "ssd"
         operator: "Equal"
         effect: "NoSchedule"
         tolerationSeconds: 3600

2. Taint the nodes that have the specialized hardware using one of the following commands:

   $ oc adm taint nodes <node-name> disktype=ssd:NoSchedule

   Or:

   $ oc adm taint nodes <node-name> disktype=ssd:PreferNoSchedule

1. To remove a taint from a node:

   $ oc adm taint nodes <node-name> <key>-

   For example:

   $ oc adm taint nodes ip-10-0-132-248.ec2.internal key1-

   Example output

   node/ip-10-0-132-248.ec2.internal untainted

2. To remove a toleration from a pod, edit the Pod spec to remove the toleration:

   spec:
     tolerations:
     - key: "key2"
       operator: "Exists"
       effect: "NoExecute"
       tolerationSeconds: 3600

1. Configure the Topology Manager allocation policy in the cpumanager-enabled custom resource (CR).

   $ oc edit KubeletConfig cpumanager-enabled

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static 1
        cpuManagerReconcilePeriod: 5s
        topologyManagerPolicy: single-numa-node 2

   1

   This parameter must be static.

   2

   Specify your selected Topology Manager allocation policy. Here, the policy is single-numa-node. Acceptable values are: default, best-effort, restricted, single-numa-node.

1. In the OpenShift Container Platform web console, navigate to Home Projects

   1. Click Create Project.
   2. Specify clusterresourceoverride-operator as the name of the project.
   3. Click Create.

2. Navigate to Operators OperatorHub.

   1. Choose ClusterResourceOverride Operator from the list of available Operators and click Install.
   2. On the Install Operator page, make sure A specific Namespace on the cluster is selected for Installation Mode.
   3. Make sure clusterresourceoverride-operator is selected for Installed Namespace.
   4. Select an Update Channel and Approval Strategy.
   5. Click Install.

3. On the Installed Operators page, click ClusterResourceOverride.

   1. On the ClusterResourceOverride Operator details page, click Create Instance.

   2. On the Create ClusterResourceOverride page, edit the YAML template to set the overcommit values as needed:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Click Create.

4. Check the current state of the admission webhook by checking the status of the cluster custom resource:

   1. On the ClusterResourceOverride Operator page, click cluster.

   2. On the ClusterResourceOverride Details age, click YAML. The mutatingWebhookConfigurationRef section appears when the webhook is called.

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
        creationTimestamp: "2019-12-18T22:35:02Z"
        generation: 1
        name: cluster
        resourceVersion: "127622"
        selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
        uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
      spec:
        podResourceOverride:
          spec:
            cpuRequestToLimitPercent: 25
            limitCPUToMemoryPercent: 200
            memoryRequestToLimitPercent: 50
      status:
      
      ....
      
          mutatingWebhookConfigurationRef: 1
            apiVersion: admissionregistration.k8s.io/v1beta1
            kind: MutatingWebhookConfiguration
            name: clusterresourceoverrides.admission.autoscaling.openshift.io
            resourceVersion: "127621"
            uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
      
      ....

      1

      Reference to the ClusterResourceOverride admission webhook.

1. Create a namespace for the Cluster Resource Override Operator:

   1. Create a Namespace object YAML file (for example, cro-namespace.yaml) for the Cluster Resource Override Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: clusterresourceoverride-operator

   2. Create the namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-namespace.yaml

2. Create an Operator group:

   1. Create an OperatorGroup object YAML file (for example, cro-og.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: clusterresourceoverride-operator
        namespace: clusterresourceoverride-operator
      spec:
        targetNamespaces:
          - clusterresourceoverride-operator

   2. Create the Operator Group:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-og.yaml

3. Create a subscription:

   1. Create a Subscription object YAML file (for example, cro-sub.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: clusterresourceoverride
        namespace: clusterresourceoverride-operator
      spec:
        channel: "4.6"
        name: clusterresourceoverride
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the subscription:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-sub.yaml

4. Create a ClusterResourceOverride custom resource (CR) object in the clusterresourceoverride-operator namespace:

   1. Change to the clusterresourceoverride-operator namespace.

      $ oc project clusterresourceoverride-operator

   2. Create a ClusterResourceOverride object YAML file (for example, cro-cr.yaml) for the Cluster Resource Override Operator:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
          name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Create the ClusterResourceOverride object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-cr.yaml

5. Verify the current state of the admission webhook by checking the status of the cluster custom resource.

   $ oc get clusterresourceoverride cluster -n clusterresourceoverride-operator -o yaml

   The mutatingWebhookConfigurationRef section appears when the webhook is called.

   Example output

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
     creationTimestamp: "2019-12-18T22:35:02Z"
     generation: 1
     name: cluster
     resourceVersion: "127622"
     selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
     uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
   spec:
     podResourceOverride:
       spec:
         cpuRequestToLimitPercent: 25
         limitCPUToMemoryPercent: 200
         memoryRequestToLimitPercent: 50
   status:
   
   ....
   
       mutatingWebhookConfigurationRef: 1
         apiVersion: admissionregistration.k8s.io/v1beta1
         kind: MutatingWebhookConfiguration
         name: clusterresourceoverrides.admission.autoscaling.openshift.io
         resourceVersion: "127621"
         uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
   
   ....

   1

   Reference to the ClusterResourceOverride admission webhook.

1. Edit the ClusterResourceOverride CR:

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
       name: cluster
   spec:
     podResourceOverride:
       spec:
         memoryRequestToLimitPercent: 50 1
         cpuRequestToLimitPercent: 25 2
         limitCPUToMemoryPercent: 200 3

   1

   Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

   2

   Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

   3

   Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

2. Ensure the following label has been added to the Namespace object for each project where you want the Cluster Resource Override Operator to control overcommit:

   apiVersion: v1
   kind: Namespace
   metadata:
   
    ...
   
     labels:
       clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true" 1
   
    ...

   1

   Add this label to each project.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a disabling CPU limits

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: disable-cpu-units 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       cpuCfsQuota: 3
         - "false"

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change.

   3

   Set the cpuCfsQuota parameter to false.

1. Edit the project object file

2. Add the following annotation:

   quota.openshift.io/cluster-resource-override-enabled: "false"

3. Create the project object:

   $ oc create -f <file-name>.yaml

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      Name:         worker
      Namespace:
      Labels:       custom-kubelet=small-pods 1

      1

      If a label has been added it appears under Labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Important

   If there is one file system, or if /var/lib/kubelet and /var/lib/containers/ are in the same file system, the settings with the highest values trigger evictions, as those are met first. The file system triggers the eviction.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       evictionSoft: 3
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard: 6
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 7
       imageMinimumGCAge: 5m 8
       imageGCHighThresholdPercent: 80 9
       imageGCLowThresholdPercent: 75 10

   1

   Name for the object.

   2

   Selector label.

   3

   Type of eviction: evictionSoft or evictionHard.

   4

   Eviction thresholds based on a specific eviction trigger signal.

   5

   Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   Eviction thresholds based on a specific eviction trigger signal. For evictionHard you must specify all of these parameters. If you do not specify all parameters, only the specified parameters are applied and the garbage collection will not function properly.

   7

   The duration to wait before transitioning out of an eviction pressure condition.

   8

   The minimum age for an unused image before the image is removed by garbage collection.

   9

   The percent of disk usage (expressed as an integer) that triggers image garbage collection.

   10

   The percent of disk usage (expressed as an integer) that image garbage collection attempts to free.

2. Create the object:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f gc-container.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/gc-container created

3. Verify that garbage collection is active. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Procedure

1. Run:

   $ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator