Nodes

1. Change to the project:

   $ oc project <project-name>

2. Run the following command:

   $ oc get pods

   For example:

   $ oc get pods -n openshift-console

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   console-698d866b78-bnshf   1/1     Running   2          165m
   console-698d866b78-m87pm   1/1     Running   2          165m

   Add the -o wide flags to view the pod IP address and the node where the pod is located.

   $ oc get pods -o wide

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE                           NOMINATED NODE
   console-698d866b78-bnshf   1/1     Running   2          166m   10.128.0.24   ip-10-0-152-71.ec2.internal    <none>
   console-698d866b78-m87pm   1/1     Running   2          166m   10.129.0.23   ip-10-0-173-237.ec2.internal   <none>

1. Run the following command:

   $ oc adm top pods

   For example:

   $ oc adm top pods -n openshift-console

   Example output

   NAME                         CPU(cores)   MEMORY(bytes)
   console-7f58c69899-q8c8k     0m           22Mi
   console-7f58c69899-xhbgg     0m           25Mi
   downloads-594fcccf94-bcxk8   3m           18Mi
   downloads-594fcccf94-kv4p6   2m           15Mi

2. Run the following command to view the usage statistics for pods with labels:

   $ oc adm top pod --selector=''

   You must choose the selector (label query) to filter on. Supports =, ==, and !=.

Procedure (UI)

1. In the OpenShift Container Platform console, navigate to Workloads → Pods or navigate to the pod through the resource you want to investigate.

   Note

   Some resources, such as builds, do not have pods to query directly. In such instances, you can locate the Logs link on the Details page for the resource.

2. Select a project from the drop-down menu.
3. Click the name of the pod you want to investigate.
4. Click Logs.

1. Write an object definition JSON file, and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod Object Definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

1. Create a Pod spec or edit existing pods to include the system-cluster-critical priority class:

   spec:
     template:
       metadata:
         name: critical-pod
       priorityClassName: system-cluster-critical 1

   1

   Default priority class for pods that should never be evicted from a node.

   Alternatively, you can specify system-node-critical for pods that are important to the cluster but can be removed if necessary.

2. Create the pod:

   $ oc create -f <file-name>.yaml

1. Perform one of the following one of the following:

   • To scale based on the percent of CPU utilization, create a HorizontalPodAutoscaler object for an existing DeploymentConfig object:

     $ oc autoscale dc/<dc-name> \1
       --min <number> \2
       --max <number> \3
       --cpu-percent=<percent> 4

     1

     Specify the name of the DeploymentConfig object. The object must exist.

     2

     Optionally, specify the minimum number of replicas when scaling down.

     3

     Specify the maximum number of replicas when scaling up.

     4

     Specify the target average CPU utilization over all the pods, represented as a percent of requested CPU. If not specified or negative, a default autoscaling policy is used.

   • To scale based on the percent of CPU utilization, create a HorizontalPodAutoscaler object for an existing replication controller:

     $ oc autoscale rc/<rc-name> 1
       --min <number> \2
       --max <number> \3
       --cpu-percent=<percent> 4

     1

     Specify the name of the replication controller. The object must exist.

     2

     Specify the minimum number of replicas when scaling down.

     3

     Specify the maximum number of replicas when scaling up.

     4

     Specify the target average CPU utilization over all the pods, represented as a percent of requested CPU. If not specified or negative, a default autoscaling policy is used.

   • To scale for a specific CPU value, create a YAML file similar to the following for an existing DeploymentConfig object or replication controller:

     1. Create a YAML file similar to the following:

        apiVersion: autoscaling/v2beta2 1
        kind: HorizontalPodAutoscaler
        metadata:
          name: cpu-autoscale 2
          namespace: default
        spec:
          scaleTargetRef:
            apiVersion: v1 3
            kind: ReplicationController 4
            name: example 5
          minReplicas: 1 6
          maxReplicas: 10 7
          metrics: 8
          - type: Resource
            resource:
              name: cpu 9
              target:
                type: AverageValue 10
                averageValue: 500m 11

        1

        Use the autoscaling/v2beta2 API.

        2

        Specify a name for this horizontal pod autoscaler object.

        3

        Specify the API version of the object to scale:

        • For a replication controller, use v1,
        • For a DeploymentConfig object, use apps.openshift.io/v1.

        4

        Specify the kind of object to scale, either ReplicationController or DeploymentConfig.

        5

        Specify the name of the object to scale. The object must exist.

        6

        Specify the minimum number of replicas when scaling down.

        7

        Specify the maximum number of replicas when scaling up.

        8

        Use the metrics parameter for memory utilization.

        9

        Specify cpu for CPU utilization.

        10

        Set to AverageValue.

        11

        Set to averageValue with the targeted CPU value.

     2. Create the horizontal pod autoscaler:

        $ oc create -f <file-name>.yaml

2. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa cpu-autoscale

   Example output

   NAME            REFERENCE                       TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   cpu-autoscale   ReplicationController/example   173m/500m       1         10        1          20m

1. View the current state of the image-registry deployment:

   $ oc get dc image-registry

   Example output

   NAME             REVISION   DESIRED   CURRENT   TRIGGERED BY
   image-registry   1          3         3         config

2. Autoscale the image-registry DeploymentConfig object:

   $ oc autoscale dc/image-registry --min=5 --max=7 --cpu-percent=75

   Example output

   horizontalpodautoscaler.autoscaling/image-registry autoscaled

3. View the new state of the deployment:

   $ oc get dc image-registry

   There are now 5 pods in the deployment:

   Example output

   NAME             REVISION   DESIRED   CURRENT   TRIGGERED BY
   image-registry   1          5         5         config

1. Create a YAML file for one of the following:

   • To scale for a specific memory value, create a HorizontalPodAutoscaler object similar to the following for an existing DeploymentConfig object or replication controller:

     Example output

     apiVersion: autoscaling/v2beta2 1
     kind: HorizontalPodAutoscaler
     metadata:
       name: hpa-resource-metrics-memory 2
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: v1 3
         kind: ReplicationController 4
         name: example 5
       minReplicas: 1 6
       maxReplicas: 10 7
       metrics: 8
       - type: Resource
         resource:
           name: memory 9
           target:
             type: AverageValue 10
             averageValue: 500Mi 11
       behavior: 12
         scaleDown:
           stabilizationWindowSeconds: 300
           policies:
           - type: Pods
             value: 4
             periodSeconds: 60
           - type: Percent
             value: 10
             periodSeconds: 60
           selectPolicy: Max

     1

     Use the autoscaling/v2beta2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a replication controller, use v1,
     • For a DeploymentConfig object, use apps.openshift.io/v1.

     4

     Specify the kind of object to scale, either ReplicationController or DeploymentConfig.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set the type to AverageValue.

     11

     Specify averageValue and a specific memory value.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

   • To scale for a percentage, create a HorizontalPodAutoscaler object similar to the following:

     Example output

     apiVersion: autoscaling/v2beta2 1
     kind: HorizontalPodAutoscaler
     metadata:
       name: memory-autoscale 2
       namespace: default
     spec:
       scaleTargetRef:
         apiVersion: apps.openshift.io/v1 3
         kind: DeploymentConfig 4
         name: example 5
       minReplicas: 1 6
       maxReplicas: 10 7
       metrics: 8
       - type: Resource
         resource:
           name: memory 9
           target:
             type: Utilization 10
             averageUtilization: 50 11
       behavior: 12
         scaleUp:
           stabilizationWindowSeconds: 180
           policies:
           - type: Pods
             value: 6
             periodSeconds: 120
           - type: Percent
             value: 10
             periodSeconds: 120
           selectPolicy: Max

     1

     Use the autoscaling/v2beta2 API.

     2

     Specify a name for this horizontal pod autoscaler object.

     3

     Specify the API version of the object to scale:

     • For a replication controller, use v1,
     • For a DeploymentConfig object, use apps.openshift.io/v1.

     4

     Specify the kind of object to scale, either ReplicationController or DeploymentConfig.

     5

     Specify the name of the object to scale. The object must exist.

     6

     Specify the minimum number of replicas when scaling down.

     7

     Specify the maximum number of replicas when scaling up.

     8

     Use the metrics parameter for memory utilization.

     9

     Specify memory for memory utilization.

     10

     Set to Utilization.

     11

     Specify averageUtilization and a target average memory utilization over all the pods, represented as a percent of requested memory. The target pods must have memory requests configured.

     12

     Optional: Specify a scaling policy to control the rate of scaling up or down.

2. Create the horizontal pod autoscaler:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f hpa.yaml

   Example output

   horizontalpodautoscaler.autoscaling/hpa-resource-metrics-memory created

3. Verify that the horizontal pod autoscaler was created:

   $ oc get hpa hpa-resource-metrics-memory

   Example output

   NAME                          REFERENCE                       TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
   hpa-resource-metrics-memory   ReplicationController/example   2441216/500Mi   1         10        1          20m

   $ oc describe hpa hpa-resource-metrics-memory

   Example output

   Name:                        hpa-resource-metrics-memory
   Namespace:                   default
   Labels:                      <none>
   Annotations:                 <none>
   CreationTimestamp:           Wed, 04 Mar 2020 16:31:37 +0530
   Reference:                   ReplicationController/example
   Metrics:                     ( current / target )
     resource memory on pods:   2441216 / 500Mi
   Min replicas:                1
   Max replicas:                10
   ReplicationController pods:  1 current / 1 desired
   Conditions:
     Type            Status  Reason              Message
     ----            ------  ------              -------
     AbleToScale     True    ReadyForNewScale    recommended size matches current size
     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
   Events:
     Type     Reason                   Age                 From                       Message
     ----     ------                   ----                ----                       -------
     Normal   SuccessfulRescale        6m34s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose VerticalPodAutoscaler from the list of available Operators, and click Install.
3. On the Install Operator page, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory openshift-vertical-pod-autoscaler namespace, which is automatically created if it does not exist.
4. Click Install.

5. Verify the installation by listing the VPA Operator components:

   1. Navigate to Workloads → Pods.
   2. Select the openshift-vertical-pod-autoscaler project from the drop-down menu and verify that there are four pods running.
   3. Navigate to Workloads → Deployments to verify that there are four deployments running.

6. Optional. Verify the installation in the OpenShift Container Platform CLI using the following command:

   $ oc get all -n openshift-vertical-pod-autoscaler

   The output shows four pods and four deplyoments:

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/vertical-pod-autoscaler-operator-85b4569c47-2gmhc   1/1     Running   0          3m13s
   pod/vpa-admission-plugin-default-67644fc87f-xq7k9       1/1     Running   0          2m56s
   pod/vpa-recommender-default-7c54764b59-8gckt            1/1     Running   0          2m56s
   pod/vpa-updater-default-7f6cc87858-47vw9                1/1     Running   0          2m56s
   
   NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   service/vpa-webhook   ClusterIP   172.30.53.206   <none>        443/TCP   2m56s
   
   NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/vertical-pod-autoscaler-operator   1/1     1            1           3m13s
   deployment.apps/vpa-admission-plugin-default       1/1     1            1           2m56s
   deployment.apps/vpa-recommender-default            1/1     1            1           2m56s
   deployment.apps/vpa-updater-default                1/1     1            1           2m56s
   
   NAME                                                          DESIRED   CURRENT   READY   AGE
   replicaset.apps/vertical-pod-autoscaler-operator-85b4569c47   1         1         1       3m13s
   replicaset.apps/vpa-admission-plugin-default-67644fc87f       1         1         1       2m56s
   replicaset.apps/vpa-recommender-default-7c54764b59            1         1         1       2m56s
   replicaset.apps/vpa-updater-default-7f6cc87858                1         1         1       2m56s

1. Change to the project where the workload object you want to scale is located.

   1. Create a VPA CR YAML file:

      apiVersion: autoscaling.k8s.io/v1
      kind: VerticalPodAutoscaler
      metadata:
        name: vpa-recommender
      spec:
        targetRef:
          apiVersion: "apps/v1"
          kind:       Deployment 1
          name:       frontend 2
        updatePolicy:
          updateMode: "Auto" 3
        resourcePolicy: 4
          containerPolicies:
          - containerName: my-opt-sidecar
            mode: "Off"

      1

      Specify the type of workload object you want this VPA to manage: Deployment, StatefulSet, Job, DaemonSet, ReplicaSet, or ReplicationController.

      2

      Specify the name of an existing workload object you want this VPA to manage.

      3

      Specify the VPA mode:

      • auto to automatically apply the recommended resources on pods associated with the controller. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests.
      • recreate to automatically apply the recommended resources on pods associated with the workload object. The VPA terminates existing pods and creates new pods with the recommended resource limits and requests. The recreate mode should be used rarely, only if you need to ensure that the pods are restarted whenever the resource request changes.
      • initial to automatically apply the recommended resources when pods associated with the workload object are created. The VPA does not update the pods as it learns new resource recommendations.
      • off to only generate resource recommendations for the pods associated with the workload object. The VPA does not update the pods as it learns new resource recommendations and does not apply the recommendations to new pods.

      4

      Optional. Specify the containers you want to opt-out and set the mode to Off.

   2. Create the VPA CR:

      $ oc create -f <file-name>.yaml

      After a few moments, the VPA learns the resource usage of the containers in the pods associated with the workload object.

      You can view the VPA recommendations using the following command:

      $ oc get vpa <vpa-name> --output yaml

      The output shows the recommendations for CPU and memory requests, similar to the following:

      Example output

      ...
      status:
      
      ...
      
        recommendation:
          containerRecommendations:
          - containerName: frontend
            lowerBound: 1
              cpu: 25m
              memory: 262144k
            target: 2
              cpu: 25m
              memory: 262144k
            uncappedTarget: 3
              cpu: 25m
              memory: 262144k
            upperBound: 4
              cpu: 262m
              memory: "274357142"
          - containerName: backend
            lowerBound:
              cpu: 12m
              memory: 131072k
            target:
              cpu: 12m
              memory: 131072k
            uncappedTarget:
              cpu: 12m
              memory: 131072k
            upperBound:
              cpu: 476m
              memory: "498558823"
      
      ...

      1

      lowerBound is the minimum recommended resource levels.

      2

      target is the recommended resource levels.

      3

      upperBound is the highest recommended resource levels.

      4

      uncappedTarget is the most recent resource recommendations.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
2. Switch to the openshift-vertical-pod-autoscaler project.
3. Find the VerticalPodAutoscaler Operator and click the Options menu. Select Uninstall Operator.
4. In the dialog box, click Uninstall.

1. Delete the secret:

   $ oc delete secret <secret_name>

2. Clear the annotations:

   $ oc annotate service <service_name> service.alpha.openshift.io/serving-cert-generation-error-

   $ oc annotate service <service_name> service.alpha.openshift.io/serving-cert-generation-error-num-

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config:

      # oc describe machineconfig <name>

      For example:

      # oc describe machineconfig 00-worker

      Example output

      Name:         00-worker
      Namespace:
      Labels:       machineconfiguration.openshift.io/role=worker 1

      1 1

      Label required for the Device Manager.

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a Device Manager CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: devicemgr 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
          machineconfiguration.openshift.io: devicemgr 2
     kubeletConfig:
       feature-gates:
         - DevicePlugins=true 3

   1

   Assign a name to CR.

   2

   Enter the label from the Machine Config Pool.

   3

   Set DevicePlugins to 'true`.

2. Create the Device Manager:

   $ oc create -f devicemgr.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/devicemgr created

3. Ensure that Device Manager was actually enabled by confirming that /var/lib/kubelet/device-plugins/kubelet.sock is created on the node. This is the UNIX domain socket on which the Device Manager gRPC server listens for new plug-in registrations. This sock file is created when the Kubelet is started only if Device Manager is enabled.

1. Create one or more priority classes:

   1. Specify a name and value for the priority.
   2. Optionally specify the globalDefault field in the priority class and a description.

2. Create a Pod spec or edit existing pods to include the name of a priority class, similar to the following:

   Sample Pod spec with priority class name

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx
     labels:
       env: test
   spec:
     containers:
     - name: nginx
       image: nginx
       imagePullPolicy: IfNotPresent
     priorityClassName: high-priority 1

   1

   Specify the priority class to use with this pod.

3. Create the pod:

   $ oc create -f <file-name>.yaml

   You can add the priority name directly to the pod configuration or to a pod template.

Procedure

1. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet abc612-msrtw-worker-us-east-1c  --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        
        ....
        
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
        ....

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east

     2. Verify that the labels are added to the node:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                          STATUS   ROLES    AGE   VERSION
        ip-10-0-142-25.ec2.internal   Ready    worker   17m   v1.18.3+002a51f

2. Add the matching node selector a pod:

   • To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:

     Example ReplicaSet object with labels

     kind: ReplicaSet
     
     ....
     
     spec:
     
     ....
     
       template:
         metadata:
           creationTimestamp: null
           labels:
             ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
             pod-template-hash: 66d5cf9464
         spec:
           nodeSelector:
             beta.kubernetes.io/os: linux
             node-role.kubernetes.io/worker: ''
             type: user-node 1

     1

     Add the node selector.

   • To add a node selector to a specific, new pod, add the selector to the Pod object directly:

     Example Pod object with a node selector

     apiVersion: v1
     kind: Pod
     
     ....
     
     spec:
       nodeSelector:
         region: east
         type: user-node

     Note

     You cannot add a node selector directly to an existing scheduled pod.

1. Create a JSON file named policy.cfg with the desired predicates and priorities.

   Sample scheduler JSON file

   {
   "kind" : "Policy",
   "apiVersion" : "v1",
   "predicates" : [ 1
           {"name" : "MaxGCEPDVolumeCount"},
           {"name" : "GeneralPredicates"},
           {"name" : "MaxAzureDiskVolumeCount"},
           {"name" : "MaxCSIVolumeCountPred"},
           {"name" : "CheckVolumeBinding"},
           {"name" : "MaxEBSVolumeCount"},
           {"name" : "MatchInterPodAffinity"},
           {"name" : "CheckNodeUnschedulable"},
           {"name" : "NoDiskConflict"},
           {"name" : "NoVolumeZoneConflict"},
           {"name" : "PodToleratesNodeTaints"}
           ],
   "priorities" : [ 2
           {"name" : "LeastRequestedPriority", "weight" : 1},
           {"name" : "BalancedResourceAllocation", "weight" : 1},
           {"name" : "ServiceSpreadingPriority", "weight" : 1},
           {"name" : "NodePreferAvoidPodsPriority", "weight" : 1},
           {"name" : "NodeAffinityPriority", "weight" : 1},
           {"name" : "TaintTolerationPriority", "weight" : 1},
           {"name" : "ImageLocalityPriority", "weight" : 1},
           {"name" : "SelectorSpreadPriority", "weight" : 1},
           {"name" : "InterPodAffinityPriority", "weight" : 1},
           {"name" : "EqualPriority", "weight" : 1}
           ]
   }

   1

   Add the predicates as needed.

   2

   Add the priorities as needed.

2. Create a config map based on the scheduler JSON file:

   $ oc create configmap -n openshift-config --from-file=policy.cfg <configmap-name> 1

   1

   Enter a name for the config map.

   For example:

   $ oc create configmap -n openshift-config --from-file=policy.cfg scheduler-policy

   Example output

   configmap/scheduler-policy created

3. Edit the Scheduler Operator custom resource to add the config map:

   $ oc patch Scheduler cluster --type='merge' -p '{"spec":{"policy":{"name":"<configmap-name>"}}}' --type=merge 1

   1

   Specify the name of the config map.

   For example:

   $ oc patch Scheduler cluster --type='merge' -p '{"spec":{"policy":{"name":"scheduler-policy"}}}' --type=merge

   After making the change to the Scheduler config resource, wait for the openshift-kube-apiserver pods to redeploy. This can take several minutes. Until the pods redeploy, new scheduler does not take effect.

4. Verify the scheduler policy is configured by viewing the log of a scheduler pod in the openshift-kube-scheduler namespace. The following command checks for the predicates and priorities that are being registered by the scheduler:

   $ oc logs <scheduler-pod> | grep predicates

   For example:

   $ oc logs openshift-kube-scheduler-ip-10-0-141-29.ec2.internal | grep predicates

   Example output

   Creating scheduler with fit predicates 'map[MaxGCEPDVolumeCount:{} MaxAzureDiskVolumeCount:{} CheckNodeUnschedulable:{} NoDiskConflict:{} NoVolumeZoneConflict:{} GeneralPredicates:{} MaxCSIVolumeCountPred:{} CheckVolumeBinding:{} MaxEBSVolumeCount:{} MatchInterPodAffinity:{} PodToleratesNodeTaints:{}]' and priority functions 'map[InterPodAffinityPriority:{} LeastRequestedPriority:{} ServiceSpreadingPriority:{} ImageLocalityPriority:{} SelectorSpreadPriority:{} EqualPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{} TaintTolerationPriority:{}]'

Procedure

1. Create a pod with a specific label in the Pod spec:

   $ cat team4.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: security-s1
     labels:
       security: S1
   spec:
     containers:
     - name: security-s1
       image: docker.io/ocpqe/hello-pod

2. When creating other pods, edit the Pod spec as follows:

   1. Use the podAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter or preferredDuringSchedulingIgnoredDuringExecution parameter:

   2. Specify the key and value that must be met. If you want the new pod to be scheduled with the other pod, use the same key and value parameters as the label on the first pod.

          podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: security
                  operator: In
                  values:
                  - S1
              topologyKey: failure-domain.beta.kubernetes.io/zone

   3. Specify an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.
   4. Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

3. Create the pod.

   $ oc create -f <pod-spec>.yaml

Procedure

1. Create a pod with a specific label in the Pod spec:

   $ cat team4.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: security-s2
     labels:
       security: S2
   spec:
     containers:
     - name: security-s2
       image: docker.io/ocpqe/hello-pod

2. When creating other pods, edit the Pod spec to set the following parameters:

3. Use the podAntiAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter or preferredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify a weight for the node, 1-100. The node that with highest weight is preferred.

   2. Specify the key and values that must be met. If you want the new pod to not be scheduled with the other pod, use the same key and value parameters as the label on the first pod.

          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: security
                    operator: In
                    values:
                    - S2
                topologyKey: kubernetes.io/hostname

   3. For a preferred rule, specify a weight, 1-100.
   4. Specify an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.
4. Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

5. Create the pod.

   $ oc create -f <pod-spec>.yaml

1. Add a label to a node using the oc label node command:

   $ oc label node node1 e2e-az-name=e2e-az1

2. In the Pod spec, use the nodeAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and value parameters as the label in the node.

   2. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the operator In to require the label to be in the node:

      Example output

      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: e2e-az-name
                  operator: In
                  values:
                  - e2e-az1
                  - e2e-az2

3. Create the pod:

   $ oc create -f e2e-az2.yaml

1. Add a label to a node using the oc label node command:

   $ oc label node node1 e2e-az-name=e2e-az3

2. In the Pod spec, use the nodeAffinity stanza to configure the preferredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify a weight for the node, as a number 1-100. The node with highest weight is preferred.

   2. Specify the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and value parameters as the label in the node:

      spec:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                - key: e2e-az-name
                  operator: In
                  values:
                  - e2e-az3

   3. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the Operator In to require the label to be in the node.

3. Create the pod.

   $ oc create -f e2e-az3.yaml

1. If you add a NoSchedule taint to a master node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with an Equal operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1" 1
           value: "value1"
           operator: "Equal"
           effect: "NoExecute"
           tolerationSeconds: 3600 2

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod can remain bound to a node before being evicted.

   For example:

   Sample pod configuration file with an Exists operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1"
           operator: "Exists" 1
           effect: "NoExecute"
           tolerationSeconds: 3600

   1

   The Exists operator does not take a value.

   This example places a taint on node1 that has key key1, value value1, and taint effect NoExecute.

2. Add a taint to a node by using the following command with the parameters described in the Taint and toleration components table:

   $ oc adm taint nodes <node_name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 key1=value1:NoExecute

   This command places a taint on node1 that has key key1, value value1, and effect NoExecute.

   Note

   If you add a NoSchedule taint to a master node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

   The tolerations on the Pod match the taint on the node. A pod with either toleration can be scheduled onto node1.

1. To remove a taint from a node:

   $ oc adm taint nodes <node-name> <key>-

   For example:

   $ oc adm taint nodes ip-10-0-132-248.ec2.internal key1-

   Example output

   node/ip-10-0-132-248.ec2.internal untainted

2. To remove a toleration from a pod, edit the Pod spec to remove the toleration:

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key2"
           operator: "Exists"
           effect: "NoExecute"
           tolerationSeconds: 3600

Procedure

1. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet abc612-msrtw-worker-us-east-1c  --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        
        ....
        
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
        ....

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east

     2. Verify that the labels are added to the node:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                          STATUS   ROLES    AGE   VERSION
        ip-10-0-142-25.ec2.internal   Ready    worker   17m   v1.18.3+002a51f

2. Add the matching node selector a pod:

   • To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:

     Example ReplicaSet object with labels

     kind: ReplicaSet
     
     ....
     
     spec:
     
     ....
     
       template:
         metadata:
           creationTimestamp: null
           labels:
             ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
             pod-template-hash: 66d5cf9464
         spec:
           nodeSelector:
             beta.kubernetes.io/os: linux
             node-role.kubernetes.io/worker: ''
             type: user-node 1

     1

     Add the node selector.

   • To add a node selector to a specific, new pod, add the selector to the Pod object directly:

     Example Pod object with a node selector

     apiVersion: v1
     kind: Pod
     
     ....
     
     spec:
       nodeSelector:
         region: east
         type: user-node

     Note

     You cannot add a node selector directly to an existing scheduled pod.

1. Edit the Scheduler Operator CR to add the default cluster-wide node selectors:

   $ oc edit scheduler cluster

   Example Scheduler Operator CR with a node selector

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   
   spec:
     defaultNodeSelector: type=user-node,region=east 1
     mastersSchedulable: false
     policy:
       name: ""

   1

   Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a machine set or editing the node directly:

   • Use a machine set to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api 1

        1

        Add a <key>/<value> pair for each label.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        Example output

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node

     3. Redeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

     2. Verify that the labels are added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

1. Create a project or edit an existing project to add the openshift.io/node-selector parameter:

   $ oc edit project <name>

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     annotations:
       openshift.io/node-selector: "type=user-node,region=east" 1
       openshift.io/description: ""
       openshift.io/display-name: ""
       openshift.io/requester: kube:admin
       openshift.io/sa.scc.mcs: s0:c30,c5
       openshift.io/sa.scc.supplemental-groups: 1000880000/10000
       openshift.io/sa.scc.uid-range: 1000880000/10000
     creationTimestamp: "2021-05-10T12:35:04Z"
     labels:
       kubernetes.io/metadata.name: demo
     name: demo
     resourceVersion: "145537"
     uid: 3f8786e3-1fcb-42e3-a0e3-e2ac54d15001
   spec:
     finalizers:
     - kubernetes

   1

   Add the openshift.io/node-selector with the appropriate <key>:<value> pairs.

2. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add labels to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the labels are added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        Example output

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node

     3. Redeploy the nodes associated with that machine set:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:

        $ oc label MachineSet abc612-msrtw-worker-us-east-1c type=user-node region=east

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

   • Add labels directly to a node:

     1. Edit the Node object to add labels:

        $ oc label <resource> <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-c-tgq49 type=user-node region=east

     2. Verify that the labels are added to the Node object using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

Procedure

1. Create a file that contains the deployment resources for the custom scheduler:

   Example custom-scheduler.yaml file

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: custom-scheduler
     namespace: kube-system 1
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: custom-scheduler-as-kube-scheduler
   subjects:
   - kind: ServiceAccount
     name: custom-scheduler
     namespace: kube-system 2
   roleRef:
     kind: ClusterRole
     name: system:kube-scheduler
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       component: scheduler
       tier: control-plane
     name: custom-scheduler
     namespace: kube-system 3
   spec:
     selector:
       matchLabels:
         component: scheduler
         tier: control-plane
     replicas: 1
     template:
       metadata:
         labels:
           component: scheduler
           tier: control-plane
           version: second
       spec:
         serviceAccountName: custom-scheduler
         containers:
         - command:
           - /usr/local/bin/kube-scheduler
           - --address=0.0.0.0
           - --leader-elect=false
           - --scheduler-name=custom-scheduler 4
           image: "<namespace>/<image_name>:<tag>" 5
           livenessProbe:
             httpGet:
               path: /healthz
               port: 10251
             initialDelaySeconds: 15
           name: kube-second-scheduler
           readinessProbe:
             httpGet:
               path: /healthz
               port: 10251
           resources:
             requests:
               cpu: '0.1'
           securityContext:
             privileged: false
           volumeMounts: []
         hostNetwork: false
         hostPID: false
         volumes: []

   1 2 3

   This procedure uses the kube-system namespace, but you can use the namespace of your choosing.

   4

   The command for your custom scheduler might require different arguments. For example, you can pass configuration as a mounted volume using the --config argument.

   5

   Specify the container image that you created for the custom scheduler.

2. Create the deployment resources in the cluster:

   $ oc create -f custom-scheduler.yaml

Procedure

1. If your cluster uses role-based access control (RBAC), add the custom scheduler name to the system:kube-scheduler cluster role.

   1. Edit the system:kube-scheduler cluster role:

      $ oc edit clusterrole system:kube-scheduler

   2. Add the name of the custom scheduler to the resourceNames lists for the leases and endpoints resources:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        annotations:
          rbac.authorization.kubernetes.io/autoupdate: "true"
        creationTimestamp: "2021-07-07T10:19:14Z"
        labels:
          kubernetes.io/bootstrapping: rbac-defaults
        name: system:kube-scheduler
        resourceVersion: "125"
        uid: 53896c70-b332-420a-b2a4-f72c822313f2
      rules:
      ...
      - apiGroups:
        - coordination.k8s.io
        resources:
        - leases
        verbs:
        - create
      - apiGroups:
        - coordination.k8s.io
        resourceNames:
        - kube-scheduler
        - custom-scheduler 1
        resources:
        - leases
        verbs:
        - get
        - update
      - apiGroups:
        - ""
        resources:
        - endpoints
        verbs:
        - create
      - apiGroups:
        - ""
        resourceNames:
        - kube-scheduler
        - custom-scheduler 2
        resources:
        - endpoints
        verbs:
        - get
        - update
      ...

      1 2

      This example uses custom-scheduler as the custom scheduler name.

2. Create a Pod configuration and specify the name of the custom scheduler in the schedulerName parameter:

   Example custom-scheduler-example.yaml file

   apiVersion: v1
   kind: Pod
   metadata:
     name: custom-scheduler-example
     labels:
       name: custom-scheduler-example
   spec:
     schedulerName: custom-scheduler 1
     containers:
     - name: pod-with-second-annotation-container
       image: docker.io/ocpqe/hello-pod

   1

   The name of the custom scheduler to use, which is custom-scheduler in this example. When no scheduler name is supplied, the pod is automatically scheduled using the default scheduler.

3. Create the pod:

   $ oc create -f custom-scheduler-example.yaml

Verification

1. Enter the following command to check that the pod was created:

   $ oc get pod custom-scheduler-example

   The custom-scheduler-example pod is listed in the output:

   NAME                       READY     STATUS    RESTARTS   AGE
   custom-scheduler-example   1/1       Running   0          4m

2. Enter the following command to check that the custom scheduler has scheduled the pod:

   $ oc describe pod custom-scheduler-example

   The scheduler, custom-scheduler, is listed as shown in the following truncated output:

   Events:
     Type    Reason          Age        From                                               Message
     ----    ------          ----       ----                                               -------
     Normal  Scheduled       <unknown>  custom-scheduler                                   Successfully assigned default/custom-scheduler-example to <node_name>

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Create the required namespace for the Kube Descheduler Operator.

   1. Navigate to Administration → Namespaces and click Create Namespace.
   2. Enter openshift-kube-descheduler-operator in the Name field and click Create.

3. Install the Kube Descheduler Operator.

   1. Navigate to Operators → OperatorHub.
   2. Type Kube Descheduler Operator into the filter box.
   3. Select the Kube Descheduler Operator and click Install.
   4. On the Install Operator page, select A specific namespace on the cluster. Select openshift-kube-descheduler-operator from the drop-down menu.
   5. Adjust the values for the Update Channel and Approval Strategy to the desired values.
   6. Click Install.

4. Create a descheduler instance.

   1. From the Operators → Installed Operators page, click the Kube Descheduler Operator.
   2. Select the Kube Descheduler tab and click Create KubeDescheduler.
   3. Edit the settings as necessary and click Create.

Procedure

1. Edit the KubeDescheduler object:

   $ oc edit kubedeschedulers.operator.openshift.io cluster -n openshift-kube-descheduler-operator

2. Specify one or more strategies in the spec.strategies section.

   apiVersion: operator.openshift.io/v1beta1
   kind: KubeDescheduler
   metadata:
     name: cluster
     namespace: openshift-kube-descheduler-operator
   spec:
     deschedulingIntervalSeconds: 3600
     strategies:
       - name: "LowNodeUtilization" 1
         params:
          - name: "CPUThreshold"
            value: "10"
          - name: "MemoryThreshold"
            value: "20"
          - name: "PodsThreshold"
            value: "30"
          - name: "MemoryTargetThreshold"
            value: "40"
          - name: "CPUTargetThreshold"
            value: "50"
          - name: "PodsTargetThreshold"
            value: "60"
          - name: "NumberOfNodes"
            value: "3"
       - name: "RemoveDuplicates" 2
       - name: "RemovePodsHavingTooManyRestarts" 3
         params:
          - name: "PodRestartThreshold"
            value: "10"
          - name: "IncludingInitContainers"
            value: "false"

   1

   The LowNodeUtilization strategy provides additional parameters, such as CPUThreshold and MemoryThreshold, that you can optionally configure.

   2

   The RemoveDuplicates, RemovePodsViolatingInterPodAntiAffinity, RemovePodsViolatingNodeAffinity, and RemovePodsViolatingNodeTaints strategies do not have any additional parameters to configure.

   3

   The RemovePodsHavingTooManyRestarts strategy requires the PodRestartThreshold parameter to be set. It also provides the optional IncludingInitContainers parameter.

   You can enable multiple strategies and the order that the strategies are specified in is not important.

3. Save the file to apply the changes.

Procedure

1. Edit the KubeDescheduler object:

   $ oc edit kubedeschedulers.operator.openshift.io cluster -n openshift-kube-descheduler-operator

2. Configure additional settings as necessary:

   apiVersion: operator.openshift.io/v1beta1
   kind: KubeDescheduler
   metadata:
     name: cluster
     namespace: openshift-kube-descheduler-operator
   spec:
     deschedulingIntervalSeconds: 3600 1
     flags:
     - --dry-run 2
     image: quay.io/openshift/origin-descheduler:4.5 3
   ...

   1

   Set number of seconds between descheduler runs. A value of 0 in this field runs the descheduler once and exits.

   2

   Set one or more flags to append to the descheduler pod. This flag must be in the format ready to pass to the binary.

   3

   Set the descheduler container image to deploy.

3. Save the file to apply the changes.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Delete the descheduler instance.

   1. From the Operators → Installed Operators page, click Kube Descheduler Operator.
   2. Select the Kube Descheduler tab.
   3. Click the Options menu next to the cluster entry and select Delete KubeDescheduler.
   4. In the confirmation dialog, click Delete.

3. Uninstall the Kube Descheduler Operator.

   1. Navigate to Operators → Installed Operators,
   2. Click the Options menu next to the Kube Descheduler Operator entry and select Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

4. Delete the openshift-kube-descheduler-operator namespace.

   1. Navigate to Administration → Namespaces.
   2. Enter openshift-kube-descheduler-operator into the filter box.
   3. Click the Options menu next to the openshift-kube-descheduler-operator entry and select Delete Namespace.
   4. In the confirmation dialog, enter openshift-kube-descheduler-operator and click Delete.

5. Delete the KubeDescheduler CRD.

   1. Navigate to Administration → Custom Resource Definitions.
   2. Enter KubeDescheduler into the filter box.
   3. Click the Options menu next to the KubeDescheduler entry and select Delete CustomResourceDefinition.
   4. In the confirmation dialog, click Delete.

1. Define the daemon set yaml file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset 1
     template:
       metadata:
         labels:
           name: hello-daemonset 2
       spec:
         nodeSelector: 3
           role: worker
         containers:
         - image: openshift/hello-openshift
           imagePullPolicy: Always
           name: registry
           ports:
           - containerPort: 80
             protocol: TCP
           resources: {}
           terminationMessagePath: /dev/termination-log
         serviceAccount: default
         terminationGracePeriodSeconds: 10

   1

   The label selector that determines which pods belong to the daemon set.

   2

   The pod template’s label selector. Must match the label selector above.

   3

   The node selector that determines on which nodes pod replicas should be deployed. A matching label must be present on the node.

2. Create the daemon set object:

   $ oc create -f daemonset.yaml

3. To verify that the pods were created, and that each node has a pod replica:

   1. Find the daemonset pods:

      $ oc get pods

      Example output

      hello-daemonset-cx6md   1/1       Running   0          2m
      hello-daemonset-e3md9   1/1       Running   0          2m

   2. View the pods to verify the pod has been placed onto the node:

      $ oc describe pod/hello-daemonset-cx6md|grep Node

      Example output

      Node:        openshift-node01.hostname.com/10.14.20.134

      $ oc describe pod/hello-daemonset-e3md9|grep Node

      Example output

      Node:        openshift-node02.hostname.com/10.14.20.137

1. Create a YAML file similar to the following:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: pi
   spec:
     parallelism: 1    1
     completions: 1    2
     activeDeadlineSeconds: 1800 3
     backoffLimit: 6   4
     template:         5
       metadata:
         name: pi
       spec:
         containers:
         - name: pi
           image: perl
           command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
         restartPolicy: OnFailure    6

   1. Optionally, specify how many pod replicas a job should run in parallel; defaults to 1.

      • For non-parallel jobs, leave unset. When unset, defaults to 1.

   2. Optionally, specify how many successful pod completions are needed to mark a job completed.

      • For non-parallel jobs, leave unset. When unset, defaults to 1.
      • For parallel jobs with a fixed completion count, specify the number of completions.
      • For parallel jobs with a work queue, leave unset. When unset defaults to the parallelism value.
   3. Optionally, specify the maximum duration the job can run.
   4. Optionally, specify the number of retries for a job. This field defaults to six.
   5. Specify the template for the pod the controller creates.

   6. Specify the restart policy of the pod:

      • Never. Do not restart the job.
      • OnFailure. Restart the job only if it fails.

      • Always. Always restart the job.

        For details on how OpenShift Container Platform uses restart policy with failed containers, see the Example States in the Kubernetes documentation.

2. Create the job:

   $ oc create -f <file-name>.yaml

1. Create a YAML file similar to the following:

   apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
     name: pi
   spec:
     schedule: "*/1 * * * *"  1
     concurrencyPolicy: "Replace" 2
     startingDeadlineSeconds: 200 3
     suspend: true            4
     successfulJobsHistoryLimit: 3 5
     failedJobsHistoryLimit: 1     6
     jobTemplate:             7
       spec:
         template:
           metadata:
             labels:          8
               parent: "cronjobpi"
           spec:
             containers:
             - name: pi
               image: perl
               command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
             restartPolicy: OnFailure 9

   1 1 1

   Schedule for the job specified in cron format. In this example, the job will run every minute.

   2 2 2

   An optional concurrency policy, specifying how to treat concurrent jobs within a cron job. Only one of the following concurrent policies may be specified. If not specified, this defaults to allowing concurrent executions.

   • Allow allows cron jobs to run concurrently.
   • Forbid forbids concurrent runs, skipping the next run if the previous has not finished yet.
   • Replace cancels the currently running job and replaces it with a new one.

   3 3 3

   An optional deadline (in seconds) for starting the job if it misses its scheduled time for any reason. Missed jobs executions will be counted as failed ones. If not specified, there is no deadline.

   4 4 4

   An optional flag allowing the suspension of a cron job. If set to true, all subsequent executions will be suspended.

   5 5 5

   The number of successful finished jobs to retain (defaults to 3).

   6 6 6

   The number of failed finished jobs to retain (defaults to 1).

   7

   Job template. This is similar to the job example.

   8

   Sets a label for jobs spawned by this cron job.

   9

   The restart policy of the pod. This does not apply to the job controller.

   Note

   The .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 3 and 1 respectively. Setting a limit to 0 corresponds to keeping none of the corresponding kind of jobs after they finish.

2. Create the cron job:

   $ oc create -f <file-name>.yaml

Procedure

1. Mark the nodes unschedulable before performing the pod evacuation.

   1. Mark the node as unschedulable:

      $ oc adm cordon <node1>

      Example output

      node/<node1> cordoned

   2. Check that the node status is NotReady,SchedulingDisabled:

      $ oc get node <node1>

      Example output

      NAME        STATUS                        ROLES     AGE       VERSION
      <node1>     NotReady,SchedulingDisabled   worker    1d        v1.18.3

2. Evacuate the pods using one of the following methods:

   • Evacuate all or selected pods on one or more nodes:

     $ oc adm drain <node1> <node2> [--pod-selector=<pod_selector>]

   • Force the deletion of bare pods using the --force option. When set to true, deletion continues even if there are pods not managed by a replication controller, replica set, job, daemon set, or stateful set:

     $ oc adm drain <node1> <node2> --force=true

   • Set a period of time in seconds for each Pod to terminate gracefully, use --grace-period. If negative, the default value specified in the Pod will be used:

     $ oc adm drain <node1> <node2> --grace-period=-1

   • Ignore pods managed by daemon sets using the --ignore-daemonsets flag set to true:

     $ oc adm drain <node1> <node2> --ignore-daemonsets=true

   • Set the length of time to wait before giving up using the --timeout flag. A value of 0 sets an infinite length of time:

     $ oc adm drain <node1> <node2> --timeout=5s

   • Delete pods even if there are pods using emptyDir using the --delete-local-data flag set to true. Local data is deleted when the node is drained:

     $ oc adm drain <node1> <node2> --delete-local-data=true

   • List objects that will be migrated without actually performing the evacuation, using the --dry-run option set to true:

     $ oc adm drain <node1> <node2>  --dry-run=true

     Instead of specifying specific node names (for example, <node1> <node2>), you can use the --selector=<node_selector> option to evacuate pods on selected nodes.

3. Mark the node as schedulable when done.

   $ oc adm uncordon <node1>

Procedure

1. Edit the schedulers.config.openshift.io resource.

   $ oc edit schedulers.config.openshift.io cluster

2. Configure the mastersSchedulable field.

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     creationTimestamp: "2019-09-10T03:04:05Z"
     generation: 1
     name: cluster
     resourceVersion: "433"
     selfLink: /apis/config.openshift.io/v1/schedulers/cluster
     uid: a636d30a-d377-11e9-88d4-0a60097bee62
   spec:
     mastersSchedulable: false 1
     policy:
       name: ""
   status: {}

   1

   Set to true to allow master nodes to be schedulable, or false to disallow master nodes to be schedulable.

3. Save the file to apply the changes.

Procedure

1. List existing MachineConfig objects for your OpenShift Container Platform cluster to determine how to label your machine config:

   $ oc get MachineConfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   00-worker                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-master-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-master-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-worker-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   01-worker-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-master-1131169f-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-master-ssh                                                                                          2.2.0             30m
   99-worker-114e8ac7-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   99-worker-ssh                                                                                          2.2.0             30m
   rendered-master-b3729e5f6124ca3678188071343115d0            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m
   rendered-worker-18ff9506c718be1e8bd0a066850065b7            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             30m

2. Create a MachineConfig object file that identifies the kernel argument (for example, 05-worker-kernelarg-selinuxpermissive.yaml)

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker1
     name: 05-worker-kernelarg-selinuxpermissive2
   spec:
     config:
       ignition:
         version: 2.2.0
     kernelArguments:
       - enforcing=03

   1

   Applies the new kernel argument only to worker nodes.

   2

   Named to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode).

   3

   Identifies the exact kernel argument as enforcing=0.

3. Create the new machine config:

   $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml

4. Check the machine configs to see that the new one was added:

   $ oc get MachineConfig

   Example output

   NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
   00-master                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   00-worker                                                   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-master-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-master-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-worker-container-runtime                                 577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   01-worker-kubelet                                           577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   
   05-worker-kernelarg-selinuxpermissive                                                                  3.1.0             105s
   
   99-master-1131169f-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   99-master-ssh                                                                                          2.2.0             30m
   99-worker-114e8ac7-dae9-11e9-b5dd-12a845e8ffd8-registries   577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   99-worker-ssh                                                                                          2.2.0             31m
   rendered-master-b3729e5f6124ca3678188071343115d0            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m
   rendered-worker-18ff9506c718be1e8bd0a066850065b7            577c2d527b09cd7a481a162c50592139caa15e20   2.2.0             31m

5. Check the nodes:

   $ oc get nodes

   Example output

   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.18.3
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.18.3
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.18.3
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.18.3
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.18.3
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.18.3

   You can see that scheduling on each worker node is disabled as the change is being applied.

6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command line arguments (in /proc/cmdline on the host):

   $ oc debug node/ip-10-0-141-105.ec2.internal

   Example output

   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`
   
   sh-4.2# cat /host/proc/cmdline
   BOOT_IMAGE=/ostree/rhcos-... console=tty0 console=ttyS0,115200n8
   rootflags=defaults,prjquota rw root=UUID=fd0... ostree=/ostree/boot.0/rhcos/16...
   coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2 enforcing=0
   
   sh-4.2# exit

   You should see the enforcing=0 argument added to the other kernel arguments.

Procedure

1. Obtain the label associated with the static CRD, Machine Config Pool, for the type of node you want to configure. Perform one of the following steps:

   1. Check current labels of the desired machine config pool.

      For example:

      $  oc get machineconfigpool  --show-labels

      Example output

      NAME      CONFIG                                             UPDATED   UPDATING   DEGRADED   LABELS
      master    rendered-master-e05b81f5ca4db1d249a1bf32f9ec24fd   True      False      False      operator.machineconfiguration.openshift.io/required-for-upgrade=
      worker    rendered-worker-f50e78e1bc06d8e82327763145bfcf62   True      False      False

   2. Add a custom label to the desired machine config pool.

      For example:

      $ oc label machineconfigpool worker custom-kubelet=enabled

2. Create a kubeletconfig custom resource (CR) for your configuration change.

   For example:

   Sample configuration for a custom-config CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-config 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: enabled 2
     kubeletConfig: 3
       podsPerCore: 10
       maxPods: 250
       systemReserved:
         cpu: 2000m
         memory: 1Gi

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change, this is the label you added to the machine config pool.

   3

   Specify the new value(s) you want to change.

3. Create the CR object.

   $ oc create -f <file-name>

   For example:

   $ oc create -f master-kube-config.yaml

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a max-pods CR

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       podsPerCore: 10 3
       maxPods: 250 4

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change.

   3

   Specify the number of pods the node can run based on the number of processor cores on the node.

   4

   Specify the number of pods the node can run to a fixed value, regardless of the properties of the node.

   Note

   Setting podsPerCore to 0 disables this limit.

   In the above example, the default value for podsPerCore is 10 and the default value for maxPods is 250. This means that unless the node has 25 cores or more, by default, podsPerCore will be the limiting factor.

2. List the MachineConfigPool CRDs to see if the change is applied. The UPDATING column reports True if the change is picked up by the Machine Config Controller:

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     False      False
   worker   worker-8cecd1236b33ee3f8a5e   False     True       False

   Once the change is complete, the UPDATED column reports True.

   $ oc get machineconfigpools

   Example output

   NAME     CONFIG                        UPDATED   UPDATING   DEGRADED
   master   master-9cc2c72f205e103bb534   False     True       False
   worker   worker-8cecd1236b33ee3f8a5e   True      False      False

Procedure

1. Run:

   $ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator

1. Edit the node specification to configure pod anti-affinity:

   apiVersion: v1
   kind: Pod
   metadata:
     name: with-pod-antiaffinity
   spec:
     affinity:
       podAntiAffinity: 1
         preferredDuringSchedulingIgnoredDuringExecution: 2
         - weight: 100 3
           podAffinityTerm:
             labelSelector:
               matchExpressions:
               - key: registry 4
                 operator: In 5
                 values:
                 - default
             topologyKey: kubernetes.io/hostname

   1

   Stanza to configure pod anti-affinity.

   2

   Defines a preferred rule.

   3

   Specifies a weight for a preferred rule. The node with the highest weight is preferred.

   4

   Description of the pod label that determines when the anti-affinity rule applies. Specify a key and value for the label.

   5

   The operator represents the relationship between the label on the existing pod and the set of values in the matchExpression parameters in the specification for the new pod. Can be In, NotIn, Exists, or DoesNotExist.

   This example assumes the container image registry pod has a label of registry=default. Pod anti-affinity can use any Kubernetes match expression.

2. Enable the MatchInterPodAffinity scheduler predicate in the scheduling policy file.

1. A list of images currently running in at least one pod.
2. A list of images available on a host.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      Name:         worker
      Namespace:
      Labels:       custom-kubelet=small-pods 1

      1

      If a label has been added it appears under Labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a container garbage collection CR:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: worker-kubeconfig 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       evictionSoft: 3
         memory.available: "500Mi" 4
         nodefs.available: "10%"
         nodefs.inodesFree: "5%"
         imagefs.available: "15%"
         imagefs.inodesFree: "10%"
       evictionSoftGracePeriod:  5
         memory.available: "1m30s"
         nodefs.available: "1m30s"
         nodefs.inodesFree: "1m30s"
         imagefs.available: "1m30s"
         imagefs.inodesFree: "1m30s"
       evictionHard:
         memory.available: "200Mi"
         nodefs.available: "5%"
         nodefs.inodesFree: "4%"
         imagefs.available: "10%"
         imagefs.inodesFree: "5%"
       evictionPressureTransitionPeriod: 0s 6
       imageMinimumGCAge: 5m 7
       imageGCHighThresholdPercent: 80 8
       imageGCLowThresholdPercent: 75 9

   1

   Name for the object.

   2

   Selector label.

   3

   Type of eviction: EvictionSoft and EvictionHard.

   4

   Eviction thresholds based on a specific eviction trigger signal.

   5

   Grace periods for the soft eviction. This parameter does not apply to eviction-hard.

   6

   The duration to wait before transitioning out of an eviction pressure condition

   7

   The minimum age for an unused image before the image is removed by garbage collection.

   8

   The percent of disk usage (expressed as an integer) which triggers image garbage collection.

   9

   The percent of disk usage (expressed as an integer) to which image garbage collection attempts to free.

2. Create the object:

   $ oc create -f <file-name>.yaml

   For example:

   $ oc create -f gc-container.yaml

   Example output

   kubeletconfig.machineconfiguration.openshift.io/gc-container created

3. Verify that garbage collection is active. The Machine Config Pool you specified in the custom resource appears with UPDATING as 'true` until the change is fully implemented:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                   UPDATED   UPDATING
   master   rendered-master-546383f80705bd5aeaba93   True      False
   worker   rendered-worker-b4c51bb33ccaae6fc4a6a5   False     True

Prerequisites

1. To help you determine values for the system-reserved setting, you can introspect the resource use for a node by using the node summary API. Enter the following command for your node:

   $ oc get --raw /api/v1/nodes/<node>/proxy/stats/summary

   For example, to access the resources from cluster.node22 node, you can enter:

   $ oc get --raw /api/v1/nodes/cluster.node22/proxy/stats/summary

   Example output

   {
       "node": {
           "nodeName": "cluster.node22",
           "systemContainers": [
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 929684480915,
                       "usageNanoCores": 190998084
                   },
                   "memory": {
                       "rssBytes": 176726016,
                       "usageBytes": 1397895168,
                       "workingSetBytes": 1050509312
                   },
                   "name": "kubelet"
               },
               {
                   "cpu": {
                       "usageCoreNanoSeconds": 128521955903,
                       "usageNanoCores": 5928600
                   },
                   "memory": {
                       "rssBytes": 35958784,
                       "usageBytes": 129671168,
                       "workingSetBytes": 102416384
                   },
                   "name": "runtime"
               }
           ]
       }
   }

2. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the Machine Config Pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods