Chapter 2. Controlling pod placement onto nodes (scheduling)

1. Create a JSON file named policy.cfg with the desired predicates and priorities.

   Sample scheduler JSON file

   {
   "kind" : "Policy",
   "apiVersion" : "v1",
   "predicates" : [ 1
           {"name" : "MaxGCEPDVolumeCount"},
           {"name" : "GeneralPredicates"},
           {"name" : "MaxAzureDiskVolumeCount"},
           {"name" : "MaxCSIVolumeCountPred"},
           {"name" : "CheckVolumeBinding"},
           {"name" : "MaxEBSVolumeCount"},
           {"name" : "MatchInterPodAffinity"},
           {"name" : "CheckNodeUnschedulable"},
           {"name" : "NoDiskConflict"},
           {"name" : "NoVolumeZoneConflict"},
           {"name" : "PodToleratesNodeTaints"}
           ],
   "priorities" : [ 2
           {"name" : "LeastRequestedPriority", "weight" : 1},
           {"name" : "BalancedResourceAllocation", "weight" : 1},
           {"name" : "ServiceSpreadingPriority", "weight" : 1},
           {"name" : "NodePreferAvoidPodsPriority", "weight" : 1},
           {"name" : "NodeAffinityPriority", "weight" : 1},
           {"name" : "TaintTolerationPriority", "weight" : 1},
           {"name" : "ImageLocalityPriority", "weight" : 1},
           {"name" : "SelectorSpreadPriority", "weight" : 1},
           {"name" : "InterPodAffinityPriority", "weight" : 1},
           {"name" : "EqualPriority", "weight" : 1}
           ]
   }

   1

   Add the predicates as needed.

   2

   Add the priorities as needed.

2. Create a config map based on the scheduler JSON file:

   $ oc create configmap -n openshift-config --from-file=policy.cfg <configmap-name> 1

   1

   Enter a name for the config map.

   For example:

   $ oc create configmap -n openshift-config --from-file=policy.cfg scheduler-policy
   
   configmap/scheduler-policy created

3. Edit the Scheduler Operator custom resource to add the config map:

   $ oc patch Scheduler cluster --type='merge' -p '{"spec":{"policy":{"name":"<configmap-name>"}}}' --type=merge 1

   1

   Specify the name of the config map.

   For example:

   $ oc patch Scheduler cluster --type='merge' -p '{"spec":{"policy":{"name":"scheduler-policy"}}}' --type=merge

   After making the change to the Scheduler config resource, wait for the openshift-kube-apiserver pods to redeploy. This can take several minutes. Until the pods redeploy, new scheduler does not take effect.

4. Verify the scheduler policy is configured by viewing the log of a scheduler pod in the openshift-kube-scheduler namespace. The following command checks for the predicates and priorities that are being registered by the scheduler:

   $ oc logs <scheduler-pod> | grep predicates

   For example:

   $ oc logs openshift-kube-scheduler-ip-10-0-141-29.ec2.internal | grep predicates
   
   Creating scheduler with fit predicates 'map[MaxGCEPDVolumeCount:{} MaxAzureDiskVolumeCount:{} CheckNodeUnschedulable:{} NoDiskConflict:{} NoVolumeZoneConflict:{} GeneralPredicates:{} MaxCSIVolumeCountPred:{} CheckVolumeBinding:{} MaxEBSVolumeCount:{} MatchInterPodAffinity:{} PodToleratesNodeTaints:{}]' and priority functions 'map[InterPodAffinityPriority:{} LeastRequestedPriority:{} ServiceSpreadingPriority:{} ImageLocalityPriority:{} SelectorSpreadPriority:{} EqualPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{} TaintTolerationPriority:{}]'

Procedure

1. Create a pod with a specific label in the Pod spec:

   $ cat team4.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: security-s1
     labels:
       security: S1
   spec:
     containers:
     - name: security-s1
       image: docker.io/ocpqe/hello-pod

2. When creating other pods, edit the Pod spec as follows:

   1. Use the podAntiAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter or preferredDuringSchedulingIgnoredDuringExecution parameter:

   2. Specify the key and value that must be met. If you want the new pod to be scheduled with the other pod, use the same key and value parameters as the label on the first pod.

          podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: security
                  operator: In
                  values:
                  - S1
              topologyKey: failure-domain.beta.kubernetes.io/zone

   3. Specify an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.
   4. Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

3. Create the pod.

   $ oc create -f <pod-spec>.yaml

Procedure

1. Create a pod with a specific label in the Pod spec:

   $ cat team4.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: security-s2
     labels:
       security: S2
   spec:
     containers:
     - name: security-s2
       image: docker.io/ocpqe/hello-pod

2. When creating other pods, edit the Pod spec to set the following parameters:

3. Use the podAntiAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter or preferredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify a weight for the node, 1-100. The node that with highest weight is preferred.

   2. Specify the key and values that must be met. If you want the new pod to not be scheduled with the other pod, use the same key and value parameters as the label on the first pod.

          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: security
                    operator: In
                    values:
                    - S2
                topologyKey: kubernetes.io/hostname

   3. For a preferred rule, specify a weight, 1-100.
   4. Specify an operator. The operator can be In, NotIn, Exists, or DoesNotExist. For example, use the operator In to require the label to be in the node.
4. Specify a topologyKey, which is a prepopulated Kubernetes label that the system uses to denote such a topology domain.

5. Create the pod.

   $ oc create -f <pod-spec>.yaml

1. Add a label to a node using the oc label node command:

   $ oc label node node1 e2e-az-name=e2e-az1

2. In the Pod spec, use the nodeAffinity stanza to configure the requiredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and value parameters as the label in the node.

   2. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the operator In to require the label to be in the node:

      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: e2e-az-name
                  operator: In
                  values:
                  - e2e-az1
                  - e2e-az2

3. Create the pod:

   $ oc create -f e2e-az2.yaml

1. Add a label to a node using the oc label node command:

   $ oc label node node1 e2e-az-name=e2e-az3

2. In the Pod spec, use the nodeAffinity stanza to configure the preferredDuringSchedulingIgnoredDuringExecution parameter:

   1. Specify a weight for the node, as a number 1-100. The node with highest weight is preferred.

   2. Specify the key and values that must be met. If you want the new pod to be scheduled on the node you edited, use the same key and value parameters as the label in the node:

      spec:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                - key: e2e-az-name
                  operator: In
                  values:
                  - e2e-az3

   3. Specify an operator. The operator can be In, NotIn, Exists, DoesNotExist, Lt, or Gt. For example, use the Operator In to require the label to be in the node.

3. Create the pod.

   $ oc create -f e2e-az3.yaml

1. If you add a NoSchedule taint to a master node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

Procedure

1. Add a toleration to a pod by editing the Pod spec to include a tolerations stanza:

   Sample pod configuration file with an Equal operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1" 1
           value: "value1"
           operator: "Equal"
           effect: "NoExecute"
           tolerationSeconds: 3600 2

   1

   The toleration parameters, as described in the Taint and toleration components table.

   2

   The tolerationSeconds parameter specifies how long a pod can remain bound to a node before being evicted.

   For example:

   Sample pod configuration file with an Exists operator

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key1"
           operator: "Exists" 1
           effect: "NoExecute"
           tolerationSeconds: 3600

   1

   The Exists operator does not take a value.

   This example places a taint on node1 that has key key1, value value1, and taint effect NoExecute.

2. Add a taint to a node by using the following command with the parameters described in the Taint and toleration components table:

   $ oc adm taint nodes <node_name> <key>=<value>:<effect>

   For example:

   $ oc adm taint nodes node1 key1=value1:NoExecute

   This command places a taint on node1 that has key key1, value value1, and effect NoExecute.

   Note

   If you add a NoSchedule taint to a master node, the node must have the node-role.kubernetes.io/master=:NoSchedule taint, which is added by default.

   For example:

   apiVersion: v1
   kind: Node
   metadata:
     annotations:
       machine.openshift.io/machine: openshift-machine-api/ci-ln-62s7gtb-f76d1-v8jxv-master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-cdc1ab7da414629332cc4c3926e6e59c
   ...
   spec:
     taints:
     - effect: NoSchedule
       key: node-role.kubernetes.io/master
   ...

   The tolerations on the Pod match the taint on the node. A pod with either toleration can be scheduled onto node1.

1. To remove a taint from a node:

   $ oc adm taint nodes <node-name> <key>-

   For example:

   $ oc adm taint nodes ip-10-0-132-248.ec2.internal key1-
   
   node/ip-10-0-132-248.ec2.internal untainted

2. To remove a toleration from a pod, edit the Pod spec to remove the toleration:

   spec:
   ....
     template:
   ....
       spec:
         tolerations:
         - key: "key2"
           operator: "Exists"
           effect: "NoExecute"
           tolerationSeconds: 3600

Procedure

1. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add a node selector to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet abc612-msrtw-worker-us-east-1c  --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the label is added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api

        Example MachineSet object

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        
        ....
        
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
        ....

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east

     2. Verify that the label is added to the node:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                          STATUS   ROLES    AGE   VERSION
        ip-10-0-142-25.ec2.internal   Ready    worker   17m   v1.18.3+002a51f

2. Add the matching node selector a pod:

   • To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:

     Example ReplicaSet object

     kind: ReplicaSet
     
     ....
     
     spec:
     
     ....
     
       template:
         metadata:
           creationTimestamp: null
           labels:
             ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
             pod-template-hash: 66d5cf9464
         spec:
           nodeSelector:
             beta.kubernetes.io/os: linux
             node-role.kubernetes.io/worker: ''
             type: user-node 1

     1

     Add the node selector.

   • To add a node selector to a specific pod, add the selector to the Pod object directly:

     Example Pod object

     apiVersion: v1
     kind: Pod
     
     ...
     
     spec:
       nodeSelector:
         <key>: <value>
     
     ...

     For example:

     Example Pod object with a node selector

     apiVersion: v1
     kind: Pod
     
     ....
     
     spec:
       nodeSelector:
         region: east
         type: user-node

1. Edit the Scheduler Operator custom resource to add the cluster node selectors:

   $ oc edit scheduler cluster

   apiVersion: config.openshift.io/v1
   kind: Scheduler
   metadata:
     name: cluster
   ...
   
   spec:
     defaultNodeSelector: type=user-node,region=east 1
     mastersSchedulable: false
     policy:
       name: ""

   1 1

   Add a node selector with the appropriate <key>:<value> pairs.

   After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster node selector does not take effect until the pods redeploy.

2. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add a node selector to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api 1

        1

        Add a <key>/<value> pair for each node selector.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the label is added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        Example output

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node

     3. Redeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. When the node is ready and available, verify that the label is added to the node by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

   • Add labels directly to a node:

     1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east

     2. Verify that the label is added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

1. Create a project or edit an existing project to add the openshift.io/node-selector parameter:

   $ oc edit project <name>

   apiVersion: v1
   kind: Project
   metadata:
     annotations:
       openshift.io/node-selector: "type=user-node,region=east" 1
       openshift.io/sa.scc.mcs: s0:c17,c14
       openshift.io/sa.scc.supplemental-groups: 1000300000/10000
       openshift.io/sa.scc.uid-range: 1000300000/10000
     creationTimestamp: 2019-06-10T14:39:45Z
     labels:
       openshift.io/run-level: "0"
     name: demo
     resourceVersion: "401885"
     selfLink: /api/v1/namespaces/openshift-kube-apiserver
     uid: 96ecc54b-8b8d-11e9-9f54-0a9ae641edd0
   spec:
     finalizers:
     - kubernetes
   status:
     phase: Active

   1

   Add the openshift.io/node-selector with the appropriate <key>:<value> pairs.

2. Add labels to a node by using a machine set or editing the node directly:

   • Use a MachineSet object to add labels to nodes managed by the machine set when a node is created:

     1. Run the following command to add a node selector to a MachineSet object:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api

     2. Verify that the label is added to the MachineSet object by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        Example output

        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node

     3. Redeploy the nodes associated with that machine set:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api

     4. Verify that the label is added to the Node object, when the node is ready and available, using the oc get command:

        $ oc label MachineSet abc612-msrtw-worker-us-east-1c type=user-node region=east

        For example:

        $ oc get nodes -l type=user-node

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f

   • Add labels directly to a node:

     1. Edit the Node object to add labels:

        $ oc label <resource> <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-c-tgq49 type=user-node region=east

     2. Verify that the label is added to the Node object using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east

        Example output

        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Create the required namespace for the Kube Descheduler Operator.

   1. Navigate to Administration Namespaces and click Create Namespace.
   2. Enter openshift-kube-descheduler-operator in the Name field and click Create.

3. Install the Kube Descheduler Operator.

   1. Navigate to Operators OperatorHub.
   2. Type Kube Descheduler Operator into the filter box.
   3. Select the Kube Descheduler Operator and click Install.
   4. On the Create Operator Subscription page, select A specific namespace on the cluster. Select openshift-kube-descheduler-operator from the drop-down menu.
   5. Adjust the values for the Update Channel and Approval Strategy to the desired values.
   6. Click Subscribe.

4. Create a descheduler instance.

   1. From the Operators Installed Operators page, click the Kube Descheduler Operator.
   2. Select the Kube Descheduler tab and click Create KubeDescheduler.
   3. Edit the settings as necessary and click Create.

Procedure

1. Edit the KubeDescheduler object:

   $ oc edit kubedeschedulers.operator.openshift.io cluster -n openshift-kube-descheduler-operator

2. Specify one or more strategies in the spec.strategies section.

   apiVersion: operator.openshift.io/v1beta1
   kind: KubeDescheduler
   metadata:
     name: cluster
     namespace: openshift-kube-descheduler-operator
   spec:
     deschedulingIntervalSeconds: 3600
     strategies:
       - name: "LowNodeUtilization" 1
         params:
          - name: "cputhreshold"
            value: "10"
          - name: "memorythreshold"
            value: "20"
          - name: "podsthreshold"
            value: "30"
          - name: "memorytargetthreshold"
            value: "40"
          - name: "cputargetthreshold"
            value: "50"
          - name: "podstargetthreshold"
            value: "60"
          - name: "nodes"
            value: "3"
       - name: "RemoveDuplicates" 2

   1

   The LowNodeUtilization strategy provides additional parameters, such as cputhreshold and memorythreshold, that you can optionally configure.

   2

   The RemoveDuplicates, RemovePodsViolatingInterPodAntiAffinity, RemovePodsViolatingNodeAffinity, and RemovePodsViolatingNodeTaints strategies do not have any additional parameters to configure.

   You can enable multiple strategies and the order that the strategies are specified in is not important.

3. Save the file to apply the changes.

Procedure

1. Edit the KubeDescheduler object:

   $ oc edit kubedeschedulers.operator.openshift.io cluster -n openshift-kube-descheduler-operator

2. Configure additional settings as necessary:

   apiVersion: operator.openshift.io/v1beta1
   kind: KubeDescheduler
   metadata:
     name: cluster
     namespace: openshift-kube-descheduler-operator
   spec:
     deschedulingIntervalSeconds: 3600 1
     flags:
     - --dry-run 2
     image: quay.io/openshift/origin-descheduler:4.4 3
   ...

   1

   Set number of seconds between descheduler runs. A value of 0 in this field runs the descheduler once and exits.

   2

   Set one or more flags to append to the descheduler pod. This flag must be in the format ready to pass to the binary.

   3

   Set the descheduler container image to deploy.

3. Save the file to apply the changes.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Delete the descheduler instance.

   1. From the Operators Installed Operators page, click Kube Descheduler Operator.
   2. Select the Kube Descheduler tab.
   3. Click the Options menu next to the cluster entry and select Delete KubeDescheduler.
   4. In the confirmation dialog, click Delete.

3. Uninstall the Kube Descheduler Operator.

   1. Navigate to Operators Installed Operators,
   2. Click the Options menu next to the Kube Descheduler Operator entry and select Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

4. Delete the openshift-kube-descheduler-operator namespace.

   1. Navigate to Administration Namespaces.
   2. Enter openshift-kube-descheduler-operator into the filter box.
   3. Click the Options menu next to the openshift-kube-descheduler-operator entry and select Delete Namespace.
   4. In the confirmation dialog, enter openshift-kube-descheduler-operator and click Delete.

5. Delete the KubeDescheduler CRD.

   1. Navigate to Administration Custom Resource Definitions.
   2. Enter KubeDescheduler into the filter box.
   3. Click the Options menu next to the KubeDescheduler entry and select Delete CustomResourceDefinition.
   4. In the confirmation dialog, click Delete.