Chapter 8. Working with clusters

Prerequisites

1. Run the OpenShift Cluster Capacity Tool, which is available as a container image from the Red Hat Ecosystem Catalog.

2. Create a sample pod spec file:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: small-pod
        labels:
          app: guestbook
          tier: frontend
      spec:
        containers:
        - name: php-redis
          image: gcr.io/google-samples/gb-frontend:v4
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 150m
              memory: 100Mi
            requests:
              cpu: 150m
              memory: 100Mi

   2. Create the cluster role:

      $ oc create -f <file_name>.yaml

      For example:

      $ oc create -f pod-spec.yaml

1. From the terminal, log in to the Red Hat Registry:

   $ podman login registry.redhat.io

2. Pull the cluster capacity tool image:

   $ podman pull registry.redhat.io/openshift4/ose-cluster-capacity

3. Run the cluster capacity tool:

   $ podman run -v $HOME/.kube:/kube:Z -v $(pwd):/cc:Z  ose-cluster-capacity \
   /bin/cluster-capacity --kubeconfig /kube/config --<pod_spec>.yaml /cc/<pod_spec>.yaml \
   --verbose

   where:

   <pod_spec>.yaml

   Specifies the pod spec to use.

   verbose

   Outputs a detailed description of how many pods can be scheduled on each node in the cluster.

   Example output

   small-pod pod requirements:
   	- CPU: 150m
   	- Memory: 100Mi
   
   The cluster can schedule 88 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: 0/5 nodes are available: 2 Insufficient cpu,
   3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't
   tolerate.
   
   Pod distribution among nodes:
   small-pod
   	- 192.168.124.214: 45 instance(s)
   	- 192.168.124.120: 43 instance(s)

   In the above example, the number of estimated pods that can be scheduled onto the cluster is 88.

1. Create the cluster role:

   1. Create a YAML file similar to the following:

      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: cluster-capacity-role
      rules:
      - apiGroups: [""]
        resources: ["pods", "nodes", "persistentvolumeclaims", "persistentvolumes", "services", "replicationcontrollers"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["apps"]
        resources: ["replicasets", "statefulsets"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["policy"]
        resources: ["poddisruptionbudgets"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "watch", "list"]

   2. Create the cluster role by running the following command:

      $ oc create -f <file_name>.yaml

      For example:

      $ oc create sa cluster-capacity-sa

2. Create the service account:

   $ oc create sa cluster-capacity-sa -n default

3. Add the role to the service account:

   $ oc adm policy add-cluster-role-to-user cluster-capacity-role \
       system:serviceaccount:<namespace>:cluster-capacity-sa

   where:

   <namespace>

   Specifies the namespace where the pod is located.

4. Define and create the pod spec:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: small-pod
        labels:
          app: guestbook
          tier: frontend
      spec:
        containers:
        - name: php-redis
          image: gcr.io/google-samples/gb-frontend:v4
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 150m
              memory: 100Mi
            requests:
              cpu: 150m
              memory: 100Mi

   2. Create the pod by running the following command:

      $ oc create -f <file_name>.yaml

      For example:

      $ oc create -f pod.yaml

5. Created a config map object by running the following command:

   $ oc create configmap cluster-capacity-configmap \
       --from-file=pod.yaml=pod.yaml

   The cluster capacity analysis is mounted in a volume using a config map object named cluster-capacity-configmap to mount the input pod spec file pod.yaml into a volume test-volume at the path /test-pod.

6. Create the job using the below example of a job specification file:

   1. Create a YAML file similar to the following:

      apiVersion: batch/v1
      kind: Job
      metadata:
        name: cluster-capacity-job
      spec:
        parallelism: 1
        completions: 1
        template:
          metadata:
            name: cluster-capacity-pod
          spec:
              containers:
              - name: cluster-capacity
                image: openshift/origin-cluster-capacity
                imagePullPolicy: "Always"
                volumeMounts:
                - mountPath: /test-pod
                  name: test-volume
                env:
                - name: CC_INCLUSTER 1
                  value: "true"
                command:
                - "/bin/sh"
                - "-ec"
                - |
                  /bin/cluster-capacity --podspec=/test-pod/pod.yaml --verbose
              restartPolicy: "Never"
              serviceAccountName: cluster-capacity-sa
              volumes:
              - name: test-volume
                configMap:
                  name: cluster-capacity-configmap

      1

      A required environment variable letting the cluster capacity tool know that it is running inside a cluster as a pod.
      The pod.yaml key of the ConfigMap object is the same as the Pod spec file name, though it is not required. By doing this, the input pod spec file can be accessed inside the pod as /test-pod/pod.yaml.

   2. Run the cluster capacity image as a job in a pod by running the following command:

      $ oc create -f cluster-capacity-job.yaml

Verification

1. Check the job logs to find the number of pods that can be scheduled in the cluster:

   $ oc logs jobs/cluster-capacity-job

   Example output

   small-pod pod requirements:
           - CPU: 150m
           - Memory: 100Mi
   
   The cluster can schedule 52 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: No nodes are available that match all of the
   following predicates:: Insufficient cpu (2).
   
   Pod distribution among nodes:
   small-pod
           - 192.168.124.214: 26 instance(s)
           - 192.168.124.120: 26 instance(s)

1. Write an object definition JSON file, and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s:

   Limited Pod object definition

   {
       "kind": "Pod",
       "spec": {
           "containers": [
               {
                   "image": "openshift/hello-openshift",
                   "name": "hello-openshift"
               }
           ]
       },
       "apiVersion": "v1",
       "metadata": {
           "name": "iperf-slow",
           "annotations": {
               "kubernetes.io/ingress-bandwidth": "10M",
               "kubernetes.io/egress-bandwidth": "10M"
           }
       }
   }

2. Create the pod using the object definition:

   $ oc create -f <file_or_dir_path>

1. Create a Pod spec or edit existing pods to include the system-cluster-critical priority class:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pdb
   spec:
     template:
       metadata:
         name: critical-pod
       priorityClassName: system-cluster-critical 1

   1

   Default priority class for pods that should never be evicted from a node.

   Alternatively, you can specify system-node-critical for pods that are important to the cluster but can be removed if necessary.

2. Create the pod:

   $ oc create -f <file-name>.yaml

1. Create a LimitRange object with your required specifications:

   apiVersion: "v1"
   kind: "LimitRange"
   metadata:
     name: "resource-limits" 1
   spec:
     limits:
       - type: "Pod" 2
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "200m"
           memory: "6Mi"
       - type: "Container" 3
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "100m"
           memory: "4Mi"
         default: 4
           cpu: "300m"
           memory: "200Mi"
         defaultRequest: 5
           cpu: "200m"
           memory: "100Mi"
         maxLimitRequestRatio: 6
           cpu: "10"
       - type: openshift.io/Image 7
         max:
           storage: 1Gi
       - type: openshift.io/ImageStream 8
         max:
           openshift.io/image-tags: 20
           openshift.io/images: 30
       - type: "PersistentVolumeClaim" 9
         min:
           storage: "2Gi"
         max:
           storage: "50Gi"

   1

   Specify a name for the LimitRange object.

   2

   To set limits for a pod, specify the minimum and maximum CPU and memory requests as needed.

   3

   To set limits for a container, specify the minimum and maximum CPU and memory requests as needed.

   4

   Optional. For a container, specify the default amount of CPU or memory that a container can use, if not specified in the Pod spec.

   5

   Optional. For a container, specify the default amount of CPU or memory that a container can request, if not specified in the Pod spec.

   6

   Optional. For a container, specify the maximum limit-to-request ratio that can be specified in the Pod spec.

   7

   To set limits for an Image object, set the maximum size of an image that can be pushed to an OpenShift image registry.

   8

   To set limits for an image stream, set the maximum number of image tags and references that can be in the ImageStream object file, as needed.

   9

   To set limits for a persistent volume claim, set the minimum and maximum amount of storage that can be requested.

2. Create the object:

   $ oc create -f <limit_range_file> -n <project> 1

   1

   Specify the name of the YAML file you created and the project where you want the limits to apply.

1. Get the list of LimitRange object defined in the project. For example, for a project called demoproject:

   $ oc get limits -n demoproject

   NAME              CREATED AT
   resource-limits   2020-07-15T17:14:23Z

2. Describe the LimitRange object you are interested in, for example the resource-limits limit range:

   $ oc describe limits resource-limits -n demoproject

   Name:                           resource-limits
   Namespace:                      demoproject
   Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
   ----                            --------                ---     ---     --------------- -------------   -----------------------
   Pod                             cpu                     200m    2       -               -               -
   Pod                             memory                  6Mi     1Gi     -               -               -
   Container                       cpu                     100m    2       200m            300m            10
   Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
   openshift.io/Image              storage                 -       1Gi     -               -               -
   openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
   openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -
   PersistentVolumeClaim           storage                 -       50Gi    -               -               -

1. Overriding the JVM maximum heap size.
2. Encouraging the JVM to release unused memory to the operating system, if appropriate.
3. Ensuring all JVM processes within a container are appropriately configured.

Procedure

1. Configure the pod to add the MEMORY_REQUEST and MEMORY_LIMIT stanzas:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: test
      spec:
        containers:
        - name: test
          image: fedora:latest
          command:
          - sleep
          - "3600"
          env:
          - name: MEMORY_REQUEST 1
            valueFrom:
              resourceFieldRef:
                containerName: test
                resource: requests.memory
          - name: MEMORY_LIMIT 2
            valueFrom:
              resourceFieldRef:
                containerName: test
                resource: limits.memory
          resources:
            requests:
              memory: 384Mi
            limits:
              memory: 512Mi

      1

      Add this stanza to discover the application memory request value.

      2

      Add this stanza to discover the application memory limit value.

   2. Create the pod by running the following command:

      $ oc create -f <file-name>.yaml

Verification

1. Access the pod using a remote shell:

   $ oc rsh test

2. Check that the requested values were applied:

   $ env | grep MEMORY | sort

   Example output

   MEMORY_LIMIT=536870912
   MEMORY_REQUEST=402653184

1. Access the pod using a remote shell:

   # oc rsh test

2. Run the following command to see the current OOM kill count in /sys/fs/cgroup/memory/memory.oom_control:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control

   Example output

   oom_kill 0

3. Run the following command to provoke an OOM kill:

   $ sed -e '' </dev/zero

   Example output

   Killed

4. Run the following command to view the exit status of the sed command:

   $ echo $?

   Example output

   137

   The 137 code indicates the container process exited with code 137, indicating it received a SIGKILL signal.

5. Run the following command to see that the OOM kill counter in /sys/fs/cgroup/memory/memory.oom_control incremented:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control

   Example output

   oom_kill 1

   If one or more processes in a pod are OOM killed, when the pod subsequently exits, whether immediately or not, it will have phase Failed and reason OOMKilled. An OOM-killed pod might be restarted depending on the value of restartPolicy. If not restarted, controllers such as the replication controller will notice the pod’s failed status and create a new pod to replace the old one.

   Use the follwing command to get the pod status:

   $ oc get pod test

   Example output

   NAME      READY     STATUS      RESTARTS   AGE
   test      0/1       OOMKilled   0          1m

   • If the pod has not restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Example output

     ...
     status:
       containerStatuses:
       - name: test
         ready: false
         restartCount: 0
         state:
           terminated:
             exitCode: 137
             reason: OOMKilled
       phase: Failed

   • If restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Example output

     ...
     status:
       containerStatuses:
       - name: test
         ready: true
         restartCount: 1
         lastState:
           terminated:
             exitCode: 137
             reason: OOMKilled
         state:
           running:
       phase: Running

1. In the OpenShift Container Platform web console, switch to the Administration Custom Resource Definitions page.
2. On the Custom Resource Definitions page, click FeatureGate.
3. On the Custom Resource Definition Details page, click the Instances tab.
4. Click the cluster feature gate, then click the YAML tab.

5. Edit the cluster instance to add specific feature sets:

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample Feature Gate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   # ...
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

   # ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
   # ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

1. Edit the FeatureGate CR named cluster:

   $ oc edit featuregate cluster

   Warning

   Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.

   Sample FeatureGate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   # ...
   spec:
     featureSet: TechPreviewNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the feature set that you want to enable:

   • TechPreviewNoUpgrade enables specific Technology Preview features.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

1. From the Administrator perspective in the web console, navigate to Compute Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to /host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

   # ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
   # ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.