Chapter 8. Working with clusters

Prerequisites

1. Run the OpenShift Cluster Capacity Tool, which is available as a container image from the Red Hat Ecosystem Catalog. See the link in the "Additional resources" section.

2. Create a sample pod spec file:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: small-pod
        labels:
          app: guestbook
          tier: frontend
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: php-redis
          image: gcr.io/google-samples/gb-frontend:v4
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 150m
              memory: 100Mi
            requests:
              cpu: 150m
              memory: 100Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

      Copy to Clipboard Toggle word wrap

   2. Create the cluster role:

      $ oc create -f <file_name>.yaml

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc create -f pod-spec.yaml

      Copy to Clipboard Toggle word wrap

Procedure

1. From the terminal, log in to the Red Hat Registry:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the cluster capacity tool image:

   $ podman pull registry.redhat.io/openshift4/ose-cluster-capacity

   Copy to Clipboard Toggle word wrap

3. Run the cluster capacity tool:

   $ podman run -v $HOME/.kube:/kube:Z -v $(pwd):/cc:Z  ose-cluster-capacity \
   /bin/cluster-capacity --kubeconfig /kube/config --<pod_spec>.yaml /cc/<pod_spec>.yaml \
   --verbose

   Copy to Clipboard Toggle word wrap

   where:

   <pod_spec>.yaml

   Specifies the pod spec to use.

   verbose

   Outputs a detailed description of how many pods can be scheduled on each node in the cluster.

   Example output

   small-pod pod requirements:
   	- CPU: 150m
   	- Memory: 100Mi
   
   The cluster can schedule 88 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: 0/5 nodes are available: 2 Insufficient cpu,
   3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't
   tolerate.
   
   Pod distribution among nodes:
   small-pod
   	- 192.168.124.214: 45 instance(s)
   	- 192.168.124.120: 43 instance(s)

   Copy to Clipboard Toggle word wrap

   In the above example, the number of estimated pods that can be scheduled onto the cluster is 88.

Procedure

1. Create the cluster role:

   1. Create a YAML file similar to the following:

      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: cluster-capacity-role
      rules:
      - apiGroups: [""]
        resources: ["pods", "nodes", "persistentvolumeclaims", "persistentvolumes", "services", "replicationcontrollers"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["apps"]
        resources: ["replicasets", "statefulsets"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["policy"]
        resources: ["poddisruptionbudgets"]
        verbs: ["get", "watch", "list"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "watch", "list"]

      Copy to Clipboard Toggle word wrap

   2. Create the cluster role by running the following command:

      $ oc create -f <file_name>.yaml

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc create sa cluster-capacity-sa

      Copy to Clipboard Toggle word wrap

2. Create the service account:

   $ oc create sa cluster-capacity-sa -n default

   Copy to Clipboard Toggle word wrap

3. Add the role to the service account:

   $ oc adm policy add-cluster-role-to-user cluster-capacity-role \
       system:serviceaccount:<namespace>:cluster-capacity-sa

   Copy to Clipboard Toggle word wrap

   where:

   <namespace>

   Specifies the namespace where the pod is located.

4. Define and create the pod spec:

   1. Create a YAML file similar to the following:

      apiVersion: v1
      kind: Pod
      metadata:
        name: small-pod
        labels:
          app: guestbook
          tier: frontend
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: php-redis
          image: gcr.io/google-samples/gb-frontend:v4
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 150m
              memory: 100Mi
            requests:
              cpu: 150m
              memory: 100Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]

      Copy to Clipboard Toggle word wrap

   2. Create the pod by running the following command:

      $ oc create -f <file_name>.yaml

      Copy to Clipboard Toggle word wrap

      For example:

      $ oc create -f pod.yaml

      Copy to Clipboard Toggle word wrap

5. Created a config map object by running the following command:

   $ oc create configmap cluster-capacity-configmap \
       --from-file=pod.yaml=pod.yaml

   Copy to Clipboard Toggle word wrap

   The cluster capacity analysis is mounted in a volume using a config map object named cluster-capacity-configmap to mount the input pod spec file pod.yaml into a volume test-volume at the path /test-pod.

6. Create the job using the below example of a job specification file:

   1. Create a YAML file similar to the following:

      apiVersion: batch/v1
      kind: Job
      metadata:
        name: cluster-capacity-job
      spec:
        parallelism: 1
        completions: 1
        template:
          metadata:
            name: cluster-capacity-pod
          spec:
              containers:
              - name: cluster-capacity
                image: openshift/origin-cluster-capacity
                imagePullPolicy: "Always"
                volumeMounts:
                - mountPath: /test-pod
                  name: test-volume
                env:
                - name: CC_INCLUSTER
                  value: "true"
                command:
                - "/bin/sh"
                - "-ec"
                - |
                  /bin/cluster-capacity --podspec=/test-pod/pod.yaml --verbose
              restartPolicy: "Never"
              serviceAccountName: cluster-capacity-sa
              volumes:
              - name: test-volume
                configMap:
                  name: cluster-capacity-configmap

      Copy to Clipboard Toggle word wrap

      where:

      spec.template.spec.containers.env

      Specifies a required environment variable that indicates the Cluster Capacity Tool is running inside a cluster as a pod.
      The pod.yaml key of the ConfigMap object is the same as the Pod spec file name, though it is not required. By doing this, the input pod spec file can be accessed inside the pod as /test-pod/pod.yaml.

   2. Run the cluster capacity image as a job in a pod by running the following command:

      $ oc create -f cluster-capacity-job.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. Check the job logs to find the number of pods that can be scheduled in the cluster:

   $ oc logs jobs/cluster-capacity-job

   Copy to Clipboard Toggle word wrap

   Example output

   small-pod pod requirements:
           - CPU: 150m
           - Memory: 100Mi
   
   The cluster can schedule 52 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: No nodes are available that match all of the
   following predicates:: Insufficient cpu (2).
   
   Pod distribution among nodes:
   small-pod
           - 192.168.124.214: 26 instance(s)
           - 192.168.124.120: 26 instance(s)

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LimitRange object with your required specifications:

   apiVersion: "v1"
   kind: "LimitRange"
   metadata:
     name: "resource-limits"
   spec:
     limits:
       - type: "Pod"
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "200m"
           memory: "6Mi"
       - type: "Container"
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "100m"
           memory: "4Mi"
         default:
           cpu: "300m"
           memory: "200Mi"
         defaultRequest:
           cpu: "200m"
           memory: "100Mi"
         maxLimitRequestRatio:
           cpu: "10"
       - type: openshift.io/Image
         max:
           storage: 1Gi
       - type: openshift.io/ImageStream
         max:
           openshift.io/image-tags: 20
           openshift.io/images: 30
       - type: "PersistentVolumeClaim"
         min:
           storage: "2Gi"
         max:
           storage: "50Gi"

   Copy to Clipboard Toggle word wrap

   where:

   metadata.name

   Specifies a name for the LimitRange object.

   spec.limit.type.Pod

   Specifies limits for a pod, specify the minimum and maximum CPU and memory requests as needed.

   spec.limit.type.Container

   Specifies limits for a container, specify the minimum and maximum CPU and memory requests as needed.

   spec.limit.type.default

   For a container, specifies the default amount of CPU or memory that a container can use, if not specified in the Pod spec. This parameter is optional.

   spec.limit.type.defaultRequest

   For a container, specifies the default amount of CPU or memory that a container can request, if not specified in the Pod spec. This parameter is optional.

   spec.limit.type.maxLimitRequestRatio

   For a container, specifies the maximum limit-to-request ratio that can be specified in the Pod spec. This parameter is optional.

   spec.limit.type.openshift.io/Image

   Specifies limits for an image object. Set the maximum size of an image that can be pushed to an OpenShift image registry.

   spec.limit.type.openshift.io/ImageStream

   Specifies limits for an image stream. Set the maximum number of image tags and references that can be in the ImageStream object file, as needed.

   spec.limit.type.openshift.io/PersistentVolueClaim

   Specifies limits for a persistent volume claim. Set the minimum and maximum amount of storage that can be requested.

2. Create the object:

   $ oc create -f <limit_range_file> -n <project>

   Copy to Clipboard Toggle word wrap

   where:

   <limit_range_file>

   Specifies the name of the YAML file you created.

   <project>

   Specifies the project where you want the limits to apply.

Procedure

1. Get the list of LimitRange objects defined in the project. For example, for a project called demoproject:

   $ oc get limits -n demoproject

   Copy to Clipboard Toggle word wrap

   NAME              CREATED AT
   resource-limits   2020-07-15T17:14:23Z

   Copy to Clipboard Toggle word wrap

2. Describe the LimitRange object you are interested in, for example the resource-limits limit range:

   $ oc describe limits resource-limits -n demoproject

   Copy to Clipboard Toggle word wrap

   Name:                           resource-limits
   Namespace:                      demoproject
   Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
   ----                            --------                ---     ---     --------------- -------------   -----------------------
   Pod                             cpu                     200m    2       -               -               -
   Pod                             memory                  6Mi     1Gi     -               -               -
   Container                       cpu                     100m    2       200m            300m            10
   Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
   openshift.io/Image              storage                 -       1Gi     -               -               -
   openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
   openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -
   PersistentVolumeClaim           storage                 -       50Gi    -               -               -

   Copy to Clipboard Toggle word wrap

1. Determine expected container memory usage

   Determine expected mean and peak container memory usage. For example, you could perform separate load testing. Remember to consider all the processes that could potentially run in parallel in the container, such as any ancillary scripts that might be spawned by the main application.

2. Determine risk appetite

   Determine risk appetite for eviction. If the risk appetite is low, the container should request memory according to the expected peak usage plus a percentage safety margin. If the risk appetite is higher, it might be more appropriate to request memory according to the expected mean usage.

3. Set container memory request

   Set the container memory request based on the above. The request should represent the application memory usage as accurately as possible. If the request is too high, cluster and quota usage will be inefficient. If the request is too low, the chances of application eviction increase.

4. Set container memory limit, if required

   Set the container memory limit, if required. Setting a limit has the effect of immediately killing a container process if the combined memory usage of all processes in the container exceeds the limit. Setting a limit might make unanticipated excess memory usage obvious early (fail fast). However, setting a limit also terminates processes abruptly.

   Note that some Red Hat OpenShift Service on AWS clusters might require a limit value to be set; some might override the request based on the limit; and some application images rely on a limit value being set as this is easier to detect than a request value.

   If the memory limit is set, it should not be set to less than the expected peak container memory usage plus a percentage safety margin.

5. Ensure applications are tuned

   Ensure your applications are tuned with respect to configured request and limit values, if appropriate. This step is particularly relevant to applications which pool memory, such as the JVM. The rest of this page discusses this.

Verification

1. Access the pod using a remote shell:

   $ oc rsh test

   Copy to Clipboard Toggle word wrap

2. Check that the requested values were applied:

   $ env | grep MEMORY | sort

   Copy to Clipboard Toggle word wrap

   Example output

   MEMORY_LIMIT=536870912
   MEMORY_REQUEST=402653184

   Copy to Clipboard Toggle word wrap

Procedure

1. Access the pod using a remote shell:

   # oc rsh <pod name>

   Copy to Clipboard Toggle word wrap

2. Run the following command to see the current OOM kill count in /sys/fs/cgroup/memory/memory.oom_control:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control

   Copy to Clipboard Toggle word wrap

   Example output

   oom_kill 0

   Copy to Clipboard Toggle word wrap

3. Run the following command to provoke an OOM kill:

   $ sed -e '' </dev/zero

   Copy to Clipboard Toggle word wrap

   Example output

   Killed

   Copy to Clipboard Toggle word wrap

4. Run the following command to see that the OOM kill counter in /sys/fs/cgroup/memory/memory.oom_control incremented:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control

   Copy to Clipboard Toggle word wrap

   Example output

   oom_kill 1

   Copy to Clipboard Toggle word wrap

   If one or more processes in a pod are OOM killed, when the pod subsequently exits, whether immediately or not, it will have phase Failed and reason OOMKilled. An OOM-killed pod might be restarted depending on the value of restartPolicy. If not restarted, controllers such as the replication controller will notice the pod’s failed status and create a new pod to replace the old one.

   Use the following command to get the pod status:

   $ oc get pod test

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      READY     STATUS      RESTARTS   AGE
   test      0/1       OOMKilled   0          1m

   Copy to Clipboard Toggle word wrap

   • If the pod has not restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Copy to Clipboard Toggle word wrap

     Example output

     apiVersion: v1
     kind: Pod
     metadata:
       name: test
     # ...
     status:
       containerStatuses:
       - name: test
         ready: false
         restartCount: 0
         state:
           terminated:
             exitCode: 137
             reason: OOMKilled
       phase: Failed

     Copy to Clipboard Toggle word wrap

   • If restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Copy to Clipboard Toggle word wrap

     Example output

     apiVersion: v1
     kind: Pod
     metadata:
       name: test
     # ...
     status:
       containerStatuses:
       - name: test
         ready: true
         restartCount: 1
         lastState:
           terminated:
             exitCode: 137
             reason: OOMKilled
         state:
           running:
       phase: Running

     Copy to Clipboard Toggle word wrap