Chapter 7. Working with clusters

1. From the terminal, log in to the Red Hat Registry:

   $ podman login registry.redhat.io

2. Pull the cluster capacity tool image:

   $ podman pull registry.redhat.io/openshift4/ose-cluster-capacity

3. Run the cluster capacity tool:

   $ podman run -v $HOME/.kube:/kube:Z -v $(pwd):/cc:Z  ose-cluster-capacity \
   /bin/cluster-capacity --kubeconfig /kube/config --podspec /cc/pod-spec.yaml \
   --verbose 1

   1

   You can also add the --verbose option to output a detailed description of how many pods can be scheduled on each node in the cluster.

   Example output

   small-pod pod requirements:
   	- CPU: 150m
   	- Memory: 100Mi
   
   The cluster can schedule 88 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: 0/5 nodes are available: 2 Insufficient cpu,
   3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't
   tolerate.
   
   Pod distribution among nodes:
   small-pod
   	- 192.168.124.214: 45 instance(s)
   	- 192.168.124.120: 43 instance(s)

   In the above example, the number of estimated pods that can be scheduled onto the cluster is 88.

1. Create the cluster role:

   $ cat << EOF| oc create -f -

   Example output

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: cluster-capacity-role
   rules:
   - apiGroups: [""]
     resources: ["pods", "nodes", "persistentvolumeclaims", "persistentvolumes", "services", "replicationcontrollers"]
     verbs: ["get", "watch", "list"]
   - apiGroups: ["apps"]
     resources: ["replicasets", "statefulsets"]
     verbs: ["get", "watch", "list"]
   - apiGroups: ["policy"]
     resources: ["poddisruptionbudgets"]
     verbs: ["get", "watch", "list"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["storageclasses"]
     verbs: ["get", "watch", "list"]
   EOF

2. Create the service account:

   $ oc create sa cluster-capacity-sa

3. Add the role to the service account:

   $ oc adm policy add-cluster-role-to-user cluster-capacity-role \
       system:serviceaccount:default:cluster-capacity-sa

4. Define and create the Pod spec:

   apiVersion: v1
   kind: Pod
   metadata:
     name: small-pod
     labels:
       app: guestbook
       tier: frontend
   spec:
     containers:
     - name: php-redis
       image: gcr.io/google-samples/gb-frontend:v4
       imagePullPolicy: Always
       resources:
         limits:
           cpu: 150m
           memory: 100Mi
         requests:
           cpu: 150m
           memory: 100Mi

5. The cluster capacity analysis is mounted in a volume using a ConfigMap object named cluster-capacity-configmap to mount input pod spec file pod.yaml into a volume test-volume at the path /test-pod.

   If you haven’t created a ConfigMap object, create one before creating the job:

   $ oc create configmap cluster-capacity-configmap \
       --from-file=pod.yaml=pod.yaml

6. Create the job using the below example of a job specification file:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: cluster-capacity-job
   spec:
     parallelism: 1
     completions: 1
     template:
       metadata:
         name: cluster-capacity-pod
       spec:
           containers:
           - name: cluster-capacity
             image: openshift/origin-cluster-capacity
             imagePullPolicy: "Always"
             volumeMounts:
             - mountPath: /test-pod
               name: test-volume
             env:
             - name: CC_INCLUSTER 1
               value: "true"
             command:
             - "/bin/sh"
             - "-ec"
             - |
               /bin/cluster-capacity --podspec=/test-pod/pod.yaml --verbose
           restartPolicy: "Never"
           serviceAccountName: cluster-capacity-sa
           volumes:
           - name: test-volume
             configMap:
               name: cluster-capacity-configmap

   1

   A required environment variable letting the cluster capacity tool know that it is running inside a cluster as a pod.
   The pod.yaml key of the ConfigMap object is the same as the Pod spec file name, though it is not required. By doing this, the input pod spec file can be accessed inside the pod as /test-pod/pod.yaml.

7. Run the cluster capacity image as a job in a pod:

   $ oc create -f cluster-capacity-job.yaml

8. Check the job logs to find the number of pods that can be scheduled in the cluster:

   $ oc logs jobs/cluster-capacity-job

   Example output

   small-pod pod requirements:
           - CPU: 150m
           - Memory: 100Mi
   
   The cluster can schedule 52 instance(s) of the pod small-pod.
   
   Termination reason: Unschedulable: No nodes are available that match all of the
   following predicates:: Insufficient cpu (2).
   
   Pod distribution among nodes:
   small-pod
           - 192.168.124.214: 26 instance(s)
           - 192.168.124.120: 26 instance(s)

1. Create a LimitRange object with your required specifications:

   apiVersion: "v1"
   kind: "LimitRange"
   metadata:
     name: "resource-limits" 1
   spec:
     limits:
       - type: "Pod" 2
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "200m"
           memory: "6Mi"
       - type: "Container" 3
         max:
           cpu: "2"
           memory: "1Gi"
         min:
           cpu: "100m"
           memory: "4Mi"
         default: 4
           cpu: "300m"
           memory: "200Mi"
         defaultRequest: 5
           cpu: "200m"
           memory: "100Mi"
         maxLimitRequestRatio: 6
           cpu: "10"
       - type: openshift.io/Image 7
         max:
           storage: 1Gi
       - type: openshift.io/ImageStream 8
         max:
           openshift.io/image-tags: 20
           openshift.io/images: 30
       - type: "PersistentVolumeClaim" 9
         min:
           storage: "2Gi"
         max:
           storage: "50Gi"

   1

   Specify a name for the LimitRange object.

   2

   To set limits for a pod, specify the minimum and maximum CPU and memory requests as needed.

   3

   To set limits for a container, specify the minimum and maximum CPU and memory requests as needed.

   4

   Optional. For a container, specify the default amount of CPU or memory that a container can use, if not specified in the Pod spec.

   5

   Optional. For a container, specify the default amount of CPU or memory that a container can request, if not specified in the Pod spec.

   6

   Optional. For a container, specify the maximum limit-to-request ratio that can be specified in the Pod spec.

   7

   To set limits for an Image object, set the maximum size of an image that can be pushed to an internal registry.

   8

   To set limits for an image stream, set the maximum number of image tags and references that can be in the ImageStream object file, as needed.

   9

   To set limits for a persistent volume claim, set the minimum and maximum amount of storage that can be requested.

2. Create the object:

   $ oc create -f <limit_range_file> -n <project> 1

   1

   Specify the name of the YAML file you created and the project where you want the limits to apply.

1. Get the list of LimitRange object defined in the project. For example, for a project called demoproject:

   $ oc get limits -n demoproject

   NAME              CREATED AT
   resource-limits   2020-07-15T17:14:23Z

2. Describe the LimitRange object you are interested in, for example the resource-limits limit range:

   $ oc describe limits resource-limits -n demoproject

   Name:                           resource-limits
   Namespace:                      demoproject
   Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
   ----                            --------                ---     ---     --------------- -------------   -----------------------
   Pod                             cpu                     200m    2       -               -               -
   Pod                             memory                  6Mi     1Gi     -               -               -
   Container                       cpu                     100m    2       200m            300m            10
   Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
   openshift.io/Image              storage                 -       1Gi     -               -               -
   openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
   openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -
   PersistentVolumeClaim           storage                 -       50Gi    -               -               -

1. Run the following command:

   $ oc delete limits <limit_name>

1. Overriding the JVM maximum heap size.
2. Encouraging the JVM to release unused memory to the operating system, if appropriate.
3. Ensuring all JVM processes within a container are appropriately configured.

Procedure

1. Configure the pod to add the MEMORY_REQUEST and MEMORY_LIMIT stanzas:

   apiVersion: v1
   kind: Pod
   metadata:
     name: test
   spec:
     containers:
     - name: test
       image: fedora:latest
       command:
       - sleep
       - "3600"
       env:
       - name: MEMORY_REQUEST 1
         valueFrom:
           resourceFieldRef:
             containerName: test
             resource: requests.memory
       - name: MEMORY_LIMIT 2
         valueFrom:
           resourceFieldRef:
             containerName: test
             resource: limits.memory
       resources:
         requests:
           memory: 384Mi
         limits:
           memory: 512Mi

   1

   Add this stanza to discover the application memory request value.

   2

   Add this stanza to discover the application memory limit value.

2. Create the pod:

   $ oc create -f <file-name>.yaml

3. Access the pod using a remote shell:

   $ oc rsh test

4. Check that the requested values were applied:

   $ env | grep MEMORY | sort

   Example output

   MEMORY_LIMIT=536870912
   MEMORY_REQUEST=402653184

1. Access the pod using a remote shell:

   # oc rsh test

2. Run the following command to see the current OOM kill count in /sys/fs/cgroup/memory/memory.oom_control:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control
   oom_kill 0

3. Run the following command to provoke an OOM kill:

   $ sed -e '' </dev/zero

   Example output

   Killed

4. Run the following command to view the exit status of the sed command:

   $ echo $?

   Example output

   137

   The 137 code indicates the container process exited with code 137, indicating it received a SIGKILL signal.

5. Run the following command to see that the OOM kill counter in /sys/fs/cgroup/memory/memory.oom_control incremented:

   $ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control
   oom_kill 1

   If one or more processes in a pod are OOM killed, when the pod subsequently exits, whether immediately or not, it will have phase Failed and reason OOMKilled. An OOM-killed pod might be restarted depending on the value of restartPolicy. If not restarted, controllers such as the replication controller will notice the pod’s failed status and create a new pod to replace the old one.

   Use the follwing command to get the pod status:

   $ oc get pod test

   Example output

   NAME      READY     STATUS      RESTARTS   AGE
   test      0/1       OOMKilled   0          1m

   • If the pod has not restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Example output

     ...
     status:
       containerStatuses:
       - name: test
         ready: false
         restartCount: 0
         state:
           terminated:
             exitCode: 137
             reason: OOMKilled
       phase: Failed

   • If restarted, run the following command to view the pod:

     $ oc get pod test -o yaml

     Example output

     ...
     status:
       containerStatuses:
       - name: test
         ready: true
         restartCount: 1
         lastState:
           terminated:
             exitCode: 137
             reason: OOMKilled
         state:
           running:
       phase: Running

1. In the OpenShift Container Platform web console, switch to the Administration Custom Resource Definitions page.
2. On the Custom Resource Definitions page, click FeatureGate.
3. On the Custom Resource Definition Details page, click the Instances tab.
4. Click the cluster feature gate, then click the YAML tab.

5. Edit the cluster instance to add specific feature sets:

   Sample Feature Gate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   ....
   
   spec:
     featureSet: IPv6DualStackNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the IPv6DualStackNoUpgrade feature set to enable the dual-stack networking mode.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

   Note

   Enabling the IPv6DualStackNoUpgrade feature set cannot be undone and prevents updates. This feature set is not recommended on production clusters.

1. From the Administrator perspective in the web console, navigate to Compute Nodes.
2. Select a node.
3. In the Node details page, click Terminal.

4. In the terminal window, change your root directory to the host:

   sh-4.2# chroot /host

5. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

    ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
    ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.

1. Edit the FeatureGate CR named cluster:

   $ oc edit featuregate cluster

   Sample FeatureGate custom resource

   apiVersion: config.openshift.io/v1
   kind: FeatureGate
   metadata:
     name: cluster 1
   spec:
     featureSet: IPv6DualStackNoUpgrade 2

   1

   The name of the FeatureGate CR must be cluster.

   2

   Add the IPv6DualStackNoUpgrade feature set to enable the dual-stack networking mode.

   After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

   Note

   Enabling the IPv6DualStackNoUpgrade feature set cannot be undone and prevents updates. This feature set is not recommended on production clusters.

1. Start a debug session for a node:

   $ oc debug node/<node_name>

2. Change your root directory to the host:

   sh-4.2# chroot /host

3. View the kubelet.conf file:

   sh-4.2# cat /etc/kubernetes/kubelet.conf

   Sample output

    ...
   featureGates:
     InsightsOperatorPullingSCA: true,
     LegacyNodeRoleBehavior: false
    ...

   The features that are listed as true are enabled on your cluster.

   Note

   The features listed vary depending upon the OpenShift Container Platform version.