Chapter 8. Working with clusters

8.1. Viewing system event information in an OpenShift Container Platform cluster

Events in OpenShift Container Platform are modeled based on events that happen to API objects in an OpenShift Container Platform cluster.

8.1.1. Understanding events

Events allow OpenShift Container Platform to record information about real-world events in a resource-agnostic manner. They also allow developers and administrators to consume information about system components in a unified way.

8.1.2. Viewing events using the CLI

You can get a list of events in a given project using the CLI.

Procedure

To view events in a project use the following command:

$ oc get events [-n <project>] 1

1: The name of the project.

For example:

$ oc get events -n openshift-config

Example output

LAST SEEN   TYPE      REASON                   OBJECT                      MESSAGE
97m         Normal    Scheduled                pod/dapi-env-test-pod       Successfully assigned openshift-config/dapi-env-test-pod to ip-10-0-171-202.ec2.internal
97m         Normal    Pulling                  pod/dapi-env-test-pod       pulling image "gcr.io/google_containers/busybox"
97m         Normal    Pulled                   pod/dapi-env-test-pod       Successfully pulled image "gcr.io/google_containers/busybox"
97m         Normal    Created                  pod/dapi-env-test-pod       Created container
9m5s        Warning   FailedCreatePodSandBox   pod/dapi-volume-test-pod    Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_dapi-volume-test-pod_openshift-config_6bc60c1f-452e-11e9-9140-0eec59c23068_0(748c7a40db3d08c07fb4f9eba774bd5effe5f0d5090a242432a73eee66ba9e22): Multus: Err adding pod to network "openshift-sdn": cannot set "openshift-sdn" ifname to "eth0": no netns: failed to Statfs "/proc/33366/ns/net": no such file or directory
8m31s       Normal    Scheduled                pod/dapi-volume-test-pod    Successfully assigned openshift-config/dapi-volume-test-pod to ip-10-0-171-202.ec2.internal

To view events in your project from the OpenShift Container Platform console.
1. Launch the OpenShift Container Platform console.
2. Click Home Events and select your project.
3. Move to resource that you want to see events. For example: Home Projects <project-name> <resource-name>.
  Many objects, such as pods and deployments, have their own Events tab as well, which shows events related to that object.

8.1.3. List of events

This section describes the events of OpenShift Container Platform.

Table 8.1. Configuration events
Name	Description
`FailedValidation`	Failed pod configuration validation.

Table 8.2. Container events
Name	Description
`BackOff`	Back-off restarting failed the container.
`Created`	Container created.
`Failed`	Pull/Create/Start failed.
`Killing`	Killing the container.
`Started`	Container started.
`Preempting`	Preempting other pods.
`ExceededGracePeriod`	Container runtime did not stop the pod within specified grace period.

Table 8.3. Health events
Name	Description
`Unhealthy`	Container is unhealthy.

Table 8.4. Image events
Name	Description
`BackOff`	Back off Ctr Start, image pull.
`ErrImageNeverPull`	The image’s NeverPull Policy is violated.
`Failed`	Failed to pull the image.
`InspectFailed`	Failed to inspect the image.
`Pulled`	Successfully pulled the image or the container image is already present on the machine.
`Pulling`	Pulling the image.

Table 8.5. Image Manager events
Name	Description
`FreeDiskSpaceFailed`	Free disk space failed.
`InvalidDiskCapacity`	Invalid disk capacity.

Table 8.6. Node events
Name	Description
`FailedMount`	Volume mount failed.
`HostNetworkNotSupported`	Host network not supported.
`HostPortConflict`	Host/port conflict.
`KubeletSetupFailed`	Kubelet setup failed.
`NilShaper`	Undefined shaper.
`NodeNotReady`	Node is not ready.
`NodeNotSchedulable`	Node is not schedulable.
`NodeReady`	Node is ready.
`NodeSchedulable`	Node is schedulable.
`NodeSelectorMismatching`	Node selector mismatch.
`OutOfDisk`	Out of disk.
`Rebooted`	Node rebooted.
`Starting`	Starting kubelet.
`FailedAttachVolume`	Failed to attach volume.
`FailedDetachVolume`	Failed to detach volume.
`VolumeResizeFailed`	Failed to expand/reduce volume.
`VolumeResizeSuccessful`	Successfully expanded/reduced volume.
`FileSystemResizeFailed`	Failed to expand/reduce file system.
`FileSystemResizeSuccessful`	Successfully expanded/reduced file system.
`FailedUnMount`	Failed to unmount volume.
`FailedMapVolume`	Failed to map a volume.
`FailedUnmapDevice`	Failed unmaped device.
`AlreadyMountedVolume`	Volume is already mounted.
`SuccessfulDetachVolume`	Volume is successfully detached.
`SuccessfulMountVolume`	Volume is successfully mounted.
`SuccessfulUnMountVolume`	Volume is successfully unmounted.
`ContainerGCFailed`	Container garbage collection failed.
`ImageGCFailed`	Image garbage collection failed.
`FailedNodeAllocatableEnforcement`	Failed to enforce System Reserved Cgroup limit.
`NodeAllocatableEnforced`	Enforced System Reserved Cgroup limit.
`UnsupportedMountOption`	Unsupported mount option.
`SandboxChanged`	Pod sandbox changed.
`FailedCreatePodSandBox`	Failed to create pod sandbox.
`FailedPodSandBoxStatus`	Failed pod sandbox status.

Table 8.7. Pod worker events
Name	Description
`FailedSync`	Pod sync failed.

Table 8.8. System Events
Name	Description
`SystemOOM`	There is an OOM (out of memory) situation on the cluster.

Table 8.9. Pod events
Name	Description
`FailedKillPod`	Failed to stop a pod.
`FailedCreatePodContainer`	Failed to create a pod container.
`Failed`	Failed to make pod data directories.
`NetworkNotReady`	Network is not ready.
`FailedCreate`	Error creating: `<error-msg>`.
`SuccessfulCreate`	Created pod: `<pod-name>`.
`FailedDelete`	Error deleting: `<error-msg>`.
`SuccessfulDelete`	Deleted pod: `<pod-id>`.

Table 8.10. Horizontal Pod AutoScaler events
Name	Description
SelectorRequired	Selector is required.
`InvalidSelector`	Could not convert selector into a corresponding internal selector object.
`FailedGetObjectMetric`	HPA was unable to compute the replica count.
`InvalidMetricSourceType`	Unknown metric source type.
`ValidMetricFound`	HPA was able to successfully calculate a replica count.
`FailedConvertHPA`	Failed to convert the given HPA.
`FailedGetScale`	HPA controller was unable to get the target’s current scale.
`SucceededGetScale`	HPA controller was able to get the target’s current scale.
`FailedComputeMetricsReplicas`	Failed to compute desired number of replicas based on listed metrics.
`FailedRescale`	New size: `<size>`; reason: `<msg>`; error: `<error-msg>`.
`SuccessfulRescale`	New size: `<size>`; reason: `<msg>`.
`FailedUpdateStatus`	Failed to update status.

Table 8.11. Network events (openshift-sdn)
Name	Description
`Starting`	Starting OpenShift SDN.
`NetworkFailed`	The pod’s network interface has been lost and the pod will be stopped.

Table 8.12. Network events (kube-proxy)
Name	Description
`NeedPods`	The service-port `<serviceName>:<port>` needs pods.

Table 8.13. Volume events
Name	Description
`FailedBinding`	There are no persistent volumes available and no storage class is set.
`VolumeMismatch`	Volume size or class is different from what is requested in claim.
`VolumeFailedRecycle`	Error creating recycler pod.
`VolumeRecycled`	Occurs when volume is recycled.
`RecyclerPod`	Occurs when pod is recycled.
`VolumeDelete`	Occurs when volume is deleted.
`VolumeFailedDelete`	Error when deleting the volume.
`ExternalProvisioning`	Occurs when volume for the claim is provisioned either manually or via external software.
`ProvisioningFailed`	Failed to provision volume.
`ProvisioningCleanupFailed`	Error cleaning provisioned volume.
`ProvisioningSucceeded`	Occurs when the volume is provisioned successfully.
`WaitForFirstConsumer`	Delay binding until pod scheduling.

Table 8.14. Lifecycle hooks
Name	Description
`FailedPostStartHook`	Handler failed for pod start.
`FailedPreStopHook`	Handler failed for pre-stop.
`UnfinishedPreStopHook`	Pre-stop hook unfinished.

Table 8.15. Deployments
Name	Description
`DeploymentCancellationFailed`	Failed to cancel deployment.
`DeploymentCancelled`	Canceled deployment.
`DeploymentCreated`	Created new replication controller.
`IngressIPRangeFull`	No available Ingress IP to allocate to service.

Table 8.16. Scheduler events
Name	Description
`FailedScheduling`	Failed to schedule pod: `<pod-namespace>/<pod-name>`. This event is raised for multiple reasons, for example: `AssumePodVolumes` failed, Binding rejected etc.
`Preempted`	By `<preemptor-namespace>/<preemptor-name>` on node `<node-name>`.
`Scheduled`	Successfully assigned `<pod-name>` to `<node-name>`.

Table 8.17. Daemon set events
Name	Description
`SelectingAll`	This daemon set is selecting all pods. A non-empty selector is required.
`FailedPlacement`	Failed to place pod on `<node-name>`.
`FailedDaemonPod`	Found failed daemon pod `<pod-name>` on node `<node-name>`, will try to kill it.

Table 8.18. LoadBalancer service events
Name	Description
`CreatingLoadBalancerFailed`	Error creating load balancer.
`DeletingLoadBalancer`	Deleting load balancer.
`EnsuringLoadBalancer`	Ensuring load balancer.
`EnsuredLoadBalancer`	Ensured load balancer.
`UnAvailableLoadBalancer`	There are no available nodes for `LoadBalancer` service.
`LoadBalancerSourceRanges`	Lists the new `LoadBalancerSourceRanges`. For example, `<old-source-range> <new-source-range>`.
`LoadbalancerIP`	Lists the new IP address. For example, `<old-ip> <new-ip>`.
`ExternalIP`	Lists external IP address. For example, `Added: <external-ip>`.
`UID`	Lists the new UID. For example, `<old-service-uid> <new-service-uid>`.
`ExternalTrafficPolicy`	Lists the new `ExternalTrafficPolicy`. For example, `<old-policy> <new-policy>`.
`HealthCheckNodePort`	Lists the new `HealthCheckNodePort`. For example, `<old-node-port> new-node-port>`.
`UpdatedLoadBalancer`	Updated load balancer with new hosts.
`LoadBalancerUpdateFailed`	Error updating load balancer with new hosts.
`DeletingLoadBalancer`	Deleting load balancer.
`DeletingLoadBalancerFailed`	Error deleting load balancer.
`DeletedLoadBalancer`	Deleted load balancer.

8.2. Estimating the number of pods your OpenShift Container Platform nodes can hold

As a cluster administrator, you can use the OpenShift Cluster Capacity Tool to view the number of pods that can be scheduled to increase the current resources before they become exhausted, and to ensure any future pods can be scheduled. This capacity comes from an individual node host in a cluster, and includes CPU, memory, disk space, and others.

8.2.1. Understanding the OpenShift Cluster Capacity Tool

The OpenShift Cluster Capacity Tool simulates a sequence of scheduling decisions to determine how many instances of an input pod can be scheduled on the cluster before it is exhausted of resources to provide a more accurate estimation.

Note

The remaining allocatable capacity is a rough estimation, because it does not count all of the resources being distributed among nodes. It analyzes only the remaining resources and estimates the available capacity that is still consumable in terms of a number of instances of a pod with given requirements that can be scheduled in a cluster.

Also, pods might only have scheduling support on particular sets of nodes based on its selection and affinity criteria. As a result, the estimation of which remaining pods a cluster can schedule can be difficult.

You can run the OpenShift Cluster Capacity Tool as a stand-alone utility from the command line, or as a job in a pod inside an OpenShift Container Platform cluster. Running the tool as job inside of a pod enables you to run it multiple times without intervention.

8.2.2. Running the OpenShift Cluster Capacity Tool on the command line

You can run the OpenShift Cluster Capacity Tool from the command line to estimate the number of pods that can be scheduled onto your cluster.

You create a sample pod spec file, which the tool uses for estimating resource usage. The pod spec specifies its resource requirements as limits or requests. The cluster capacity tool takes the pod’s resource requirements into account for its estimation analysis.

Prerequisites

Run the OpenShift Cluster Capacity Tool, which is available as a container image from the Red Hat Ecosystem Catalog.

Create a sample pod spec file:

Create a YAML file similar to the following:

apiVersion: v1
kind: Pod
metadata:
  name: small-pod
  labels:
    app: guestbook
    tier: frontend
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: php-redis
    image: gcr.io/google-samples/gb-frontend:v4
    imagePullPolicy: Always
    resources:
      limits:
        cpu: 150m
        memory: 100Mi
      requests:
        cpu: 150m
        memory: 100Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]

Create the cluster role:

$ oc create -f <file_name>.yaml

For example:

$ oc create -f pod-spec.yaml

Procedure

To use the cluster capacity tool on the command line:

From the terminal, log in to the Red Hat Registry:
```
$ podman login registry.redhat.io
```

Pull the cluster capacity tool image:

$ podman pull registry.redhat.io/openshift4/ose-cluster-capacity

Run the cluster capacity tool:

$ podman run -v $HOME/.kube:/kube:Z -v $(pwd):/cc:Z  ose-cluster-capacity \
/bin/cluster-capacity --kubeconfig /kube/config --<pod_spec>.yaml /cc/<pod_spec>.yaml \
--verbose

where:

<pod_spec>.yaml: Specifies the pod spec to use.
verbose: Outputs a detailed description of how many pods can be scheduled on each node in the cluster.

Example output

small-pod pod requirements:
	- CPU: 150m
	- Memory: 100Mi

The cluster can schedule 88 instance(s) of the pod small-pod.

Termination reason: Unschedulable: 0/5 nodes are available: 2 Insufficient cpu,
3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't
tolerate.

Pod distribution among nodes:
small-pod
	- 192.168.124.214: 45 instance(s)
	- 192.168.124.120: 43 instance(s)

In the above example, the number of estimated pods that can be scheduled onto the cluster is 88.

8.2.3. Running the OpenShift Cluster Capacity Tool as a job inside a pod

Running the OpenShift Cluster Capacity Tool as a job inside of a pod allows you to run the tool multiple times without needing user intervention. You run the OpenShift Cluster Capacity Tool as a job by using a ConfigMap object.

Prerequisites

Download and install OpenShift Cluster Capacity Tool.

Procedure

To run the cluster capacity tool:

Create the cluster role:

Create a YAML file similar to the following:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-capacity-role
rules:
- apiGroups: [""]
  resources: ["pods", "nodes", "persistentvolumeclaims", "persistentvolumes", "services", "replicationcontrollers"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["apps"]
  resources: ["replicasets", "statefulsets"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "watch", "list"]

Create the cluster role by running the following command:
```
$ oc create -f <file_name>.yaml
```
For example:
```
$ oc create sa cluster-capacity-sa
```

Create the service account:

$ oc create sa cluster-capacity-sa -n default

Add the role to the service account:

$ oc adm policy add-cluster-role-to-user cluster-capacity-role \
    system:serviceaccount:<namespace>:cluster-capacity-sa

where:

<namespace>: Specifies the namespace where the pod is located.

Define and create the pod spec:

Create a YAML file similar to the following:

apiVersion: v1
kind: Pod
metadata:
  name: small-pod
  labels:
    app: guestbook
    tier: frontend
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: php-redis
    image: gcr.io/google-samples/gb-frontend:v4
    imagePullPolicy: Always
    resources:
      limits:
        cpu: 150m
        memory: 100Mi
      requests:
        cpu: 150m
        memory: 100Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]

Create the pod by running the following command:
```
$ oc create -f <file_name>.yaml
```
For example:
```
$ oc create -f pod.yaml
```

Created a config map object by running the following command:
```
$ oc create configmap cluster-capacity-configmap \
    --from-file=pod.yaml=pod.yaml
```
The cluster capacity analysis is mounted in a volume using a config map object named cluster-capacity-configmap to mount the input pod spec file pod.yaml into a volume test-volume at the path /test-pod.

Create the job using the below example of a job specification file:

Create a YAML file similar to the following:

apiVersion: batch/v1
kind: Job
metadata:
  name: cluster-capacity-job
spec:
  parallelism: 1
  completions: 1
  template:
    metadata:
      name: cluster-capacity-pod
    spec:
        containers:
        - name: cluster-capacity
          image: openshift/origin-cluster-capacity
          imagePullPolicy: "Always"
          volumeMounts:
          - mountPath: /test-pod
            name: test-volume
          env:
          - name: CC_INCLUSTER 1
            value: "true"
          command:
          - "/bin/sh"
          - "-ec"
          - |
            /bin/cluster-capacity --podspec=/test-pod/pod.yaml --verbose
        restartPolicy: "Never"
        serviceAccountName: cluster-capacity-sa
        volumes:
        - name: test-volume
          configMap:
            name: cluster-capacity-configmap

1: A required environment variable letting the cluster capacity tool know that it is running inside a cluster as a pod.
The pod.yaml key of the ConfigMap object is the same as the Pod spec file name, though it is not required. By doing this, the input pod spec file can be accessed inside the pod as /test-pod/pod.yaml.

Run the cluster capacity image as a job in a pod by running the following command:
```
$ oc create -f cluster-capacity-job.yaml
```

Verification

Check the job logs to find the number of pods that can be scheduled in the cluster:

$ oc logs jobs/cluster-capacity-job

Example output

small-pod pod requirements:
        - CPU: 150m
        - Memory: 100Mi

The cluster can schedule 52 instance(s) of the pod small-pod.

Termination reason: Unschedulable: No nodes are available that match all of the
following predicates:: Insufficient cpu (2).

Pod distribution among nodes:
small-pod
        - 192.168.124.214: 26 instance(s)
        - 192.168.124.120: 26 instance(s)

8.3. Restrict resource consumption with limit ranges

By default, containers run with unbounded compute resources on an OpenShift Container Platform cluster. With limit ranges, you can restrict resource consumption for specific objects in a project:

pods and containers: You can set minimum and maximum requirements for CPU and memory for pods and their containers.
Image streams: You can set limits on the number of images and tags in an ImageStream object.
Images: You can limit the size of images that can be pushed to an internal registry.
Persistent volume claims (PVC): You can restrict the size of the PVCs that can be requested.

If a pod does not meet the constraints imposed by the limit range, the pod cannot be created in the namespace.

8.3.1. About limit ranges

A limit range, defined by a LimitRange object, restricts resource consumption in a project. In the project you can set specific resource limits for a pod, container, image, image stream, or persistent volume claim (PVC).

All requests to create and modify resources are evaluated against each LimitRange object in the project. If the resource violates any of the enumerated constraints, the resource is rejected.

The following shows a limit range object for all components: pod, container, image, image stream, or PVC. You can configure limits for any or all of these components in the same object. You create a different limit range object for each project where you want to control resources.

Sample limit range object for a container

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits"
spec:
  limits:
    - type: "Container"
      max:
        cpu: "2"
        memory: "1Gi"
      min:
        cpu: "100m"
        memory: "4Mi"
      default:
        cpu: "300m"
        memory: "200Mi"
      defaultRequest:
        cpu: "200m"
        memory: "100Mi"
      maxLimitRequestRatio:
        cpu: "10"

8.3.1.1. About component limits

The following examples show limit range parameters for each component. The examples are broken out for clarity. You can create a single LimitRange object for any or all components as necessary.

8.3.1.1.1. Container limits

A limit range allows you to specify the minimum and maximum CPU and memory that each container in a pod can request for a specific project. If a container is created in the project, the container CPU and memory requests in the Pod spec must comply with the values set in the LimitRange object. If not, the pod does not get created.

The container CPU or memory request and limit must be greater than or equal to the min resource constraint for containers that are specified in the LimitRange object.
The container CPU or memory request and limit must be less than or equal to the max resource constraint for containers that are specified in the LimitRange object.
If the LimitRange object defines a max CPU, you do not need to define a CPU request value in the Pod spec. But you must specify a CPU limit value that satisfies the maximum CPU constraint specified in the limit range.
The ratio of the container limits to requests must be less than or equal to the maxLimitRequestRatio value for containers that is specified in the LimitRange object.
If the LimitRange object defines a maxLimitRequestRatio constraint, any new containers must have both a request and a limit value. OpenShift Container Platform calculates the limit-to-request ratio by dividing the limit by the request. This value should be a non-negative integer greater than 1.
For example, if a container has cpu: 500 in the limit value, and cpu: 100 in the request value, the limit-to-request ratio for cpu is 5. This ratio must be less than or equal to the maxLimitRequestRatio.

If the Pod spec does not specify a container resource memory or limit, the default or defaultRequest CPU and memory values for containers specified in the limit range object are assigned to the container.

Container LimitRange object definition

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: "Container"
      max:
        cpu: "2" 2
        memory: "1Gi" 3
      min:
        cpu: "100m" 4
        memory: "4Mi" 5
      default:
        cpu: "300m" 6
        memory: "200Mi" 7
      defaultRequest:
        cpu: "200m" 8
        memory: "100Mi" 9
      maxLimitRequestRatio:
        cpu: "10" 10

1: The name of the LimitRange object.
2: The maximum amount of CPU that a single container in a pod can request.
3: The maximum amount of memory that a single container in a pod can request.
4: The minimum amount of CPU that a single container in a pod can request.
5: The minimum amount of memory that a single container in a pod can request.
6: The default amount of CPU that a container can use if not specified in the Pod spec.
7: The default amount of memory that a container can use if not specified in the Pod spec.
8: The default amount of CPU that a container can request if not specified in the Pod spec.
9: The default amount of memory that a container can request if not specified in the Pod spec.
10: The maximum limit-to-request ratio for a container.

8.3.1.1.2. Pod limits

A limit range allows you to specify the minimum and maximum CPU and memory limits for all containers across a pod in a given project. To create a container in the project, the container CPU and memory requests in the Pod spec must comply with the values set in the LimitRange object. If not, the pod does not get created.

If the Pod spec does not specify a container resource memory or limit, the default or defaultRequest CPU and memory values for containers specified in the limit range object are assigned to the container.

Across all containers in a pod, the following must hold true:

The container CPU or memory request and limit must be greater than or equal to the min resource constraints for pods that are specified in the LimitRange object.
The container CPU or memory request and limit must be less than or equal to the max resource constraints for pods that are specified in the LimitRange object.
The ratio of the container limits to requests must be less than or equal to the maxLimitRequestRatio constraint specified in the LimitRange object.

Pod LimitRange object definition

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: "Pod"
      max:
        cpu: "2" 2
        memory: "1Gi" 3
      min:
        cpu: "200m" 4
        memory: "6Mi" 5
      maxLimitRequestRatio:
        cpu: "10" 6

1: The name of the limit range object.
2: The maximum amount of CPU that a pod can request across all containers.
3: The maximum amount of memory that a pod can request across all containers.
4: The minimum amount of CPU that a pod can request across all containers.
5: The minimum amount of memory that a pod can request across all containers.
6: The maximum limit-to-request ratio for a container.

8.3.1.1.3. Image limits

A LimitRange object allows you to specify the maximum size of an image that can be pushed to an OpenShift image registry.

When pushing images to an OpenShift image registry, the following must hold true:

The size of the image must be less than or equal to the max size for images that is specified in the LimitRange object.

Image LimitRange object definition

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: openshift.io/Image
      max:
        storage: 1Gi 2

1: The name of the LimitRange object.
2: The maximum size of an image that can be pushed to an OpenShift image registry.

Note

To prevent blobs that exceed the limit from being uploaded to the registry, the registry must be configured to enforce quotas.

Warning

The image size is not always available in the manifest of an uploaded image. This is especially the case for images built with Docker 1.10 or higher and pushed to a v2 registry. If such an image is pulled with an older Docker daemon, the image manifest is converted by the registry to schema v1 lacking all the size information. No storage limit set on images prevent it from being uploaded.

The issue is being addressed.

8.3.1.1.4. Image stream limits

A LimitRange object allows you to specify limits for image streams.

For each image stream, the following must hold true:

The number of image tags in an ImageStream specification must be less than or equal to the openshift.io/image-tags constraint in the LimitRange object.
The number of unique references to images in an ImageStream specification must be less than or equal to the openshift.io/images constraint in the limit range object.

Imagestream LimitRange object definition

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: openshift.io/ImageStream
      max:
        openshift.io/image-tags: 20 2
        openshift.io/images: 30 3

1: The name of the LimitRange object.
2: The maximum number of unique image tags in the imagestream.spec.tags parameter in imagestream spec.
3: The maximum number of unique image references in the imagestream.status.tags parameter in the imagestream spec.

The openshift.io/image-tags resource represents unique image references. Possible references are an ImageStreamTag, an ImageStreamImage and a DockerImage. Tags can be created using the oc tag and oc import-image commands. No distinction is made between internal and external references. However, each unique reference tagged in an ImageStream specification is counted just once. It does not restrict pushes to an internal container image registry in any way, but is useful for tag restriction.

The openshift.io/images resource represents unique image names recorded in image stream status. It allows for restriction of a number of images that can be pushed to the OpenShift image registry. Internal and external references are not distinguished.

8.3.1.1.5. Persistent volume claim limits

A LimitRange object allows you to restrict the storage requested in a persistent volume claim (PVC).

Across all persistent volume claims in a project, the following must hold true:

The resource request in a persistent volume claim (PVC) must be greater than or equal the min constraint for PVCs that is specified in the LimitRange object.
The resource request in a persistent volume claim (PVC) must be less than or equal the max constraint for PVCs that is specified in the LimitRange object.

PVC LimitRange object definition

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: "PersistentVolumeClaim"
      min:
        storage: "2Gi" 2
      max:
        storage: "50Gi" 3

1: The name of the LimitRange object.
2: The minimum amount of storage that can be requested in a persistent volume claim.
3: The maximum amount of storage that can be requested in a persistent volume claim.

8.3.2. Creating a Limit Range

To apply a limit range to a project:

Create a LimitRange object with your required specifications:
```
apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "resource-limits" 1
spec:
  limits:
    - type: "Pod" 2
      max:
        cpu: "2"
        memory: "1Gi"
      min:
        cpu: "200m"
        memory: "6Mi"
    - type: "Container" 3
      max:
        cpu: "2"
        memory: "1Gi"
      min:
        cpu: "100m"
        memory: "4Mi"
      default: 4
        cpu: "300m"
        memory: "200Mi"
      defaultRequest: 5
        cpu: "200m"
        memory: "100Mi"
      maxLimitRequestRatio: 6
        cpu: "10"
    - type: openshift.io/Image 7
      max:
        storage: 1Gi
    - type: openshift.io/ImageStream 8
      max:
        openshift.io/image-tags: 20
        openshift.io/images: 30
    - type: "PersistentVolumeClaim" 9
      min:
        storage: "2Gi"
      max:
        storage: "50Gi"
```
1
Specify a name for the LimitRange object.
2
To set limits for a pod, specify the minimum and maximum CPU and memory requests as needed.
3
To set limits for a container, specify the minimum and maximum CPU and memory requests as needed.
4
Optional. For a container, specify the default amount of CPU or memory that a container can use, if not specified in the Pod spec.
5
Optional. For a container, specify the default amount of CPU or memory that a container can request, if not specified in the Pod spec.
6
Optional. For a container, specify the maximum limit-to-request ratio that can be specified in the Pod spec.
7
To set limits for an Image object, set the maximum size of an image that can be pushed to an OpenShift image registry.
8
To set limits for an image stream, set the maximum number of image tags and references that can be in the ImageStream object file, as needed.
9
To set limits for a persistent volume claim, set the minimum and maximum amount of storage that can be requested.
Create the object:
```
$ oc create -f <limit_range_file> -n <project> 1
```
1
Specify the name of the YAML file you created and the project where you want the limits to apply.

8.3.3. Viewing a limit

You can view any limits defined in a project by navigating in the web console to the project’s Quota page.

You can also use the CLI to view limit range details:

Get the list of LimitRange object defined in the project. For example, for a project called demoproject:
```
$ oc get limits -n demoproject
```
```
NAME              CREATED AT
resource-limits   2020-07-15T17:14:23Z
```

Describe the LimitRange object you are interested in, for example the resource-limits limit range:

$ oc describe limits resource-limits -n demoproject

Name:                           resource-limits
Namespace:                      demoproject
Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
----                            --------                ---     ---     --------------- -------------   -----------------------
Pod                             cpu                     200m    2       -               -               -
Pod                             memory                  6Mi     1Gi     -               -               -
Container                       cpu                     100m    2       200m            300m            10
Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
openshift.io/Image              storage                 -       1Gi     -               -               -
openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -
PersistentVolumeClaim           storage                 -       50Gi    -               -               -

8.3.4. Deleting a Limit Range

To remove any active LimitRange object to no longer enforce the limits in a project:

Run the following command:
```
$ oc delete limits <limit_name>
```

8.4. Configuring cluster memory to meet container memory and risk requirements

As a cluster administrator, you can help your clusters operate efficiently through managing application memory by:

Determining the memory and risk requirements of a containerized application component and configuring the container memory parameters to suit those requirements.
Configuring containerized application runtimes (for example, OpenJDK) to adhere optimally to the configured container memory parameters.
Diagnosing and resolving memory-related error conditions associated with running in a container.

8.4.1. Understanding managing application memory

It is recommended to fully read the overview of how OpenShift Container Platform manages Compute Resources before proceeding.

For each kind of resource (memory, CPU, storage), OpenShift Container Platform allows optional request and limit values to be placed on each container in a pod.

Note the following about memory requests and memory limits:

Memory request
- The memory request value, if specified, influences the OpenShift Container Platform scheduler. The scheduler considers the memory request when scheduling a container to a node, then fences off the requested memory on the chosen node for the use of the container.
- If a node’s memory is exhausted, OpenShift Container Platform prioritizes evicting its containers whose memory usage most exceeds their memory request. In serious cases of memory exhaustion, the node OOM killer may select and kill a process in a container based on a similar metric.
- The cluster administrator can assign quota or assign default values for the memory request value.
- The cluster administrator can override the memory request values that a developer specifies, to manage cluster overcommit.
Memory limit
- The memory limit value, if specified, provides a hard limit on the memory that can be allocated across all the processes in a container.
- If the memory allocated by all of the processes in a container exceeds the memory limit, the node Out of Memory (OOM) killer will immediately select and kill a process in the container.
- If both memory request and limit are specified, the memory limit value must be greater than or equal to the memory request.
- The cluster administrator can assign quota or assign default values for the memory limit value.
- The minimum memory limit is 12 MB. If a container fails to start due to a Cannot allocate memory pod event, the memory limit is too low. Either increase or remove the memory limit. Removing the limit allows pods to consume unbounded node resources.

8.4.1.1. Managing application memory strategy

The steps for sizing application memory on OpenShift Container Platform are as follows:

Determine expected container memory usage
Determine expected mean and peak container memory usage, empirically if necessary (for example, by separate load testing). Remember to consider all the processes that may potentially run in parallel in the container: for example, does the main application spawn any ancillary scripts?
Determine risk appetite
Determine risk appetite for eviction. If the risk appetite is low, the container should request memory according to the expected peak usage plus a percentage safety margin. If the risk appetite is higher, it may be more appropriate to request memory according to the expected mean usage.
Set container memory request
Set container memory request based on the above. The more accurately the request represents the application memory usage, the better. If the request is too high, cluster and quota usage will be inefficient. If the request is too low, the chances of application eviction increase.
Set container memory limit, if required
Set container memory limit, if required. Setting a limit has the effect of immediately killing a container process if the combined memory usage of all processes in the container exceeds the limit, and is therefore a mixed blessing. On the one hand, it may make unanticipated excess memory usage obvious early ("fail fast"); on the other hand it also terminates processes abruptly.
Note that some OpenShift Container Platform clusters may require a limit value to be set; some may override the request based on the limit; and some application images rely on a limit value being set as this is easier to detect than a request value.
If the memory limit is set, it should not be set to less than the expected peak container memory usage plus a percentage safety margin.
Ensure application is tuned
Ensure application is tuned with respect to configured request and limit values, if appropriate. This step is particularly relevant to applications which pool memory, such as the JVM. The rest of this page discusses this.

Additional resources

Understanding compute resources and containers

8.4.2. Understanding OpenJDK settings for OpenShift Container Platform

The default OpenJDK settings do not work well with containerized environments. As a result, some additional Java memory settings must always be provided whenever running the OpenJDK in a container.

The JVM memory layout is complex, version dependent, and describing it in detail is beyond the scope of this documentation. However, as a starting point for running OpenJDK in a container, at least the following three memory-related tasks are key:

Overriding the JVM maximum heap size.
Encouraging the JVM to release unused memory to the operating system, if appropriate.
Ensuring all JVM processes within a container are appropriately configured.

Optimally tuning JVM workloads for running in a container is beyond the scope of this documentation, and may involve setting multiple additional JVM options.

8.4.2.1. Understanding how to override the JVM maximum heap size

For many Java workloads, the JVM heap is the largest single consumer of memory. Currently, the OpenJDK defaults to allowing up to 1/4 (1/-XX:MaxRAMFraction) of the compute node’s memory to be used for the heap, regardless of whether the OpenJDK is running in a container or not. It is therefore essential to override this behavior, especially if a container memory limit is also set.

There are at least two ways the above can be achieved:

If the container memory limit is set and the experimental options are supported by the JVM, set -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap.
Note
The UseCGroupMemoryLimitForHeap option has been removed in JDK 11. Use -XX:+UseContainerSupport instead.
This sets -XX:MaxRAM to the container memory limit, and the maximum heap size (-XX:MaxHeapSize / -Xmx) to 1/-XX:MaxRAMFraction (1/4 by default).
Directly override one of -XX:MaxRAM, -XX:MaxHeapSize or -Xmx.
This option involves hard-coding a value, but has the advantage of allowing a safety margin to be calculated.

8.4.2.2. Understanding how to encourage the JVM to release unused memory to the operating system

By default, the OpenJDK does not aggressively return unused memory to the operating system. This may be appropriate for many containerized Java workloads, but notable exceptions include workloads where additional active processes co-exist with a JVM within a container, whether those additional processes are native, additional JVMs, or a combination of the two.

Java-based agents can use the following JVM arguments to encourage the JVM to release unused memory to the operating system:

-XX:+UseParallelGC
-XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4
-XX:AdaptiveSizePolicyWeight=90.

These arguments are intended to return heap memory to the operating system whenever allocated memory exceeds 110% of in-use memory (-XX:MaxHeapFreeRatio), spending up to 20% of CPU time in the garbage collector (-XX:GCTimeRatio). At no time will the application heap allocation be less than the initial heap allocation (overridden by -XX:InitialHeapSize / -Xms). Detailed additional information is available Tuning Java’s footprint in OpenShift (Part 1), Tuning Java’s footprint in OpenShift (Part 2), and at OpenJDK and Containers.

8.4.2.3. Understanding how to ensure all JVM processes within a container are appropriately configured

In the case that multiple JVMs run in the same container, it is essential to ensure that they are all configured appropriately. For many workloads it will be necessary to grant each JVM a percentage memory budget, leaving a perhaps substantial additional safety margin.

Many Java tools use different environment variables (JAVA_OPTS, GRADLE_OPTS, and so on) to configure their JVMs and it can be challenging to ensure that the right settings are being passed to the right JVM.

The JAVA_TOOL_OPTIONS environment variable is always respected by the OpenJDK, and values specified in JAVA_TOOL_OPTIONS will be overridden by other options specified on the JVM command line. By default, to ensure that these options are used by default for all JVM workloads run in the Java-based agent image, the OpenShift Container Platform Jenkins Maven agent image sets:

JAVA_TOOL_OPTIONS="-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true"

Note

The UseCGroupMemoryLimitForHeap option has been removed in JDK 11. Use -XX:+UseContainerSupport instead.

This does not guarantee that additional options are not required, but is intended to be a helpful starting point.

8.4.3. Finding the memory request and limit from within a pod

An application wishing to dynamically discover its memory request and limit from within a pod should use the Downward API.

Procedure

Configure the pod to add the MEMORY_REQUEST and MEMORY_LIMIT stanzas:

Create a YAML file similar to the following:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: test
    image: fedora:latest
    command:
    - sleep
    - "3600"
    env:
    - name: MEMORY_REQUEST 1
      valueFrom:
        resourceFieldRef:
          containerName: test
          resource: requests.memory
    - name: MEMORY_LIMIT 2
      valueFrom:
        resourceFieldRef:
          containerName: test
          resource: limits.memory
    resources:
      requests:
        memory: 384Mi
      limits:
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]

1: Add this stanza to discover the application memory request value.
2: Add this stanza to discover the application memory limit value.

Create the pod by running the following command:
```
$ oc create -f <file-name>.yaml
```

Verification

Access the pod using a remote shell:
```
$ oc rsh test
```

Check that the requested values were applied:

$ env | grep MEMORY | sort

Example output

MEMORY_LIMIT=536870912
MEMORY_REQUEST=402653184

Note

The memory limit value can also be read from inside the container by the /sys/fs/cgroup/memory/memory.limit_in_bytes file.

8.4.4. Understanding OOM kill policy

OpenShift Container Platform can kill a process in a container if the total memory usage of all the processes in the container exceeds the memory limit, or in serious cases of node memory exhaustion.

When a process is Out of Memory (OOM) killed, this might result in the container exiting immediately. If the container PID 1 process receives the SIGKILL, the container will exit immediately. Otherwise, the container behavior is dependent on the behavior of the other processes.

For example, a container process exited with code 137, indicating it received a SIGKILL signal.

If the container does not exit immediately, an OOM kill is detectable as follows:

Access the pod using a remote shell:
```
# oc rsh test
```
Run the following command to see the current OOM kill count in /sys/fs/cgroup/memory/memory.oom_control:
```
$ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control
```
Example output
```
oom_kill 0
```
Run the following command to provoke an OOM kill:
```
$ sed -e '' </dev/zero
```
Example output
```
Killed
```
Run the following command to view the exit status of the sed command:
```
$ echo $?
```
Example output
```
137
```
The 137 code indicates the container process exited with code 137, indicating it received a SIGKILL signal.
Run the following command to see that the OOM kill counter in /sys/fs/cgroup/memory/memory.oom_control incremented:
```
$ grep '^oom_kill ' /sys/fs/cgroup/memory/memory.oom_control
```
Example output
```
oom_kill 1
```
If one or more processes in a pod are OOM killed, when the pod subsequently exits, whether immediately or not, it will have phase Failed and reason OOMKilled. An OOM-killed pod might be restarted depending on the value of restartPolicy. If not restarted, controllers such as the replication controller will notice the pod’s failed status and create a new pod to replace the old one.
Use the follwing command to get the pod status:
```
$ oc get pod test
```
Example output
```
NAME      READY     STATUS      RESTARTS   AGE
test      0/1       OOMKilled   0          1m
```
- If the pod has not restarted, run the following command to view the pod:
```
$ oc get pod test -o yaml
```
  Example output
```
...
status:
  containerStatuses:
  - name: test
    ready: false
    restartCount: 0
    state:
      terminated:
        exitCode: 137
        reason: OOMKilled
  phase: Failed
```
- If restarted, run the following command to view the pod:
```
$ oc get pod test -o yaml
```
  Example output
```
...
status:
  containerStatuses:
  - name: test
    ready: true
    restartCount: 1
    lastState:
      terminated:
        exitCode: 137
        reason: OOMKilled
    state:
      running:
  phase: Running
```

8.4.5. Understanding pod eviction

OpenShift Container Platform may evict a pod from its node when the node’s memory is exhausted. Depending on the extent of memory exhaustion, the eviction may or may not be graceful. Graceful eviction implies the main process (PID 1) of each container receiving a SIGTERM signal, then some time later a SIGKILL signal if the process has not exited already. Non-graceful eviction implies the main process of each container immediately receiving a SIGKILL signal.

An evicted pod has phase Failed and reason Evicted. It will not be restarted, regardless of the value of restartPolicy. However, controllers such as the replication controller will notice the pod’s failed status and create a new pod to replace the old one.

$ oc get pod test

Example output

NAME      READY     STATUS    RESTARTS   AGE
test      0/1       Evicted   0          1m

$ oc get pod test -o yaml

Example output

...
status:
  message: 'Pod The node was low on resource: [MemoryPressure].'
  phase: Failed
  reason: Evicted

8.5. Configuring your cluster to place pods on overcommitted nodes

In an overcommitted state, the sum of the container compute resource requests and limits exceeds the resources available on the system. For example, you might want to use overcommitment in development environments where a trade-off of guaranteed performance for capacity is acceptable.

Containers can specify compute resource requests and limits. Requests are used for scheduling your container and provide a minimum service guarantee. Limits constrain the amount of compute resource that can be consumed on your node.

The scheduler attempts to optimize the compute resource use across all nodes in your cluster. It places pods onto specific nodes, taking the pods' compute resource requests and nodes' available capacity into consideration.

OpenShift Container Platform administrators can control the level of overcommit and manage container density on nodes. You can configure cluster-level overcommit using the ClusterResourceOverride Operator to override the ratio between requests and limits set on developer containers. In conjunction with node overcommit and project memory and CPU limits and defaults, you can adjust the resource limit and request to achieve the desired level of overcommit.

Note

In OpenShift Container Platform, you must enable cluster-level overcommit. Node overcommitment is enabled by default. See Disabling overcommitment for a node.

8.5.1. Resource requests and overcommitment

For each compute resource, a container may specify a resource request and limit. Scheduling decisions are made based on the request to ensure that a node has enough capacity available to meet the requested value. If a container specifies limits, but omits requests, the requests are defaulted to the limits. A container is not able to exceed the specified limit on the node.

The enforcement of limits is dependent upon the compute resource type. If a container makes no request or limit, the container is scheduled to a node with no resource guarantees. In practice, the container is able to consume as much of the specified resource as is available with the lowest local priority. In low resource situations, containers that specify no resource requests are given the lowest quality of service.

Scheduling is based on resources requested, while quota and hard limits refer to resource limits, which can be set higher than requested resources. The difference between request and limit determines the level of overcommit; for instance, if a container is given a memory request of 1Gi and a memory limit of 2Gi, it is scheduled based on the 1Gi request being available on the node, but could use up to 2Gi; so it is 200% overcommitted.

8.5.2. Cluster-level overcommit using the Cluster Resource Override Operator

The Cluster Resource Override Operator is an admission webhook that allows you to control the level of overcommit and manage container density across all the nodes in your cluster. The Operator controls how nodes in specific projects can exceed defined memory and CPU limits.

You must install the Cluster Resource Override Operator using the OpenShift Container Platform console or CLI as shown in the following sections. During the installation, you create a ClusterResourceOverride custom resource (CR), where you set the level of overcommit, as shown in the following example:

apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
    name: cluster 1
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 2
      cpuRequestToLimitPercent: 25 3
      limitCPUToMemoryPercent: 200 4
# ...

1: The name must be cluster.
2: Optional. If a container memory limit has been specified or defaulted, the memory request is overridden to this percentage of the limit, between 1-100. The default is 50.
3: Optional. If a container CPU limit has been specified or defaulted, the CPU request is overridden to this percentage of the limit, between 1-100. The default is 25.
4: Optional. If a container memory limit has been specified or defaulted, the CPU limit is overridden to a percentage of the memory limit, if specified. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request (if configured). The default is 200.

Note

The Cluster Resource Override Operator overrides have no effect if limits have not been set on containers. Create a LimitRange object with default limits per individual project or configure limits in Pod specs for the overrides to apply.

When configured, overrides can be enabled per-project by applying the following label to the Namespace object for each project:

apiVersion: v1
kind: Namespace
metadata:

# ...

  labels:
    clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true"

# ...

The Operator watches for the ClusterResourceOverride CR and ensures that the ClusterResourceOverride admission webhook is installed into the same namespace as the operator.

8.5.2.1. Installing the Cluster Resource Override Operator using the web console

You can use the OpenShift Container Platform web console to install the Cluster Resource Override Operator to help control overcommit in your cluster.

Prerequisites

The Cluster Resource Override Operator has no effect if limits have not been set on containers. You must specify default limits for a project using a LimitRange object or configure limits in Pod specs for the overrides to apply.

Procedure

To install the Cluster Resource Override Operator using the OpenShift Container Platform web console:

In the OpenShift Container Platform web console, navigate to Home Projects
1. Click Create Project.
2. Specify clusterresourceoverride-operator as the name of the project.
3. Click Create.
Navigate to Operators OperatorHub.
1. Choose ClusterResourceOverride Operator from the list of available Operators and click Install.
2. On the Install Operator page, make sure A specific Namespace on the cluster is selected for Installation Mode.
3. Make sure clusterresourceoverride-operator is selected for Installed Namespace.
4. Select an Update Channel and Approval Strategy.
5. Click Install.
On the Installed Operators page, click ClusterResourceOverride.
1. On the ClusterResourceOverride Operator details page, click Create ClusterResourceOverride.
2. On the Create ClusterResourceOverride page, click YAML view and edit the YAML template to set the overcommit values as needed:
```
apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
  name: cluster 1
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 2
      cpuRequestToLimitPercent: 25 3
      limitCPUToMemoryPercent: 200 4
# ...
```
  1
  The name must be cluster.
  2
  Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
  3
  Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
  4
  Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
3. Click Create.

Check the current state of the admission webhook by checking the status of the cluster custom resource:

On the ClusterResourceOverride Operator page, click cluster.

On the ClusterResourceOverride Details page, click YAML. The mutatingWebhookConfigurationRef section appears when the webhook is called.

apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
  creationTimestamp: "2019-12-18T22:35:02Z"
  generation: 1
  name: cluster
  resourceVersion: "127622"
  selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
  uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
spec:
  podResourceOverride:
    spec:
      cpuRequestToLimitPercent: 25
      limitCPUToMemoryPercent: 200
      memoryRequestToLimitPercent: 50
status:

# ...

    mutatingWebhookConfigurationRef: 1
      apiVersion: admissionregistration.k8s.io/v1
      kind: MutatingWebhookConfiguration
      name: clusterresourceoverrides.admission.autoscaling.openshift.io
      resourceVersion: "127621"
      uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3

# ...

1: Reference to the ClusterResourceOverride admission webhook.

8.5.2.2. Installing the Cluster Resource Override Operator using the CLI

You can use the OpenShift Container Platform CLI to install the Cluster Resource Override Operator to help control overcommit in your cluster.

Prerequisites

The Cluster Resource Override Operator has no effect if limits have not been set on containers. You must specify default limits for a project using a LimitRange object or configure limits in Pod specs for the overrides to apply.

Procedure

To install the Cluster Resource Override Operator using the CLI:

Create a namespace for the Cluster Resource Override Operator:
1. Create a Namespace object YAML file (for example, cro-namespace.yaml) for the Cluster Resource Override Operator:
```
apiVersion: v1
kind: Namespace
metadata:
  name: clusterresourceoverride-operator
```
2. Create the namespace:
```
$ oc create -f <file-name>.yaml
```
  For example:
```
$ oc create -f cro-namespace.yaml
```

Create an Operator group:

Create an OperatorGroup object YAML file (for example, cro-og.yaml) for the Cluster Resource Override Operator:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: clusterresourceoverride-operator
  namespace: clusterresourceoverride-operator
spec:
  targetNamespaces:
    - clusterresourceoverride-operator

Create the Operator Group:

$ oc create -f <file-name>.yaml

For example:

$ oc create -f cro-og.yaml

Create a subscription:

Create a Subscription object YAML file (for example, cro-sub.yaml) for the Cluster Resource Override Operator:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: clusterresourceoverride
  namespace: clusterresourceoverride-operator
spec:
  channel: "4.16"
  name: clusterresourceoverride
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the subscription:

$ oc create -f <file-name>.yaml

For example:

$ oc create -f cro-sub.yaml

Create a ClusterResourceOverride custom resource (CR) object in the clusterresourceoverride-operator namespace:
1. Change to the clusterresourceoverride-operator namespace.
```
$ oc project clusterresourceoverride-operator
```
2. Create a ClusterResourceOverride object YAML file (for example, cro-cr.yaml) for the Cluster Resource Override Operator:
```
apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
    name: cluster 1
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 2
      cpuRequestToLimitPercent: 25 3
      limitCPUToMemoryPercent: 200 4
```
  1
  The name must be cluster.
  2
  Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
  3
  Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
  4
  Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
3. Create the ClusterResourceOverride object:
```
$ oc create -f <file-name>.yaml
```
  For example:
```
$ oc create -f cro-cr.yaml
```

Verify the current state of the admission webhook by checking the status of the cluster custom resource.

$ oc get clusterresourceoverride cluster -n clusterresourceoverride-operator -o yaml

The mutatingWebhookConfigurationRef section appears when the webhook is called.

Example output

apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
  creationTimestamp: "2019-12-18T22:35:02Z"
  generation: 1
  name: cluster
  resourceVersion: "127622"
  selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
  uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
spec:
  podResourceOverride:
    spec:
      cpuRequestToLimitPercent: 25
      limitCPUToMemoryPercent: 200
      memoryRequestToLimitPercent: 50
status:

# ...

    mutatingWebhookConfigurationRef: 1
      apiVersion: admissionregistration.k8s.io/v1
      kind: MutatingWebhookConfiguration
      name: clusterresourceoverrides.admission.autoscaling.openshift.io
      resourceVersion: "127621"
      uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3

# ...

1: Reference to the ClusterResourceOverride admission webhook.

8.5.2.3. Configuring cluster-level overcommit

The Cluster Resource Override Operator requires a ClusterResourceOverride custom resource (CR) and a label for each project where you want the Operator to control overcommit.

Prerequisites

The Cluster Resource Override Operator has no effect if limits have not been set on containers. You must specify default limits for a project using a LimitRange object or configure limits in Pod specs for the overrides to apply.

Procedure

To modify cluster-level overcommit:

Edit the ClusterResourceOverride CR:
```
apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
    name: cluster
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 1
      cpuRequestToLimitPercent: 25 2
      limitCPUToMemoryPercent: 200 3
# ...
```
1
Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.
2
Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.
3
Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.
Ensure the following label has been added to the Namespace object for each project where you want the Cluster Resource Override Operator to control overcommit:
```
apiVersion: v1
kind: Namespace
metadata:

# ...

  labels:
    clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true" 1

# ...
```
1
Add this label to each project.

8.5.3. Node-level overcommit

You can use various ways to control overcommit on specific nodes, such as quality of service (QOS) guarantees, CPU limits, or reserve resources. You can also disable overcommit for specific nodes and specific projects.

8.5.3.1. Understanding compute resources and containers

The node-enforced behavior for compute resources is specific to the resource type.

8.5.3.1.1. Understanding container CPU requests

A container is guaranteed the amount of CPU it requests and is additionally able to consume excess CPU available on the node, up to any limit specified by the container. If multiple containers are attempting to use excess CPU, CPU time is distributed based on the amount of CPU requested by each container.

For example, if one container requested 500m of CPU time and another container requested 250m of CPU time, then any extra CPU time available on the node is distributed among the containers in a 2:1 ratio. If a container specified a limit, it will be throttled not to use more CPU than the specified limit. CPU requests are enforced using the CFS shares support in the Linux kernel. By default, CPU limits are enforced using the CFS quota support in the Linux kernel over a 100ms measuring interval, though this can be disabled.

8.5.3.1.2. Understanding container memory requests

A container is guaranteed the amount of memory it requests. A container can use more memory than requested, but once it exceeds its requested amount, it could be terminated in a low memory situation on the node. If a container uses less memory than requested, it will not be terminated unless system tasks or daemons need more memory than was accounted for in the node’s resource reservation. If a container specifies a limit on memory, it is immediately terminated if it exceeds the limit amount.

8.5.3.2. Understanding overcomitment and quality of service classes

A node is overcommitted when it has a pod scheduled that makes no request, or when the sum of limits across all pods on that node exceeds available machine capacity.

In an overcommitted environment, it is possible that the pods on the node will attempt to use more compute resource than is available at any given point in time. When this occurs, the node must give priority to one pod over another. The facility used to make this decision is referred to as a Quality of Service (QoS) Class.

A pod is designated as one of three QoS classes with decreasing order of priority:

Table 8.19. Quality of Service Classes
Priority	Class Name	Description
1 (highest)	Guaranteed	If limits and optionally requests are set (not equal to 0) for all resources and they are equal, then the pod is classified as Guaranteed.
2	Burstable	If requests and optionally limits are set (not equal to 0) for all resources, and they are not equal, then the pod is classified as Burstable.
3 (lowest)	BestEffort	If requests and limits are not set for any of the resources, then the pod is classified as BestEffort.

Memory is an incompressible resource, so in low memory situations, containers that have the lowest priority are terminated first:

Guaranteed containers are considered top priority, and are guaranteed to only be terminated if they exceed their limits, or if the system is under memory pressure and there are no lower priority containers that can be evicted.
Burstable containers under system memory pressure are more likely to be terminated once they exceed their requests and no other BestEffort containers exist.
BestEffort containers are treated with the lowest priority. Processes in these containers are first to be terminated if the system runs out of memory.

8.5.3.2.1. Understanding how to reserve memory across quality of service tiers

You can use the qos-reserved parameter to specify a percentage of memory to be reserved by a pod in a particular QoS level. This feature attempts to reserve requested resources to exclude pods from lower OoS classes from using resources requested by pods in higher QoS classes.

OpenShift Container Platform uses the qos-reserved parameter as follows:

A value of qos-reserved=memory=100% will prevent the Burstable and BestEffort QoS classes from consuming memory that was requested by a higher QoS class. This increases the risk of inducing OOM on BestEffort and Burstable workloads in favor of increasing memory resource guarantees for Guaranteed and Burstable workloads.
A value of qos-reserved=memory=50% will allow the Burstable and BestEffort QoS classes to consume half of the memory requested by a higher QoS class.
A value of qos-reserved=memory=0% will allow a Burstable and BestEffort QoS classes to consume up to the full node allocatable amount if available, but increases the risk that a Guaranteed workload will not have access to requested memory. This condition effectively disables this feature.

8.5.3.3. Understanding swap memory and QOS

You can disable swap by default on your nodes to preserve quality of service (QOS) guarantees. Otherwise, physical resources on a node can oversubscribe, affecting the resource guarantees the Kubernetes scheduler makes during pod placement.

For example, if two guaranteed pods have reached their memory limit, each container could start using swap memory. Eventually, if there is not enough swap space, processes in the pods can be terminated due to the system being oversubscribed.

Failing to disable swap results in nodes not recognizing that they are experiencing MemoryPressure, resulting in pods not receiving the memory they made in their scheduling request. As a result, additional pods are placed on the node to further increase memory pressure, ultimately increasing your risk of experiencing a system out of memory (OOM) event.

Important

If swap is enabled, any out-of-resource handling eviction thresholds for available memory will not work as expected. Take advantage of out-of-resource handling to allow pods to be evicted from a node when it is under memory pressure, and rescheduled on an alternative node that has no such pressure.

8.5.3.4. Understanding nodes overcommitment

In an overcommitted environment, it is important to properly configure your node to provide best system behavior.

When the node starts, it ensures that the kernel tunable flags for memory management are set properly. The kernel should never fail memory allocations unless it runs out of physical memory.

To ensure this behavior, OpenShift Container Platform configures the kernel to always overcommit memory by setting the vm.overcommit_memory parameter to 1, overriding the default operating system setting.

OpenShift Container Platform also configures the kernel not to panic when it runs out of memory by setting the vm.panic_on_oom parameter to 0. A setting of 0 instructs the kernel to call oom_killer in an Out of Memory (OOM) condition, which kills processes based on priority

You can view the current setting by running the following commands on your nodes:

$ sysctl -a |grep commit

Example output

#...
vm.overcommit_memory = 0
#...

$ sysctl -a |grep panic

Example output

#...
vm.panic_on_oom = 0
#...

Note

The above flags should already be set on nodes, and no further action is required.

You can also perform the following configurations for each node:

Disable or enforce CPU limits using CPU CFS quotas
Reserve resources for system processes
Reserve memory across quality of service tiers

8.5.3.5. Disabling or enforcing CPU limits using CPU CFS quotas

Nodes by default enforce specified CPU limits using the Completely Fair Scheduler (CFS) quota support in the Linux kernel.

If you disable CPU limit enforcement, it is important to understand the impact on your node:

If a container has a CPU request, the request continues to be enforced by CFS shares in the Linux kernel.
If a container does not have a CPU request, but does have a CPU limit, the CPU request defaults to the specified CPU limit, and is enforced by CFS shares in the Linux kernel.
If a container has both a CPU request and limit, the CPU request is enforced by CFS shares in the Linux kernel, and the CPU limit has no impact on the node.

Prerequisites

Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:

$ oc edit machineconfigpool <name>

For example:

$ oc edit machineconfigpool worker

Example output

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  creationTimestamp: "2022-11-16T15:34:25Z"
  generation: 4
  labels:
    pools.operator.machineconfiguration.openshift.io/worker: "" 1
  name: worker

1: The label appears under Labels.

Tip

If the label is not present, add a key/value pair such as:

$ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

Create a custom resource (CR) for your configuration change.

Sample configuration for a disabling CPU limits

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: disable-cpu-units 1
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: "" 2
  kubeletConfig:
    cpuCfsQuota: false 3

1: Assign a name to CR.
2: Specify the label from the machine config pool.
3: Set the cpuCfsQuota parameter to false.

Run the following command to create the CR:
```
$ oc create -f <file_name>.yaml
```

8.5.3.6. Reserving resources for system processes

To provide more reliable scheduling and minimize node resource overcommitment, each node can reserve a portion of its resources for use by system daemons that are required to run on your node for your cluster to function. In particular, it is recommended that you reserve resources for incompressible resources such as memory.

Procedure

To explicitly reserve resources for non-pod processes, allocate node resources by specifying resources available for scheduling. For more details, see Allocating Resources for Nodes.

8.5.3.7. Disabling overcommitment for a node

When enabled, overcommitment can be disabled on each node.

Procedure

To disable overcommitment in a node run the following command on that node:

$ sysctl -w vm.overcommit_memory=0

8.5.4. Project-level limits

To help control overcommit, you can set per-project resource limit ranges, specifying memory and CPU limits and defaults for a project that overcommit cannot exceed.

For information on project-level resource limits, see Additional resources.

Alternatively, you can disable overcommitment for specific projects.

8.5.4.1. Disabling overcommitment for a project

When enabled, overcommitment can be disabled per-project. For example, you can allow infrastructure components to be configured independently of overcommitment.

Procedure

Create or edit the namespace object file.

Add the following annotation:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    quota.openshift.io/cluster-resource-override-enabled: "false" <.>
# ...

<.> Setting this annotation to false disables overcommit for this namespace.

8.5.5. Additional resources

8.6. Configuring the Linux cgroup version on your nodes

As of OpenShift Container Platform 4.14, OpenShift Container Platform uses Linux control group version 2 (cgroup v2) in your cluster. If you are using cgroup v1 on OpenShift Container Platform 4.13 or earlier, migrating to OpenShift Container Platform 4.14 or later will not automatically update your cgroup configuration to version 2. A fresh installation of OpenShift Container Platform 4.14 or later will use cgroup v2 by default. However, you can enable Linux control group version 1 (cgroup v1) upon installation.

Important

cgroup v1 is a deprecated feature. Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments.

For the most recent list of major functionality that has been deprecated or removed within OpenShift Container Platform, refer to the Deprecated and removed features section of the OpenShift Container Platform release notes.

cgroup v2 is the current version of the Linux cgroup API. cgroup v2 offers several improvements over cgroup v1, including a unified hierarchy, safer sub-tree delegation, new features such as Pressure Stall Information, and enhanced resource management and isolation. However, cgroup v2 has different CPU, memory, and I/O management characteristics than cgroup v1. Therefore, some workloads might experience slight differences in memory or CPU usage on clusters that run cgroup v2.

You can change between cgroup v1 and cgroup v2, as needed. Enabling cgroup v1 in OpenShift Container Platform disables all cgroup v2 controllers and hierarchies in your cluster.

Note

If you run third-party monitoring and security agents that depend on the cgroup file system, update the agents to a version that supports cgroup v2.
If you have configured cgroup v2 and run cAdvisor as a stand-alone daemon set for monitoring pods and containers, update cAdvisor to v0.43.0 or later.
If you deploy Java applications, use versions that fully support cgroup v2, such as the following packages:
- OpenJDK / HotSpot: jdk8u372, 11.0.16, 15 and later
- NodeJs 20.3.0 or later
- IBM Semeru Runtimes: jdk8u345-b01, 11.0.16.0, 17.0.4.0, 18.0.2.0 and later
- IBM SDK Java Technology Edition Version (IBM Java): 8.0.7.15 and later

8.6.1. Configuring Linux cgroup

You can enable Linux control group version 1 (cgroup v1) or Linux control group version 2 (cgroup v2) by editing the node.config object. The default is cgroup v2.

Important

cgroup v1 is a deprecated feature. Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments.

For the most recent list of major functionality that has been deprecated or removed within OpenShift Container Platform, refer to the Deprecated and removed features section of the OpenShift Container Platform release notes.

Prerequisites

You have a running OpenShift Container Platform cluster that uses version 4.12 or later.
You are logged in to the cluster as a user with administrative privileges.

Procedure

Enable cgroup v1 on nodes:

Edit the node.config object:
```
$ oc edit nodes.config/cluster
```

Edit the spec.cgroupMode parameter:

Example node.config object

apiVersion: config.openshift.io/v2
kind: Node
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-07-08T16:02:51Z"
  generation: 1
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v2
    kind: ClusterVersion
    name: version
    uid: 36282574-bf9f-409e-a6cd-3032939293eb
  resourceVersion: "1865"
  uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
spec:
  cgroupMode: "v1" 1
...

1: Specify v1 to enable cgroup v1 or v2 for cgroup v2.

Verification

Check the machine configs to see that the new machine configs were added:

$ oc get mc

Example output

NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
97-master-generated-kubelet                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
99-worker-generated-kubelet                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
99-master-ssh                                                                                 3.2.0             40m
99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
99-worker-ssh                                                                                 3.2.0             40m
rendered-master-23d4317815a5f854bd3553d689cfe2e9   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             10s 1
rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             33m
rendered-worker-dcc7f1b92892d34db74d6832bcc9ccd4   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.2.0             10s

1: New machine configs are created, as expected.

Check that the new kernelArguments were added to the new machine configs:

$ oc describe mc <name>

Example output for cgroup v2

apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 05-worker-kernelarg-selinuxpermissive
spec:
  kernelArguments:
    systemd_unified_cgroup_hierarchy=1 1
    cgroup_no_v1="all" 2
    psi=1 3

1: Enables cgroup v2 in systemd.
2: Disables cgroup v1.
3: Enables the Linux Pressure Stall Information (PSI) feature.

Example output for cgroup v1

apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 05-worker-kernelarg-selinuxpermissive
spec:
  kernelArguments:
    systemd.unified_cgroup_hierarchy=0 1
    systemd.legacy_systemd_cgroup_controller=1 2

1: Disables cgroup v2.
2: Enables cgroup v1 in systemd.

Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied:

$ oc get nodes

Example output

NAME                                       STATUS                     ROLES    AGE   VERSION
ci-ln-fm1qnwt-72292-99kt6-master-0         Ready,SchedulingDisabled   master   58m   v1.29.4
ci-ln-fm1qnwt-72292-99kt6-master-1         Ready                      master   58m   v1.29.4
ci-ln-fm1qnwt-72292-99kt6-master-2         Ready                      master   58m   v1.29.4
ci-ln-fm1qnwt-72292-99kt6-worker-a-h5gt4   Ready,SchedulingDisabled   worker   48m   v1.29.4
ci-ln-fm1qnwt-72292-99kt6-worker-b-7vtmd   Ready                      worker   48m   v1.29.4
ci-ln-fm1qnwt-72292-99kt6-worker-c-rhzkv   Ready                      worker   48m   v1.29.4

After a node returns to the Ready state, start a debug session for that node:
```
$ oc debug node/<node_name>
```
Set /host as the root directory within the debug shell:
```
sh-4.4# chroot /host
```
Check that the sys/fs/cgroup/cgroup2fs or sys/fs/cgroup/tmpfs file is present on your nodes:
```
$ stat -c %T -f /sys/fs/cgroup
```
Example output for cgroup v2
```
cgroup2fs
```
Example output for cgroup v1
```
tmpfs
```

Additional resources

OpenShift Container Platform installation overview

8.7. Enabling features using feature gates

As an administrator, you can use feature gates to enable features that are not part of the default set of features.

8.7.1. Understanding feature gates

You can use the FeatureGate custom resource (CR) to enable specific feature sets in your cluster. A feature set is a collection of OpenShift Container Platform features that are not enabled by default.

You can activate the following feature set by using the FeatureGate CR:

TechPreviewNoUpgrade. This feature set is a subset of the current Technology Preview features. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them, while leaving the features disabled on production clusters.
Warning
Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.
The following Technology Preview features are enabled by this feature set:
- External cloud providers. Enables support for external cloud providers for clusters on vSphere, AWS, Azure, and GCP. Support for OpenStack is GA. This is an internal feature that most users do not need to interact with. (ExternalCloudProvider)
- Shared Resources CSI Driver in OpenShift Builds. Enables the Container Storage Interface (CSI). (CSIDriverSharedResource)
- Swap memory on nodes. Enables swap memory use for OpenShift Container Platform workloads on a per-node basis. (NodeSwap)
- OpenStack Machine API Provider. This gate has no effect and is planned to be removed from this feature set in a future release. (MachineAPIProviderOpenStack)
- Insights Operator. Enables the InsightsDataGather CRD, which allows users to configure some Insights data gathering options. The feature set also enables the DataGather CRD, which allows users to run Insights data gathering on-demand. (InsightsConfigAPI)
- Retroactive Default Storage Class. Enables OpenShift Container Platform to retroactively assign the default storage class to PVCs if there was no default storage class when the PVC was created.(RetroactiveDefaultStorageClass)
- Dynamic Resource Allocation API. Enables a new API for requesting and sharing resources between pods and containers. This is an internal feature that most users do not need to interact with. (DynamicResourceAllocation)
- Pod security admission enforcement. Enables the restricted enforcement mode for pod security admission. Instead of only logging a warning, pods are rejected if they violate pod security standards. (OpenShiftPodSecurityAdmission)
- StatefulSet pod availability upgrading limits. Enables users to define the maximum number of statefulset pods unavailable during updates which reduces application downtime. (MaxUnavailableStatefulSet)
- MatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. (admissionWebhookMatchConditions)
- gcpLabelsTags
- vSphereStaticIPs
- routeExternalCertificate
- automatedEtcdBackup
- gcpClusterHostedDNS
- vSphereControlPlaneMachineset
- dnsNameResolver
- machineConfigNodes
- metricsServer
- installAlternateInfrastructureAWS
- sdnLiveMigration
- mixedCPUsAllocation
- managedBootImages
- onClusterBuild
- signatureStores
- DisableKubeletCloudCredentialProviders
- BareMetalLoadBalancer
- ClusterAPIInstallAWS
- ClusterAPIInstallNutanix
- ClusterAPIInstallOpenStack
- ClusterAPIInstallVSphere
- HardwareSpeed
- KMSv1
- NetworkDiagnosticsConfig
- VSphereDriverConfiguration
- ExternalOIDC
- ChunkSizeMiB
- ClusterAPIInstallGCP
- ClusterAPIInstallPowerVS
- EtcdBackendQuota
- Example
- ImagePolicy
- InsightsConfig
- InsightsOnDemandDataGather
- MetricsCollectionProfiles
- NewOLM
- NodeDisruptionPolicy
- PinnedImages
- PlatformOperators
- ServiceAccountTokenNodeBinding
- ServiceAccountTokenNodeBindingValidation
- ServiceAccountTokenPodNodeInfo
- TranslateStreamCloseWebsocketRequests
- UpgradeStatus
- VSphereMultiVCenters
- VolumeGroupSnapshot

For more information about the features activated by the TechPreviewNoUpgrade feature gate, see the following topics:

8.7.2. Enabling feature sets at installation

You can enable feature sets for all nodes in the cluster by editing the install-config.yaml file before you deploy the cluster.

Prerequisites

You have an install-config.yaml file.

Procedure

Use the featureSet parameter to specify the name of the feature set you want to enable, such as TechPreviewNoUpgrade:
Warning
Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.
Sample install-config.yaml file with an enabled feature set
```
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    aws:
      rootVolume:
        iops: 2000
        size: 500
        type: io1
      metadataService:
        authentication: Optional
      type: c5.4xlarge
      zones:
      - us-west-2c
  replicas: 3
featureSet: TechPreviewNoUpgrade
```
Save the file and reference it when using the installation program to deploy the cluster.

Verification

You can verify that the feature gates are enabled by looking at the kubelet.conf file on a node after the nodes return to the ready state.

From the Administrator perspective in the web console, navigate to Compute Nodes.
Select a node.
In the Node details page, click Terminal.
In the terminal window, change your root directory to /host:
```
sh-4.2# chroot /host
```
View the kubelet.conf file:
```
sh-4.2# cat /etc/kubernetes/kubelet.conf
```
Sample output
```
# ...
featureGates:
  InsightsOperatorPullingSCA: true,
  LegacyNodeRoleBehavior: false
# ...
```
The features that are listed as true are enabled on your cluster.
Note
The features listed vary depending upon the OpenShift Container Platform version.

8.7.3. Enabling feature sets using the web console

You can use the OpenShift Container Platform web console to enable feature sets for all of the nodes in a cluster by editing the FeatureGate custom resource (CR).

Procedure

To enable feature sets:

In the OpenShift Container Platform web console, switch to the Administration Custom Resource Definitions page.
On the Custom Resource Definitions page, click FeatureGate.
On the Custom Resource Definition Details page, click the Instances tab.
Click the cluster feature gate, then click the YAML tab.
Edit the cluster instance to add specific feature sets:
Warning
Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.
Sample Feature Gate custom resource
```
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster 1
# ...
spec:
  featureSet: TechPreviewNoUpgrade 2
```
1
The name of the FeatureGate CR must be cluster.
2
Add the feature set that you want to enable:
TechPreviewNoUpgrade enables specific Technology Preview features.
After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

Verification

You can verify that the feature gates are enabled by looking at the kubelet.conf file on a node after the nodes return to the ready state.

From the Administrator perspective in the web console, navigate to Compute Nodes.
Select a node.
In the Node details page, click Terminal.
In the terminal window, change your root directory to /host:
```
sh-4.2# chroot /host
```
View the kubelet.conf file:
```
sh-4.2# cat /etc/kubernetes/kubelet.conf
```
Sample output
```
# ...
featureGates:
  InsightsOperatorPullingSCA: true,
  LegacyNodeRoleBehavior: false
# ...
```
The features that are listed as true are enabled on your cluster.
Note
The features listed vary depending upon the OpenShift Container Platform version.

8.7.4. Enabling feature sets using the CLI

You can use the OpenShift CLI (oc) to enable feature sets for all of the nodes in a cluster by editing the FeatureGate custom resource (CR).

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

To enable feature sets:

Edit the FeatureGate CR named cluster:
```
$ oc edit featuregate cluster
```
Warning
Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. You should not enable this feature set on production clusters.
Sample FeatureGate custom resource
```
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster 1
# ...
spec:
  featureSet: TechPreviewNoUpgrade 2
```
1
The name of the FeatureGate CR must be cluster.
2
Add the feature set that you want to enable:
TechPreviewNoUpgrade enables specific Technology Preview features.
After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.

Verification

You can verify that the feature gates are enabled by looking at the kubelet.conf file on a node after the nodes return to the ready state.

From the Administrator perspective in the web console, navigate to Compute Nodes.
Select a node.
In the Node details page, click Terminal.
In the terminal window, change your root directory to /host:
```
sh-4.2# chroot /host
```
View the kubelet.conf file:
```
sh-4.2# cat /etc/kubernetes/kubelet.conf
```
Sample output
```
# ...
featureGates:
  InsightsOperatorPullingSCA: true,
  LegacyNodeRoleBehavior: false
# ...
```
The features that are listed as true are enabled on your cluster.
Note
The features listed vary depending upon the OpenShift Container Platform version.

8.8. Improving cluster stability in high latency environments using worker latency profiles

If the cluster administrator has performed latency tests for platform verification, they can discover the need to adjust the operation of the cluster to ensure stability in cases of high latency. The cluster administrator needs to change only one parameter, recorded in a file, which controls four parameters affecting how supervisory processes read status and interpret the health of the cluster. Changing only the one parameter provides cluster tuning in an easy, supportable manner.

The Kubelet process provides the starting point for monitoring cluster health. The Kubelet sets status values for all nodes in the OpenShift Container Platform cluster. The Kubernetes Controller Manager (kube controller) reads the status values every 10 seconds, by default. If the kube controller cannot read a node status value, it loses contact with that node after a configured period. The default behavior is:

The node controller on the control plane updates the node health to Unhealthy and marks the node Ready condition`Unknown`.
In response, the scheduler stops scheduling pods to that node.
The Node Lifecycle Controller adds a node.kubernetes.io/unreachable taint with a NoExecute effect to the node and schedules any pods on the node for eviction after five minutes, by default.

This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager might not receive an update from a healthy node due to network latency. The Kubelet evicts pods from the node even though the node is healthy.

To avoid this problem, you can use worker latency profiles to adjust the frequency that the Kubelet and the Kubernetes Controller Manager wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly if network latency between the control plane and the worker nodes is not optimal.

These worker latency profiles contain three sets of parameters that are predefined with carefully tuned values to control the reaction of the cluster to increased latency. There is no need to experimentally find the best values manually.

You can configure worker latency profiles when installing a cluster or at any time you notice increased latency in your cluster network.

8.8.1. Understanding worker latency profiles

Worker latency profiles are four different categories of carefully-tuned parameters. The four parameters which implement these values are node-status-update-frequency, node-monitor-grace-period, default-not-ready-toleration-seconds and default-unreachable-toleration-seconds. These parameters can use values which allow you to control the reaction of the cluster to latency issues without needing to determine the best values by using manual methods.

Important

Setting these parameters manually is not supported. Incorrect parameter settings adversely affect cluster stability.

All worker latency profiles configure the following parameters:

node-status-update-frequency: Specifies how often the kubelet posts node status to the API server.
node-monitor-grace-period: Specifies the amount of time in seconds that the Kubernetes Controller Manager waits for an update from a kubelet before marking the node unhealthy and adding the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint to the node.
default-not-ready-toleration-seconds: Specifies the amount of time in seconds after marking a node unhealthy that the Kube API Server Operator waits before evicting pods from that node.
default-unreachable-toleration-seconds: Specifies the amount of time in seconds after marking a node unreachable that the Kube API Server Operator waits before evicting pods from that node.

The following Operators monitor the changes to the worker latency profiles and respond accordingly:

The Machine Config Operator (MCO) updates the node-status-update-frequency parameter on the worker nodes.
The Kubernetes Controller Manager updates the node-monitor-grace-period parameter on the control plane nodes.
The Kubernetes API Server Operator updates the default-not-ready-toleration-seconds and default-unreachable-toleration-seconds parameters on the control plane nodes.

Although the default configuration works in most cases, OpenShift Container Platform offers two other worker latency profiles for situations where the network is experiencing higher latency than usual. The three worker latency profiles are described in the following sections:

Default worker latency profile

With the Default profile, each Kubelet updates its status every 10 seconds (node-status-update-frequency). The Kube Controller Manager checks the statuses of Kubelet every 5 seconds.

The Kubernetes Controller Manager waits 40 seconds (node-monitor-grace-period) for a status update from Kubelet before considering the Kubelet unhealthy. If no status is made available to the Kubernetes Controller Manager, it then marks the node with the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint and evicts the pods on that node.

If a pod is on a node that has the NoExecute taint, the pod runs according to tolerationSeconds. If the node has no taint, it will be evicted in 300 seconds (default-not-ready-toleration-seconds and default-unreachable-toleration-seconds settings of the Kube API Server).

Profile	Component	Parameter	Value
Default	kubelet	`node-status-update-frequency`	10s
	Kubelet Controller Manager	`node-monitor-grace-period`	40s
	Kubernetes API Server Operator	`default-not-ready-toleration-seconds`	300s
	Kubernetes API Server Operator	`default-unreachable-toleration-seconds`	300s

Medium worker latency profile

Use the MediumUpdateAverageReaction profile if the network latency is slightly higher than usual.

The MediumUpdateAverageReaction profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds parameter, the eviction waits for the period specified by that parameter.

The Kubernetes Controller Manager waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts.

Profile	Component	Parameter	Value
MediumUpdateAverageReaction	kubelet	`node-status-update-frequency`	20s
	Kubelet Controller Manager	`node-monitor-grace-period`	2m
	Kubernetes API Server Operator	`default-not-ready-toleration-seconds`	60s
	Kubernetes API Server Operator	`default-unreachable-toleration-seconds`	60s

Low worker latency profile

Use the LowUpdateSlowReaction profile if the network latency is extremely high.

The LowUpdateSlowReaction profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds parameter, the eviction waits for the period specified by that parameter.

The Kubernetes Controller Manager waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts.

Profile	Component	Parameter	Value
LowUpdateSlowReaction	kubelet	`node-status-update-frequency`	1m
	Kubelet Controller Manager	`node-monitor-grace-period`	5m
	Kubernetes API Server Operator	`default-not-ready-toleration-seconds`	60s
	Kubernetes API Server Operator	`default-unreachable-toleration-seconds`	60s

8.8.2. Using and changing worker latency profiles

To change a worker latency profile to deal with network latency, edit the node.config object to add the name of the profile. You can change the profile at any time as latency increases or decreases.

You must move one worker latency profile at a time. For example, you cannot move directly from the Default profile to the LowUpdateSlowReaction worker latency profile. You must move from the Default worker latency profile to the MediumUpdateAverageReaction profile first, then to LowUpdateSlowReaction. Similarly, when returning to the Default profile, you must move from the low profile to the medium profile first, then to Default.

Note

You can also configure worker latency profiles upon installing an OpenShift Container Platform cluster.

Procedure

To move from the default worker latency profile:

Move to the medium worker latency profile:

Edit the node.config object:
```
$ oc edit nodes.config/cluster
```

Add spec.workerLatencyProfile: MediumUpdateAverageReaction:

Example node.config object

apiVersion: config.openshift.io/v1
kind: Node
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-07-08T16:02:51Z"
  generation: 1
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 36282574-bf9f-409e-a6cd-3032939293eb
  resourceVersion: "1865"
  uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
spec:
  workerLatencyProfile: MediumUpdateAverageReaction 1

# ...

1: Specifies the medium worker latency policy.

Scheduling on each worker node is disabled as the change is being applied.

Optional: Move to the low worker latency profile:

Edit the node.config object:
```
$ oc edit nodes.config/cluster
```

Change the spec.workerLatencyProfile value to LowUpdateSlowReaction:

Example node.config object

apiVersion: config.openshift.io/v1
kind: Node
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-07-08T16:02:51Z"
  generation: 1
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 36282574-bf9f-409e-a6cd-3032939293eb
  resourceVersion: "1865"
  uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
spec:
  workerLatencyProfile: LowUpdateSlowReaction 1

# ...

1: Specifies use of the low worker latency policy.

Scheduling on each worker node is disabled as the change is being applied.

Verification

When all nodes return to the Ready condition, you can use the following command to look in the Kubernetes Controller Manager to ensure it was applied:

$ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5

Example output

# ...
    - lastTransitionTime: "2022-07-11T19:47:10Z"
      reason: ProfileUpdated
      status: "False"
      type: WorkerLatencyProfileProgressing
    - lastTransitionTime: "2022-07-11T19:47:10Z" 1
      message: all static pod revision(s) have updated latency profile
      reason: ProfileUpdated
      status: "True"
      type: WorkerLatencyProfileComplete
    - lastTransitionTime: "2022-07-11T19:20:11Z"
      reason: AsExpected
      status: "False"
      type: WorkerLatencyProfileDegraded
    - lastTransitionTime: "2022-07-11T19:20:36Z"
      status: "False"
# ...

1: Specifies that the profile is applied and active.

To change the medium profile to default or change the default to medium, edit the node.config object and set the spec.workerLatencyProfile parameter to the appropriate value.

8.1. Viewing system event information in an OpenShift Container Platform cluster

8.1.1. Understanding events

8.1.2. Viewing events using the CLI

8.1.3. List of events

8.2. Estimating the number of pods your OpenShift Container Platform nodes can hold

8.2.1. Understanding the OpenShift Cluster Capacity Tool

8.2.2. Running the OpenShift Cluster Capacity Tool on the command line

8.2.3. Running the OpenShift Cluster Capacity Tool as a job inside a pod

8.3. Restrict resource consumption with limit ranges

8.3.1. About limit ranges

8.3.1.1. About component limits

8.3.1.1.1. Container limits

8.3.1.1.2. Pod limits

8.3.1.1.3. Image limits

8.3.1.1.4. Image stream limits

8.3.1.1.5. Persistent volume claim limits

8.3.2. Creating a Limit Range

8.3.3. Viewing a limit

8.3.4. Deleting a Limit Range

8.4. Configuring cluster memory to meet container memory and risk requirements

8.4.1. Understanding managing application memory

8.4.1.1. Managing application memory strategy

8.4.2. Understanding OpenJDK settings for OpenShift Container Platform

8.4.2.1. Understanding how to override the JVM maximum heap size

8.4.2.2. Understanding how to encourage the JVM to release unused memory to the operating system

8.4.2.3. Understanding how to ensure all JVM processes within a container are appropriately configured

8.4.3. Finding the memory request and limit from within a pod

8.4.4. Understanding OOM kill policy

8.4.5. Understanding pod eviction

8.5. Configuring your cluster to place pods on overcommitted nodes

8.5.1. Resource requests and overcommitment

8.5.2. Cluster-level overcommit using the Cluster Resource Override Operator

8.5.2.1. Installing the Cluster Resource Override Operator using the web console

8.5.2.2. Installing the Cluster Resource Override Operator using the CLI

8.5.2.3. Configuring cluster-level overcommit

8.5.3. Node-level overcommit

8.5.3.1. Understanding compute resources and containers

8.5.3.1.1. Understanding container CPU requests

8.5.3.1.2. Understanding container memory requests

8.5.3.2. Understanding overcomitment and quality of service classes

8.5.3.2.1. Understanding how to reserve memory across quality of service tiers

8.5.3.3. Understanding swap memory and QOS

8.5.3.4. Understanding nodes overcommitment

8.5.3.5. Disabling or enforcing CPU limits using CPU CFS quotas

8.5.3.6. Reserving resources for system processes

8.5.3.7. Disabling overcommitment for a node

8.5.4. Project-level limits

8.5.4.1. Disabling overcommitment for a project

8.5.5. Additional resources

8.6. Configuring the Linux cgroup version on your nodes

8.6.1. Configuring Linux cgroup

8.7. Enabling features using feature gates

8.7.1. Understanding feature gates

8.7.2. Enabling feature sets at installation

8.7.3. Enabling feature sets using the web console

8.7.4. Enabling feature sets using the CLI

8.8. Improving cluster stability in high latency environments using worker latency profiles

8.8.1. Understanding worker latency profiles

8.8.2. Using and changing worker latency profiles

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links