Chapter 6. Working with clusters

1. Determine expected container memory usage

   Determine expected mean and peak container memory usage, empirically if necessary (for example, by separate load testing). Remember to consider all the processes that may potentially run in parallel in the container: for example, does the main application spawn any ancillary scripts?

2. Determine risk appetite

   Determine risk appetite for eviction. If the risk appetite is low, the container should request memory according to the expected peak usage plus a percentage safety margin. If the risk appetite is higher, it may be more appropriate to request memory according to the expected mean usage.

3. Set container memory request

   Set container memory request based on the above. The more accurately the request represents the application memory usage, the better. If the request is too high, cluster and quota usage will be inefficient. If the request is too low, the chances of application eviction increase.

4. Set container memory limit, if required

   Set container memory limit, if required. Setting a limit has the effect of immediately killing a container process if the combined memory usage of all processes in the container exceeds the limit, and is therefore a mixed blessing. On the one hand, it may make unanticipated excess memory usage obvious early ("fail fast"); on the other hand it also terminates processes abruptly.

   Note that some OpenShift Container Platform clusters may require a limit value to be set; some may override the request based on the limit; and some application images rely on a limit value being set as this is easier to detect than a request value.

   If the memory limit is set, it should not be set to less than the expected peak container memory usage plus a percentage safety margin.

5. Ensure application is tuned

   Ensure application is tuned with respect to configured request and limit values, if appropriate. This step is particularly relevant to applications which pool memory, such as the JVM. The rest of this page discusses this.

1. If the container memory limit is set and the experimental options are supported by the JVM, set -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap.

   Note

   The UseCGroupMemoryLimitForHeap option has been removed in JDK 11. Use -XX:+UseContainerSupport instead.

   This sets -XX:MaxRAM to the container memory limit, and the maximum heap size (-XX:MaxHeapSize / -Xmx) to 1/-XX:MaxRAMFraction (1/4 by default).

2. Directly override one of -XX:MaxRAM, -XX:MaxHeapSize or -Xmx.

   This option involves hard-coding a value, but has the advantage of allowing a safety margin to be calculated.

1. In the OpenShift Container Platform web console, navigate to Home Projects

   1. Click Create Project.
   2. Specify clusterresourceoverride-operator as the name of the project.
   3. Click Create.

2. Navigate to Operators OperatorHub.

   1. Choose ClusterResourceOverride Operator from the list of available Operators and click Install.
   2. On the Install Operator page, make sure A specific Namespace on the cluster is selected for Installation Mode.
   3. Make sure clusterresourceoverride-operator is selected for Installed Namespace.
   4. Select an Update Channel and Approval Strategy.
   5. Click Install.

3. On the Installed Operators page, click ClusterResourceOverride.

   1. On the ClusterResourceOverride Operator details page, click Create Instance.

   2. On the Create ClusterResourceOverride page, edit the YAML template to set the overcommit values as needed:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Click Create.

4. Check the current state of the admission webhook by checking the status of the cluster custom resource:

   1. On the ClusterResourceOverride Operator page, click cluster.

   2. On the ClusterResourceOverride Details age, click YAML. The mutatingWebhookConfigurationRef section appears when the webhook is called.

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
        creationTimestamp: "2019-12-18T22:35:02Z"
        generation: 1
        name: cluster
        resourceVersion: "127622"
        selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
        uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
      spec:
        podResourceOverride:
          spec:
            cpuRequestToLimitPercent: 25
            limitCPUToMemoryPercent: 200
            memoryRequestToLimitPercent: 50
      status:
      
      ....
      
          mutatingWebhookConfigurationRef: 1
            apiVersion: admissionregistration.k8s.io/v1beta1
            kind: MutatingWebhookConfiguration
            name: clusterresourceoverrides.admission.autoscaling.openshift.io
            resourceVersion: "127621"
            uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
      
      ....

      1

      Reference to the ClusterResourceOverride admission webhook.

1. Create a namespace for the Cluster Resource Override Operator:

   1. Create a Namespace object YAML file (for example, cro-namespace.yaml) for the Cluster Resource Override Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: clusterresourceoverride-operator

   2. Create the namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-namespace.yaml

2. Create an Operator group:

   1. Create an OperatorGroup object YAML file (for example, cro-og.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: clusterresourceoverride-operator
        namespace: clusterresourceoverride-operator
      spec:
        targetNamespaces:
          - clusterresourceoverride-operator

   2. Create the Operator Group:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-og.yaml

3. Create a subscription:

   1. Create a Subscription object YAML file (for example, cro-sub.yaml) for the Cluster Resource Override Operator:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: clusterresourceoverride
        namespace: clusterresourceoverride-operator
      spec:
        channel: "4.5"
        name: clusterresourceoverride
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the subscription:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-sub.yaml

4. Create a ClusterResourceOverride custom resource (CR) object in the clusterresourceoverride-operator namespace:

   1. Change to the clusterresourceoverride-operator namespace.

      $ oc project clusterresourceoverride-operator

   2. Create a ClusterResourceOverride object YAML file (for example, cro-cr.yaml) for the Cluster Resource Override Operator:

      apiVersion: operator.autoscaling.openshift.io/v1
      kind: ClusterResourceOverride
      metadata:
          name: cluster 1
      spec:
        podResourceOverride:
          spec:
            memoryRequestToLimitPercent: 50 2
            cpuRequestToLimitPercent: 25 3
            limitCPUToMemoryPercent: 200 4

      1

      The name must be cluster.

      2

      Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

      3

      Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

      4

      Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

   3. Create the ClusterResourceOverride object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f cro-cr.yaml

5. Verify the current state of the admission webhook by checking the status of the cluster custom resource.

   $ oc get clusterresourceoverride cluster -n clusterresourceoverride-operator -o yaml

   The mutatingWebhookConfigurationRef section appears when the webhook is called.

   Example output

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"operator.autoscaling.openshift.io/v1","kind":"ClusterResourceOverride","metadata":{"annotations":{},"name":"cluster"},"spec":{"podResourceOverride":{"spec":{"cpuRequestToLimitPercent":25,"limitCPUToMemoryPercent":200,"memoryRequestToLimitPercent":50}}}}
     creationTimestamp: "2019-12-18T22:35:02Z"
     generation: 1
     name: cluster
     resourceVersion: "127622"
     selfLink: /apis/operator.autoscaling.openshift.io/v1/clusterresourceoverrides/cluster
     uid: 978fc959-1717-4bd1-97d0-ae00ee111e8d
   spec:
     podResourceOverride:
       spec:
         cpuRequestToLimitPercent: 25
         limitCPUToMemoryPercent: 200
         memoryRequestToLimitPercent: 50
   status:
   
   ....
   
       mutatingWebhookConfigurationRef: 1
         apiVersion: admissionregistration.k8s.io/v1beta1
         kind: MutatingWebhookConfiguration
         name: clusterresourceoverrides.admission.autoscaling.openshift.io
         resourceVersion: "127621"
         uid: 98b3b8ae-d5ce-462b-8ab5-a729ea8f38f3
   
   ....

   1

   Reference to the ClusterResourceOverride admission webhook.

1. Edit the ClusterResourceOverride CR:

   apiVersion: operator.autoscaling.openshift.io/v1
   kind: ClusterResourceOverride
   metadata:
   -   name: cluster
   spec:
      memoryRequestToLimitPercent: 50 1
      cpuRequestToLimitPercent: 25 2
      limitCPUToMemoryPercent: 200 3

   1

   Optional. Specify the percentage to override the container memory limit, if used, between 1-100. The default is 50.

   2

   Optional. Specify the percentage to override the container CPU limit, if used, between 1-100. The default is 25.

   3

   Optional. Specify the percentage to override the container memory limit, if used. Scaling 1Gi of RAM at 100 percent is equal to 1 CPU core. This is processed prior to overriding the CPU request, if configured. The default is 200.

2. Ensure the following label has been added to the Namespace object for each project where you want the Cluster Resource Override Operator to control overcommit:

   apiVersion: v1
   kind: Namespace
   metadata:
   
   ....
   
     labels:
       clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true" 1
   
   ....

   1

   Add this label to each project.

Prerequisites

1. Obtain the label associated with the static MachineConfigPool CRD for the type of node you want to configure. Perform one of the following steps:

   1. View the machine config pool:

      $ oc describe machineconfigpool <name>

      For example:

      $ oc describe machineconfigpool worker

      Example output

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        creationTimestamp: 2019-02-08T14:52:39Z
        generation: 1
        labels:
          custom-kubelet: small-pods 1

      1

      If a label has been added it appears under labels.

   2. If the label is not present, add a key/value pair:

      $ oc label machineconfigpool worker custom-kubelet=small-pods

Procedure

1. Create a custom resource (CR) for your configuration change.

   Sample configuration for a disabling CPU limits

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: disable-cpu-units 1
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: small-pods 2
     kubeletConfig:
       cpuCfsQuota: 3
         - "false"

   1

   Assign a name to CR.

   2

   Specify the label to apply the configuration change.

   3

   Set the cpuCfsQuota parameter to false.

1. Edit the project object file

2. Add the following annotation:

   quota.openshift.io/cluster-resource-override-enabled: "false"

3. Create the project object:

   $ oc create -f <file-name>.yaml