Scalability and performance

Procedure

1. Run:

   $ oc get machineconfig

   Copy to Clipboard Toggle word wrap

   This provides a list of the available machine configuration objects you can select. By default, the two kubelet-related configs are 01-master-kubelet and 01-worker-kubelet.

2. To check the current value of max Pods per node, run:

   # oc describe node <node-ip> | grep Allocatable -A6

   Copy to Clipboard Toggle word wrap

   Look for value: pods: <value>.

   For example:

   # oc describe node ip-172-31-128-158.us-east-2.compute.internal | grep Allocatable -A6
   Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3500m
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15341844Ki
    pods:                        250

   Copy to Clipboard Toggle word wrap

3. To set the max Pods per node on the worker nodes, create a YAML file that contains the kubelet configuration. For example, max-worker-pods.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigSelector: 01-worker-kubelet
     kubeletConfig:
       maxPods: 250

   Copy to Clipboard Toggle word wrap

   The rate at which the kubelet talks to the API server depends on queries per second (QPS) and burst values. The default values, 5 for kubeAPIQPS and 10 for kubeAPIBurst, are good enough if there are limited pods running on each node. Updating the kubelet QPS and burst rates is recommended if there are enough CPU and memory resources on the node:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: set-max-pods
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: large-pods
     kubeletConfig:
       maxPods: <pod_count>
       kubeAPIBurst: <burst_rate>
       kubeAPIQPS: <QPS>

   Copy to Clipboard Toggle word wrap

   1. Run:

      $ oc label machineconfigpool worker custom-kubelet=large-pods

      Copy to Clipboard Toggle word wrap

   2. Run:

      $ oc create -f change-maxPods-cr.yaml

      Copy to Clipboard Toggle word wrap

   3. Run:

      $ oc get kubeletconfig

      Copy to Clipboard Toggle word wrap

      This should return set-max-pods.

      Depending on the number of worker nodes in the cluster, wait for the worker nodes to be rebooted one by one. For a cluster with 3 worker nodes, this could take about 10 to 15 minutes.

4. Check for maxPods changing for the worker nodes:

   $ oc describe node

   Copy to Clipboard Toggle word wrap

   1. Verify the change by running:

      $ oc get kubeletconfigs set-max-pods -o yaml

      Copy to Clipboard Toggle word wrap

      This should show a status of True and type:Success.

1. Set maxUnavailable to the desired value.

   spec:
     maxUnavailable: <node_count>

   Copy to Clipboard Toggle word wrap
   Important

   When setting the value, consider the number of worker nodes that can be unavailable without affecting the applications running on the cluster.

Procedure

1. Run:

   $ oc get Tuned/default -o yaml -n openshift-cluster-node-tuning-operator

   Copy to Clipboard Toggle word wrap

Procedure

1. To pull the origin-tests container image, run:

   $ sudo podman pull quay.io/openshift/origin-tests:4.1

   Copy to Clipboard Toggle word wrap

Procedure

1. Execute Cluster Loader using the built-in test configuration, which deploys five template builds and waits for them to complete:

   $ sudo podman run -v ${LOCAL_KUBECONFIG}:/root/.kube/config -i
   quay.io/openshift/origin-tests:4.1 /bin/bash -c 'export KUBECONFIG=/root/.kube/config && \
   openshift-tests run-test "[Feature:Performance][Serial][Slow] Load cluster should load the \
   cluster [Suite:openshift]"'

   Copy to Clipboard Toggle word wrap

   Alternatively, execute Cluster Loader with a user-defined configuration by setting the environment variable for VIPERCONFIG:

   $ sudo podman run -v ${LOCAL_KUBECONFIG}:/root/.kube/config -i
   quay.io/openshift/origin-tests:4.1 /bin/bash -c 'export KUBECONFIG=/root/.kube/config && \
   export VIPERCONFIG=config/test && \
   openshift-tests run-test "[Feature:Performance][Serial][Slow] Load cluster should load the \
   cluster [Suite:openshift]"'

   Copy to Clipboard Toggle word wrap

   In this example, there is a subdirectory called config/ with a configuration file called test.yml. In the command line, exclude the extension of the configuration file, as the tool will automatically determine the file type and extension.

Procedure

1. Optional: Label a node:

   # oc label node perf-node.example.com cpumanager=true

   Copy to Clipboard Toggle word wrap

2. Edit the MachineConfigPool of the nodes where CPU Manager should be enabled. In this example, all workers have CPU Manager enabled:

   # oc edit machineconfigpool worker

   Copy to Clipboard Toggle word wrap

3. Add a label to the worker MachineConfigPool:

   metadata:
     creationTimestamp: 2019-xx-xxx
     generation: 3
     labels:
       custom-kubelet: cpumanager-enabled

   Copy to Clipboard Toggle word wrap

4. Create a KubeletConfig, cpumanager-kubeletconfig.yaml, custom resource (CR). Refer to the label created in the previous step to have the correct nodes updated with the new KubeletConfig. See the machineConfigPoolSelector section:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: cpumanager-enabled
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: cpumanager-enabled
     kubeletConfig:
        cpuManagerPolicy: static
        cpuManagerReconcilePeriod: 5s

   Copy to Clipboard Toggle word wrap

5. Create the dynamic KubeletConfig:

   # oc create -f cpumanager-kubeletconfig.yaml

   Copy to Clipboard Toggle word wrap

   This adds the CPU Manager feature to the KubeletConfig and, if needed, the Machine Config Operator (MCO) reboots the node. To enable CPU Manager, a reboot is not needed.

6. Check for the merged KubeletConfig:

   # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7
   
          "ownerReferences": [
               {
                   "apiVersion": "machineconfiguration.openshift.io/v1",
                   "kind": "KubeletConfig",
                   "name": "cpumanager-enabled",
                   "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878"
               }
           ],

   Copy to Clipboard Toggle word wrap

7. Check the worker for the updated kubelet.conf:

   # oc debug node/perf-node.example.com
   sh-4.4# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager
   cpuManagerPolicy: static        

   1

   
   cpuManagerReconcilePeriod: 5s   

   2

   Copy to Clipboard Toggle word wrap

   1 2

   These settings were defined when you created the KubeletConfig CR.

8. Create a Pod that requests a core or multiple cores. Both limits and requests must have their CPU value set to a whole integer. That is the number of cores that will be dedicated to this Pod:

   # cat cpumanager-pod.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     generateName: cpumanager-
   spec:
     containers:
     - name: cpumanager
       image: gcr.io/google_containers/pause-amd64:3.0
       resources:
         requests:
           cpu: 1
           memory: "1G"
         limits:
           cpu: 1
           memory: "1G"
     nodeSelector:
       cpumanager: "true"

   Copy to Clipboard Toggle word wrap

9. Create the Pod:

   # oc create -f cpumanager-pod.yaml

   Copy to Clipboard Toggle word wrap

10. Verify that the Pod is scheduled to the node that you labeled:

    # oc describe pod cpumanager
    Name:               cpumanager-6cqz7
    Namespace:          default
    Priority:           0
    PriorityClassName:  <none>
    Node:  perf-node.example.com/xxx.xx.xx.xxx
    ...
     Limits:
          cpu:     1
          memory:  1G
        Requests:
          cpu:        1
          memory:     1G
    ...
    QoS Class:       Guaranteed
    Node-Selectors:  cpumanager=true

    Copy to Clipboard Toggle word wrap

11. Verify that the cgroups are set up correctly. Get the process ID (PID) of the pause process:

    # ├─init.scope
    │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
    └─kubepods.slice
      ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice
      │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope
      │ └─32706 /pause

    Copy to Clipboard Toggle word wrap

    Pods of quality of service (QoS) tier Guaranteed are placed within the kubepods.slice. Pods of other QoS tiers end up in child cgroups of kubepods:

    # cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope
    # for i in `ls cpuset.cpus tasks` ; do echo -n "$i "; cat $i ; done
    cpuset.cpus 1
    tasks 32706

    Copy to Clipboard Toggle word wrap

12. Check the allowed CPU list for the task:

    # grep ^Cpus_allowed_list /proc/32706/status
     Cpus_allowed_list:    1

    Copy to Clipboard Toggle word wrap

13. Verify that another pod (in this case, the pod in the burstable QoS tier) on the system cannot run on the core allocated for the Guaranteed pod:

    # cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus
    
    0

    Copy to Clipboard Toggle word wrap

    # oc describe node perf-node.example.com
    ...
    Capacity:
     attachable-volumes-aws-ebs:  39
     cpu:                         2
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      8162900Ki
     pods:                        250
    Allocatable:
     attachable-volumes-aws-ebs:  39
     cpu:                         1500m
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      7548500Ki
     pods:                        250
    -------                               ----                           ------------  ----------  ---------------  -------------  ---
      default                                 cpumanager-6cqz7               1 (66%)       1 (66%)     1G (12%)         1G (12%)       29m
    
    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      Resource                    Requests          Limits
      --------                    --------          ------
      cpu                         1440m (96%)       1 (66%)

    Copy to Clipboard Toggle word wrap

    This VM has two CPU cores. You set kube-reserved to 500 millicores, meaning half of one core is subtracted from the total capacity of the node to arrive at the Node Allocatable amount. You can see that Allocatable CPU is 1500 millicores. This means you can run one of the CPU Manager pods since each will take one whole core. A whole core is equivalent to 1000 millicores. If you try to schedule a second pod, the system will accept the pod, but it will never be scheduled:

    NAME                    READY   STATUS    RESTARTS   AGE
    cpumanager-6cqz7        1/1     Running   0          33m
    cpumanager-7qc2t        0/1     Pending   0          11s

    Copy to Clipboard Toggle word wrap

1. Create a YAML configuration file, `cluster-monitoring-config.yml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusOperator:
         baseImage: quay.io/coreos/prometheus-operator
         prometheusConfigReloaderBaseImage: quay.io/coreos/prometheus-config-reloader
         configReloaderBaseImage: quay.io/coreos/configmap-reload
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} 

   1

   
         baseImage: openshift/prometheus
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: gp2
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} 

   2

   
       alertmanagerMain:
         baseImage: openshift/prometheus-alertmanager
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: gp2
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} 

   3

   
       nodeExporter:
         baseImage: openshift/prometheus-node-exporter
       kubeRbacProxy:
         baseImage: quay.io/coreos/kube-rbac-proxy
       kubeStateMetrics:
         baseImage: quay.io/coreos/kube-state-metrics
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       grafana:
         baseImage: grafana/grafana
         nodeSelector:
           node-role.kubernetes.io/infra: ""
       auth:
         baseImage: openshift/oauth-proxy
       k8sPrometheusAdapter:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
   metadata:
     name: cluster-monitoring-config
   namespace: openshift-monitoring

   Copy to Clipboard Toggle word wrap

   1

   A typical value is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.

   2

   A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

   3

   A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Set the values like the retention period and storage sizes.

3. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. Label the node so that the Node Tuning Operator knows on which node to apply the tuned profile, which describes how many huge pages should be allocated:

   $ oc label node <node_using_hugepages> hugepages=true

   Copy to Clipboard Toggle word wrap

2. Create a file with the following content and name it hugepages_tuning.yaml:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: hugepages 

   1

   
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile: 

   2

   
     - data: |
         [main]
         summary=Configuration for hugepages
         include=openshift-node
   
         [vm]
         transparent_hugepages=never
   
         [sysctl]
         vm.nr_hugepages=1024
       name: node-hugepages
     recommend:
     - match: 

   3

   
       - label: hugepages
       priority: 30
       profile: node-hugepages

   Copy to Clipboard Toggle word wrap

   1

   Set the name parameter value to hugepages.

   2

   Set the profile section to allocate huge pages.

   3

   Set the match section to associate the profile to nodes with the hugepages label.

3. Create the custom hugepages tuned profile by using the hugepages_tuning.yaml file:

   $ oc create -f hugepages_tuning.yaml

   Copy to Clipboard Toggle word wrap

4. After creating the profile, the Operator applies the new profile to the correct node and allocates huge pages. Check the logs of a tuned pod on a node using huge pages to verify:

   $ oc logs <tuned_pod_on_node_using_hugepages> \
       -n openshift-cluster-node-tuning-operator | grep 'applied$' | tail -n1
   
   2019-08-08 07:20:41,286 INFO     tuned.daemon.daemon: static tuning from profile 'node-hugepages' applied

   Copy to Clipboard Toggle word wrap