Scalability and performance

1. Create a YAML configuration file, cluster-monitoring-config.yaml. For example:

   apiVersion: v1
   kind: ConfigMap
   data:
     config.yaml: |
       prometheusK8s:
         retention: {{PROMETHEUS_RETENTION_PERIOD}} 

   1

   
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 

   2

   
             resources:
               requests:
                 storage: {{PROMETHEUS_STORAGE_SIZE}} 

   3

   
       alertmanagerMain:
         nodeSelector:
           node-role.kubernetes.io/infra: ""
         volumeClaimTemplate:
           spec:
             storageClassName: {{STORAGE_CLASS}} 

   4

   
             resources:
               requests:
                 storage: {{ALERTMANAGER_STORAGE_SIZE}} 

   5

   
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring

   Copy to Clipboard Toggle word wrap

   1

   The default value of Prometheus retention is PROMETHEUS_RETENTION_PERIOD=15d. Units are measured in time using one of these suffixes: s, m, h, d.

   2 4

   The storage class for your cluster.

   3

   A typical value is PROMETHEUS_STORAGE_SIZE=2000Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

   5

   A typical value is ALERTMANAGER_STORAGE_SIZE=20Gi. Storage values can be a plain integer or a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.

2. Add values for the retention period, storage class, and storage sizes.
3. Save the file.

4. Apply the changes by running:

   $ oc create -f cluster-monitoring-config.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Attach the new disk to the cluster and verify that the disk is detected in the node by running the lsblk command in a debug shell:

   $ oc debug node/<node_name>

   Copy to Clipboard Toggle word wrap

   # lsblk

   Copy to Clipboard Toggle word wrap

   Note the device name of the new disk reported by the lsblk command.

2. Create the following script and name it etcd-find-secondary-device.sh:

   #!/bin/bash
   set -uo pipefail
   
   for device in <device_type_glob>; do 

   1

   
   /usr/sbin/blkid "${device}" &> /dev/null
    if [ $? == 2  ]; then
       echo "secondary device found ${device}"
       echo "creating filesystem for etcd mount"
       mkfs.xfs -L var-lib-etcd -f "${device}" &> /dev/null
       udevadm settle
       touch /etc/var-lib-etcd-mount
       exit
    fi
   done
   echo "Couldn't find secondary block device!" >&2
   exit 77

   Copy to Clipboard Toggle word wrap

   1

   Replace <device_type_glob> with a shell glob for your block device type. For SCSI or SATA drives, use /dev/sd*; for virtual drives, use /dev/vd*; for NVMe drives, use /dev/nvme*[0-9]*n*.

3. Create a base64-encoded string from the etcd-find-secondary-device.sh script and note its contents:

   $ base64 -w0 etcd-find-secondary-device.sh

   Copy to Clipboard Toggle word wrap

4. Create a MachineConfig YAML file named etcd-mc.yml with contents such as the following:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 98-var-lib-etcd
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/find-secondary-device
             mode: 0755
             contents:
               source: data:text/plain;charset=utf-8;base64,<encoded_etcd_find_secondary_device_script> 

   1

   
       systemd:
         units:
           - name: find-secondary-device.service
             enabled: true
             contents: |
               [Unit]
               Description=Find secondary device
               DefaultDependencies=false
               After=systemd-udev-settle.service
               Before=local-fs-pre.target
               ConditionPathExists=!/etc/var-lib-etcd-mount
   
               [Service]
               RemainAfterExit=yes
               ExecStart=/etc/find-secondary-device
   
               RestartForceExitStatus=77
   
               [Install]
               WantedBy=multi-user.target
           - name: var-lib-etcd.mount
             enabled: true
             contents: |
               [Unit]
               Before=local-fs.target
   
               [Mount]
               What=/dev/disk/by-label/var-lib-etcd
               Where=/var/lib/etcd
               Type=xfs
               TimeoutSec=120s
   
               [Install]
               RequiredBy=local-fs.target
           - name: sync-var-lib-etcd-to-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Sync etcd data if new mount is empty
               DefaultDependencies=no
               After=var-lib-etcd.mount var.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecCondition=/usr/bin/test ! -d /var/lib/etcd/member
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 1
               ExecStart=/bin/rsync -ar /sysroot/ostree/deploy/rhcos/var/lib/etcd/ /var/lib/etcd/
               ExecStart=/usr/sbin/semanage fcontext -a -t container_var_lib_t '/var/lib/etcd(/.*)?'
               ExecStart=/usr/sbin/setsebool -P rsync_full_access 0
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target
           - name: restorecon-var-lib-etcd.service
             enabled: true
             contents: |
               [Unit]
               Description=Restore recursive SELinux security contexts
               DefaultDependencies=no
               After=var-lib-etcd.mount
               Before=crio.service
   
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecStart=/sbin/restorecon -R /var/lib/etcd/
               TimeoutSec=0
   
               [Install]
               WantedBy=multi-user.target graphical.target

   Copy to Clipboard Toggle word wrap

   1

   Replace <encoded_etcd_find_secondary_device_script> with the encoded script contents that you noted.

Procedure

1. To determine how many GPUs are available on a node in your cluster, use the following command:

   $ oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'

   Copy to Clipboard Toggle word wrap

   Example output

                       openshift.com/gpu-accelerator=true
   Capacity:
    nvidia.com/gpu:  2
   Allocatable:
    nvidia.com/gpu:  2
    nvidia.com/gpu:  0           0

   Copy to Clipboard Toggle word wrap

   In this example, 2 GPUs are available.

2. Use this command to set a quota in the namespace nvidia. In this example, the quota is 1:

   $ cat gpu-quota.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: gpu-quota
     namespace: nvidia
   spec:
     hard:
       requests.nvidia.com/gpu: 1

   Copy to Clipboard Toggle word wrap

3. Create the quota with the following command:

   $ oc create -f gpu-quota.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   resourcequota/gpu-quota created

   Copy to Clipboard Toggle word wrap

4. Verify that the namespace has the correct quota set using the following command:

   $ oc describe quota gpu-quota -n nvidia

   Copy to Clipboard Toggle word wrap

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  0     1

   Copy to Clipboard Toggle word wrap

5. Run a pod that asks for a single GPU with the following command:

   $ oc create pod gpu-pod.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: gpu-pod-s46h7
     namespace: nvidia
   spec:
     restartPolicy: OnFailure
     containers:
     - name: rhel7-gpu-pod
       image: rhel7
       env:
         - name: NVIDIA_VISIBLE_DEVICES
           value: all
         - name: NVIDIA_DRIVER_CAPABILITIES
           value: "compute,utility"
         - name: NVIDIA_REQUIRE_CUDA
           value: "cuda>=5.0"
   
       command: ["sleep"]
       args: ["infinity"]
   
       resources:
         limits:
           nvidia.com/gpu: 1

   Copy to Clipboard Toggle word wrap

6. Verify that the pod is running bwith the following command:

   $ oc get pods

   Copy to Clipboard Toggle word wrap

   Example output

   NAME              READY     STATUS      RESTARTS   AGE
   gpu-pod-s46h7     1/1       Running     0          1m

   Copy to Clipboard Toggle word wrap

7. Verify that the quota Used counter is correct by running the following command:

   $ oc describe quota gpu-quota -n nvidia

   Copy to Clipboard Toggle word wrap

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  1     1

   Copy to Clipboard Toggle word wrap

8. Using the following command, attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:

   $ oc create -f gpu-pod.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1

   Copy to Clipboard Toggle word wrap

   This Forbidden error message occurs because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.

1. First, get the list of quotas defined in the project. For example, for a project called demoproject:

   $ oc get quota -n demoproject
   NAME                AGE
   besteffort          11m
   compute-resources   2m
   core-object-counts  29m

   Copy to Clipboard Toggle word wrap

2. Describe the quota you are interested in, for example the core-object-counts quota:

   $ oc describe quota core-object-counts -n demoproject
   Name:			core-object-counts
   Namespace:		demoproject
   Resource		Used	Hard
   --------		----	----
   configmaps		3	10
   persistentvolumeclaims	0	4
   replicationcontrollers	3	20
   secrets			9	10
   services		2	10

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the object:

   $ oc create -f <limit_range_file> -n <project>

   Copy to Clipboard Toggle word wrap

Procedure

1. Get the list of limit range objects that are defined in the project. For example, a project called demoproject:

   $ oc get limits -n demoproject

   Copy to Clipboard Toggle word wrap

   Example Output

   NAME              AGE
   resource-limits   6d

   Copy to Clipboard Toggle word wrap

2. Describe the limit range. For example, for a limit range called resource-limits:

   $ oc describe limits resource-limits -n demoproject

   Copy to Clipboard Toggle word wrap

   Example Output

   Name:                           resource-limits
   Namespace:                      demoproject
   Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
   ----                            --------                ---     ---     --------------- -------------   -----------------------
   Pod                             cpu                     200m    2       -               -               -
   Pod                             memory                  6Mi     1Gi     -               -               -
   Container                       cpu                     100m    2       200m            300m            10
   Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
   openshift.io/Image              storage                 -       1Gi     -               -               -
   openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
   openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the following MCO sample profile into a YAML file. For example, enable-rfs.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 50-enable-rfs
   spec:
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all%20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D%3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20-c%20%27for%20x%20in%20/sys/%24DEVPATH/queues/rx-%2A%3B%20do%20echo%208192%20%3E%20%24x/rps_flow_cnt%3B%20%20done%27%22%0A
           filesystem: root
           mode: 0644
           path: /etc/udev/rules.d/70-persistent-net.rules
         - contents:
             source: data:text/plain;charset=US-ASCII,%23%20define%20sock%20flow%20enbtried%20for%20%20Receive%20Flow%20Steering%20%28RFS%29%0Anet.core.rps_sock_flow_entries%3D8192%0A
           filesystem: root
           mode: 0644
           path: /etc/sysctl.d/95-enable-rps.conf

   Copy to Clipboard Toggle word wrap

2. Create the MCO profile:

   $ oc create -f enable-rfs.yaml

   Copy to Clipboard Toggle word wrap

3. Verify that an entry named 50-enable-rfs is listed:

   $ oc get mc

   Copy to Clipboard Toggle word wrap

4. To deactivate, enter:

   $ oc delete mc 50-enable-rfs

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the following MCO sample profile into a YAML file for the control plane node. For example, 05-master-kernelarg-hpav.yaml:

   $ cat 05-master-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 05-master-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

   Copy to Clipboard Toggle word wrap

2. Copy the following MCO sample profile into a YAML file for the compute node. For example, 05-worker-kernelarg-hpav.yaml:

   $ cat 05-worker-kernelarg-hpav.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 05-worker-kernelarg-hpav
   spec:
     config:
       ignition:
         version: 3.1.0
     kernelArguments:
       - rd.dasd=800-805

   Copy to Clipboard Toggle word wrap
   Note

   You must modify the rd.dasd arguments to fit the device IDs.

3. Create the MCO profiles:

   $ oc create -f 05-master-kernelarg-hpav.yaml

   Copy to Clipboard Toggle word wrap

   $ oc create -f 05-worker-kernelarg-hpav.yaml

   Copy to Clipboard Toggle word wrap

4. To deactivate, enter:

   $ oc delete -f 05-master-kernelarg-hpav.yaml

   Copy to Clipboard Toggle word wrap

   $ oc delete -f 05-worker-kernelarg-hpav.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Open /etc/libvirt/qemu.conf with an editor of your choice.
2. Go to the cgroup_controllers line.
3. Duplicate the entire line and remove the leading number sign (#) from the copy.

4. Remove the cpuset entry, as follows:

   cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ]

   Copy to Clipboard Toggle word wrap

5. For the new setting to take effect, you must restart the libvirtd daemon:

   1. Stop all virtual machines.

   2. Run the following command:

      # systemctl restart libvirtd

      Copy to Clipboard Toggle word wrap
   3. Restart the virtual machines.