5.5. Configuring higher VM workload density

Procedure

1. Configure the kubelet service to permit swap usage:

   1. Create or edit a KubeletConfig file with the parameters shown in the following example:

      Example of a KubeletConfig file

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-config
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: ''  # MCP
            #machine.openshift.io/cluster-api-machine-role: worker # machine
            #node-role.kubernetes.io/worker: '' # node
        kubeletConfig:
          failSwapOn: false

   2. Wait for the worker nodes to sync with the new configuration by running the following command:

      $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

2. Provision swap by creating a MachineConfig object:

   1. Create a MachineConfig file with the parameters shown in the following example:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: 90-worker-swap
      spec:
        config:
          ignition:
            version: 3.5.0
          storage:
            files:
            - contents:
                source: data:text/plain;charset=utf-8;base64,YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24KZmFpbFN3YXBPbjogZmFsc2UK
              mode: 420
              overwrite: true
              path: /etc/openshift/kubelet.conf.d/90-swap.conf
          systemd:
            units:
              - contents: |
                  [Unit]
                  Description=Enable swap
                  ConditionFirstBoot=no
                  ConditionPathExists=/var/tmp/swapfile
      
                  [Service]
                  Type=oneshot
                  ExecStart=/bin/sh -c "sudo swapon /var/tmp/swapfile"
      
                  [Install]
                  RequiredBy=kubelet-dependencies.target
                enabled: true
                name: swap-enable.service
              - contents: |
                  [Unit]
                  Description=Provision and enable swap
                  ConditionFirstBoot=no
                  ConditionPathExists=!/var/tmp/swapfile
      
                  [Service]
                  Type=oneshot
                  Environment=SWAP_SIZE_MB=5000
                  ExecStart=/bin/sh -c "sudo fallocate -l ${SWAP_SIZE_MB}M /var/tmp/swapfile && \
                  sudo chmod 600 /var/tmp/swapfile && \
                  sudo mkswap /var/tmp/swapfile && \
                  sudo swapon /var/tmp/swapfile && \
                  free -h"
      
                  [Install]
                  RequiredBy=kubelet-dependencies.target
                enabled: true
                name: swap-provision.service
              - contents: |
                  [Unit]
                  Description=Restrict swap for system slice
                  ConditionFirstBoot=no
      
                  [Service]
                  Type=oneshot
                  ExecStart=/bin/sh -c "sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""
      
                  [Install]
                  RequiredBy=kubelet-dependencies.target
                enabled: true
                name: cgroup-system-slice-config.service

      To have enough swap space for the worst-case scenario, make sure to have at least as much swap space provisioned as overcommitted RAM. Calculate the amount of swap space to be provisioned on a node by using the following formula:

      NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

      Example

      NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
                     = 16 GB * (1.5 - 1)
                     = 16 GB * (0.5)
                     =  8 GB

3. Create a privileged service account by running the following commands:

   $ oc adm new-project wasp

   $ oc create sa -n wasp wasp

   $ oc create clusterrolebinding wasp --clusterrole=cluster-admin --serviceaccount=wasp:wasp

   $ oc adm policy add-scc-to-user -n wasp privileged -z wasp

4. Wait for the worker nodes to sync with the new configuration by running the following command:

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

5. Determine the pull URL for the wasp agent image by running the following command:

   $ oc get csv -n openshift-cnv -l=operators.coreos.com/kubevirt-hyperconverged.openshift-cnv -ojson | jq '.items[0].spec.relatedImages[] | select(.name|test(".*wasp-agent.*")) | .image'

6. Deploy wasp-agent by creating a DaemonSet object as shown in the following example:

   kind: DaemonSet
   apiVersion: apps/v1
   metadata:
     name: wasp-agent
     namespace: wasp
     labels:
       app: wasp
       tier: node
   spec:
     selector:
       matchLabels:
         name: wasp
     template:
       metadata:
         annotations:
           description: >-
             Configures swap for workloads
         labels:
           name: wasp
       spec:
         containers:
           - env:
               - name: SWAP_UTILIZATION_THRESHOLD_FACTOR
                 value: "0.8"
               - name: MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND
                 value: "1000000000"
               - name: MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND
                 value: "1000000000"
               - name: AVERAGE_WINDOW_SIZE_SECONDS
                 value: "30"
               - name: VERBOSITY
                 value: "1"
               - name: FSROOT
                 value: /host
               - name: NODE_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
             image: >-
               quay.io/openshift-virtualization/wasp-agent:v4.19 

   1

   
             imagePullPolicy: Always
             name: wasp-agent
             resources:
               requests:
                 cpu: 100m
                 memory: 50M
             securityContext:
               privileged: true
             volumeMounts:
               - mountPath: /host
                 name: host
               - mountPath: /rootfs
                 name: rootfs
         hostPID: true
         hostUsers: true
         priorityClassName: system-node-critical
         serviceAccountName: wasp
         terminationGracePeriodSeconds: 5
         volumes:
           - hostPath:
               path: /
             name: host
           - hostPath:
               path: /
             name: rootfs
     updateStrategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 10%
         maxSurge: 0

   1 1

   Replace the image value with the image URL from the previous step.

7. Deploy alerting rules by creating a PrometheusRule object. For example:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       tier: node
       wasp.io: ""
     name: wasp-rules
     namespace: wasp
   spec:
     groups:
       - name: alerts.rules
         rules:
           - alert: NodeHighSwapActivity
             annotations:
               description: High swap activity detected at {{ $labels.instance }}. The rate
                 of swap out and swap in exceeds 200 in both operations in the last minute.
                 This could indicate memory pressure and may affect system performance.
               runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/docs/runbooks/NodeHighSwapActivity.md
               summary: High swap activity detected at {{ $labels.instance }}.
             expr: rate(node_vmstat_pswpout[1m]) > 200 and rate(node_vmstat_pswpin[1m]) >
               200
             for: 1m
             labels:
               kubernetes_operator_component: kubevirt
               kubernetes_operator_part_of: kubevirt
               operator_health_impact: warning
               severity: warning

8. Add the cluster-monitoring label to the wasp namespace by running the following command:

   $ oc label namespace wasp openshift.io/cluster-monitoring="true"

9. Enable memory overcommitment in OpenShift Virtualization by using the web console or the CLI.

   • Web console

     1. In the OpenShift Container Platform web console, go to Virtualization Overview Settings General settings Memory density.
     2. Set Enable memory density to on.

   • CLI

     • Configure your OpenShift Virtualization to enable higher memory density and set the overcommit rate:

       $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \
         { \
         "op": "replace", \
         "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \
         "value": 150 \
         } \
       ]'

       Successful output

       hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

Verification

1. To verify the deployment of wasp-agent, run the following command:

   $ oc rollout status ds wasp-agent -n wasp

   If the deployment is successful, the following message is displayed:

   Example output

   daemon set "wasp-agent" successfully rolled out

2. To verify that swap is correctly provisioned, complete the following steps:

   1. View a list of worker nodes by running the following command:

      $ oc get nodes -l node-role.kubernetes.io/worker

   2. Select a node from the list and display its memory usage by running the following command:

      $ oc debug node/<selected_node> -- free -m

      Replace <selected_node> with the node name.

      If swap is provisioned, an amount greater than zero is displayed in the Swap: row.

      Expand

      표 5.1. Example output

      	total	used	free	shared	buff/cache	available
      Mem:	31846	23155	1044	6014	14483	8690
      Swap:	8191	2337	5854

      Show more

3. Verify the OpenShift Virtualization memory overcommitment configuration by running the following command:

   $ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'

   Example output

   {"memoryOvercommitPercentage":150}

   The returned value must match the value you had previously configured.