5.5. 配置更高的虚拟机工作负载密度

流程

1. 将 kubelet 服务配置为允许交换使用：

   1. 使用以下示例中显示的参数创建或编辑 KubeletConfig 文件：

      KubeletConfig 文件示例

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-config
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: ''  # MCP
            #machine.openshift.io/cluster-api-machine-role: worker # machine
            #node-role.kubernetes.io/worker: '' # node
        kubeletConfig:
          failSwapOn: false

      Copy to Clipboard Toggle word wrap

   2. 运行以下命令，等待 worker 节点与新配置同步：

      $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

      Copy to Clipboard Toggle word wrap

2. 通过创建 MachineConfig 对象来置备 swap。例如：

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 90-worker-swap
   spec:
     config:
       ignition:
         version: 3.5.0
       systemd:
         units:
           - contents: |
               [Unit]
               Description=Provision and enable swap
               ConditionFirstBoot=no
               ConditionPathExists=!/var/tmp/swapfile
   
               [Service]
               Type=oneshot
               Environment=SWAP_SIZE_MB=5000
               ExecStart=/bin/sh -c "sudo dd if=/dev/zero of=/var/tmp/swapfile count=${SWAP_SIZE_MB} bs=1M && \
               sudo chmod 600 /var/tmp/swapfile && \
               sudo mkswap /var/tmp/swapfile && \
               sudo swapon /var/tmp/swapfile && \
               free -h"
   
               [Install]
               RequiredBy=kubelet-dependencies.target
             enabled: true
             name: swap-provision.service
           - contents: |
               [Unit]
               Description=Restrict swap for system slice
               ConditionFirstBoot=no
   
               [Service]
               Type=oneshot
               ExecStart=/bin/sh -c "sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""
   
               [Install]
               RequiredBy=kubelet-dependencies.target
             enabled: true
             name: cgroup-system-slice-config.service

   Copy to Clipboard Toggle word wrap

   为了在出现最糟糕的情况时有足够的交换空间，请确保置备的交换空间的数量最少与过量使用 RAM 相同。使用以下公式计算节点上置备的 swap 空间量：

   NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

   Copy to Clipboard Toggle word wrap

   例如

   NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
                  = 16 GB * (1.5 - 1)
                  = 16 GB * (0.5)
                  =  8 GB

   Copy to Clipboard Toggle word wrap

3. 运行以下命令来创建特权服务帐户：

   $ oc adm new-project wasp

   Copy to Clipboard Toggle word wrap

   $ oc create sa -n wasp wasp

   Copy to Clipboard Toggle word wrap

   $ oc create clusterrolebinding wasp --clusterrole=cluster-admin --serviceaccount=wasp:wasp

   Copy to Clipboard Toggle word wrap

   $ oc adm policy add-scc-to-user -n wasp privileged -z wasp

   Copy to Clipboard Toggle word wrap

4. 运行以下命令，等待 worker 节点与新配置同步：

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

   Copy to Clipboard Toggle word wrap

5. 运行以下命令，确定 wasp 代理镜像的 pull URL：

   $ oc get csv -n openshift-cnv -l=operators.coreos.com/kubevirt-hyperconverged.openshift-cnv -ojson | jq '.items[0].spec.relatedImages[] | select(.name|test(".*wasp-agent.*")) | .image'

   Copy to Clipboard Toggle word wrap

6. 通过创建一个 DaemonSet 对象来部署 wasp-agent，如下例所示：

   kind: DaemonSet
   apiVersion: apps/v1
   metadata:
     name: wasp-agent
     namespace: wasp
     labels:
       app: wasp
       tier: node
   spec:
     selector:
       matchLabels:
         name: wasp
     template:
       metadata:
         annotations:
           description: >-
             Configures swap for workloads
         labels:
           name: wasp
       spec:
         containers:
           - env:
               - name: SWAP_UTILIZATION_THRESHOLD_FACTOR
                 value: "0.8"
               - name: MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND
                 value: "1000000000"
               - name: MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND
                 value: "1000000000"
               - name: AVERAGE_WINDOW_SIZE_SECONDS
                 value: "30"
               - name: VERBOSITY
                 value: "1"
               - name: FSROOT
                 value: /host
               - name: NODE_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
             image: >-
               quay.io/openshift-virtualization/wasp-agent:v4.19 

   1

   
             imagePullPolicy: Always
             name: wasp-agent
             resources:
               requests:
                 cpu: 100m
                 memory: 50M
             securityContext:
               privileged: true
             volumeMounts:
               - mountPath: /host
                 name: host
               - mountPath: /rootfs
                 name: rootfs
         hostPID: true
         hostUsers: true
         priorityClassName: system-node-critical
         serviceAccountName: wasp
         terminationGracePeriodSeconds: 5
         volumes:
           - hostPath:
               path: /
             name: host
           - hostPath:
               path: /
             name: rootfs
     updateStrategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 10%
         maxSurge: 0

   Copy to Clipboard Toggle word wrap

   1

   将 image 值替换为上一步中的镜像 URL

7. 通过创建 PrometheusRule 对象来部署警报规则。例如：

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       tier: node
       wasp.io: ""
     name: wasp-rules
     namespace: wasp
   spec:
     groups:
       - name: alerts.rules
         rules:
           - alert: NodeHighSwapActivity
             annotations:
               description: High swap activity detected at {{ $labels.instance }}. The rate
                 of swap out and swap in exceeds 200 in both operations in the last minute.
                 This could indicate memory pressure and may affect system performance.
               runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/docs/runbooks/NodeHighSwapActivity.md
               summary: High swap activity detected at {{ $labels.instance }}.
             expr: rate(node_vmstat_pswpout[1m]) > 200 and rate(node_vmstat_pswpin[1m]) >
               200
             for: 1m
             labels:
               kubernetes_operator_component: kubevirt
               kubernetes_operator_part_of: kubevirt
               operator_health_impact: warning
               severity: warning

   Copy to Clipboard Toggle word wrap

8. 运行以下命令，将 cluster-monitoring 标签添加到 wasp 命名空间中：

   $ oc label namespace wasp openshift.io/cluster-monitoring="true"

   Copy to Clipboard Toggle word wrap

9. 使用 Web 控制台或 CLI 在 OpenShift Virtualization 中启用内存过量使用。

   • Web 控制台

     1. 在 OpenShift Container Platform web 控制台中，进入 Virtualization Overview Settings General settings Memory density。
     2. 将 Enable memory density 设置为 on。

   • CLI

     • 配置 OpenShift Virtualization 以启用更高的内存密度并设置过量分配率：

       $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \
         { \
         "op": "replace", \
         "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \
         "value": 150 \
         } \
       ]'

       Copy to Clipboard Toggle word wrap

       成功输出

       hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

       Copy to Clipboard Toggle word wrap

验证

1. 要验证 wasp-agent 的部署，请运行以下命令：

   $ oc rollout status ds wasp-agent -n wasp

   Copy to Clipboard Toggle word wrap

   如果部署成功，则会显示以下信息：

   输出示例

   daemon set "wasp-agent" successfully rolled out

   Copy to Clipboard Toggle word wrap

2. 要验证是否正确置备了 swap，请完成以下步骤：

   1. 运行以下命令，查看 worker 节点列表：

      $ oc get nodes -l node-role.kubernetes.io/worker

      Copy to Clipboard Toggle word wrap

   2. 运行以下命令，从列表中选择节点并显示其内存用量：

      $ oc debug node/<selected_node> -- free -m 

      1

      Copy to Clipboard Toggle word wrap

      1

      将 <selected_node> 替换为节点名称。

      如果置备了交换，则 Swap: 行会显示大于零的数量。

      Expand

      表 5.1. 输出示例

      	total	使用的	free	shared	buff/cache	可用
      Mem:	31846	23155	1044	6014	14483	8690
      Swap:	8191	2337	5854

      Show more

3. 运行以下命令验证 OpenShift Virtualization 内存过量使用配置：

   $ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'

   Copy to Clipboard Toggle word wrap

   输出示例

   {"memoryOvercommitPercentage":150}

   Copy to Clipboard Toggle word wrap

   返回的值必须与之前配置的值匹配。

流程

1. 删除 wasp-agent DaemonSet：

   $ oc delete daemonset wasp-agent -n wasp

   Copy to Clipboard Toggle word wrap

2. 如果部署，删除警报规则：

   $ oc delete prometheusrule wasp-rules -n wasp

   Copy to Clipboard Toggle word wrap

3. 另外，如果没有其他资源，删除 is p 命名空间：

   $ oc delete namespace wasp

   Copy to Clipboard Toggle word wrap

4. 恢复内存过量使用配置：

   $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged \
     --type='json' \
     -p='[{"op": "remove", "path": "/spec/higherWorkloadDensity"}]'

   Copy to Clipboard Toggle word wrap

5. 删除置备交换内存的 MachineConfig ：

   $ oc delete machineconfig 90-worker-swap

   Copy to Clipboard Toggle word wrap

6. 删除关联的 KubeletConfig ：

   $ oc delete kubeletconfig custom-config

   Copy to Clipboard Toggle word wrap

7. 等待 worker 节点协调：

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

   Copy to Clipboard Toggle word wrap