5.5. 配置更高的虚拟机工作负载密度

流程

1. 将 kubelet 服务配置为允许交换使用：

   1. 使用以下示例中显示的参数创建或编辑 KubeletConfig 文件：

      KubeletConfig 文件示例

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-config
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: ''  # MCP
            #machine.openshift.io/cluster-api-machine-role: worker # machine
            #node-role.kubernetes.io/worker: '' # node
        kubeletConfig:
          failSwapOn: false

   2. 运行以下命令，等待 worker 节点与新配置同步：

      $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

2. 通过创建 MachineConfig 对象来置备 swap。例如：

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 90-worker-swap
   spec:
     config:
       ignition:
         version: 3.4.0
       systemd:
         units:
           - contents: |
               [Unit]
               Description=Provision and enable swap
               ConditionFirstBoot=no
   
               [Service]
               Type=oneshot
               Environment=SWAP_SIZE_MB=5000
               ExecStart=/bin/sh -c "sudo dd if=/dev/zero of=/var/tmp/swapfile count=${SWAP_SIZE_MB} bs=1M && \
               sudo chmod 600 /var/tmp/swapfile && \
               sudo mkswap /var/tmp/swapfile && \
               sudo swapon /var/tmp/swapfile && \
               free -h && \
               sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""
   
               [Install]
               RequiredBy=kubelet-dependencies.target
             enabled: true
             name: swap-provision.service

   为了在出现最糟糕的情况时有足够的交换空间，请确保置备的交换空间的数量最少与过量使用 RAM 相同。使用以下公式计算节点上置备的 swap 空间量：

   NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

   Example

   NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
                  = 16 GB * (1.5 - 1)
                  = 16 GB * (0.5)
                  =  8 GB

3. 运行以下命令来创建特权服务帐户：

   $ oc adm new-project wasp

   $ oc create sa -n wasp wasp

   $ oc create clusterrolebinding wasp --clusterrole=cluster-admin --serviceaccount=wasp:wasp

   $ oc adm policy add-scc-to-user -n wasp privileged -z wasp

4. 运行以下命令，等待 worker 节点与新配置同步：

   $ oc wait mcp worker --for condition=Updated=True --timeout=-1s

5. 运行以下命令，确定 wasp 代理镜像的 pull URL：

   $ oc get csv -n openshift-cnv -l=operators.coreos.com/kubevirt-hyperconverged.openshift-cnv -ojson | jq '.items[0].spec.relatedImages[] | select(.name|test(".*wasp-agent.*")) | .image'

6. 通过创建一个 DaemonSet 对象来部署 wasp-agent，如下例所示：

   kind: DaemonSet
   apiVersion: apps/v1
   metadata:
     name: wasp-agent
     namespace: wasp
     labels:
       app: wasp
       tier: node
   spec:
     selector:
       matchLabels:
         name: wasp
     template:
       metadata:
         annotations:
           description: >-
             Configures swap for workloads
         labels:
           name: wasp
       spec:
         containers:
           - env:
               - name: SWAP_UTILIZATION_THRESHOLD_FACTOR
                 value: "0.8"
               - name: MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND
                 value: "1000"
               - name: MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND
                 value: "1000"
               - name: AVERAGE_WINDOW_SIZE_SECONDS
                 value: "30"
               - name: VERBOSITY
                 value: "1"
               - name: FSROOT
                 value: /host
               - name: NODE_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
             image: >-
               quay.io/openshift-virtualization/wasp-agent:v4.17 1
             imagePullPolicy: Always
             name: wasp-agent
             resources:
               requests:
                 cpu: 100m
                 memory: 50M
             securityContext:
               privileged: true
             volumeMounts:
               - mountPath: /host
                 name: host
               - mountPath: /rootfs
                 name: rootfs
         hostPID: true
         hostUsers: true
         priorityClassName: system-node-critical
         serviceAccountName: wasp
         terminationGracePeriodSeconds: 5
         volumes:
           - hostPath:
               path: /
             name: host
           - hostPath:
               path: /
             name: rootfs
     updateStrategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 10%
         maxSurge: 0

   1

   将 image 值替换为上一步中的镜像 URL

7. 通过创建 PrometheusRule 对象来部署警报规则。例如：

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       tier: node
       wasp.io: ""
     name: wasp-rules
     namespace: wasp
   spec:
     groups:
       - name: alerts.rules
         rules:
           - alert: NodeHighSwapActivity
             annotations:
               description: High swap activity detected at {{ $labels.instance }}. The rate
                 of swap out and swap in exceeds 200 in both operations in the last minute.
                 This could indicate memory pressure and may affect system performance.
               runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/docs/runbooks/NodeHighSwapActivity.md
               summary: High swap activity detected at {{ $labels.instance }}.
             expr: rate(node_vmstat_pswpout[1m]) > 200 and rate(node_vmstat_pswpin[1m]) >
               200
             for: 1m
             labels:
               kubernetes_operator_component: kubevirt
               kubernetes_operator_part_of: kubevirt
               operator_health_impact: warning
               severity: warning

8. 运行以下命令，将 cluster-monitoring 标签添加到 wasp 命名空间中：

   $ oc label namespace wasp openshift.io/cluster-monitoring="true"

9. 使用 Web 控制台或 CLI 在 OpenShift Virtualization 中启用内存过量使用。

   • Web 控制台

     1. 在 OpenShift Container Platform web 控制台中，进入 Virtualization Overview Settings General settings Memory density。
     2. 将 Enable memory density 设置为 on。

   • CLI

     • 配置 OpenShift Virtualization 以启用更高的内存密度并设置过量使用率：

       $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \
         { \
         "op": "replace", \
         "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \
         "value": 150 \
         } \
       ]'

       成功输出

       hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

验证

1. 要验证 wasp-agent 的部署，请运行以下命令：

   $ oc rollout status ds wasp-agent -n wasp

   如果部署成功，则会显示以下信息：

   输出示例

   daemon set "wasp-agent" successfully rolled out

2. 要验证是否正确置备了 swap，请完成以下步骤：

   1. 运行以下命令，查看 worker 节点列表：

      $ oc get nodes -l node-role.kubernetes.io/worker

   2. 运行以下命令，从列表中选择节点并显示其内存用量：

      $ oc debug node/<selected_node> -- free -m 1

      1

      将 <selected_node> 替换为节点名称。

      如果置备了交换，则 Swap: 行会显示大于零的数量。

      表 5.1. 输出示例

      	total	使用的	free	shared	buff/cache	可用
      Mem:	31846	23155	1044	6014	14483	8690
      Swap:	8191	2337	5854

3. 运行以下命令验证 OpenShift Virtualization 内存过量使用配置：

   $ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'

   输出示例

   {"memoryOvercommitPercentage":150}

   返回的值必须与之前配置的值匹配。