7.13.
7.13.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.1.1. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: with-limits spec: running: false template: spec: domain: # ... resources: requests: memory: 128Mi limits: memory: 256Mi1
7.13.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
- 참고
7.13.2.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.2.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
metadata:
name: example-vm-node-selector
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
template:
spec:
nodeSelector:
example-key-1: example-value-1
example-key-2: example-value-2
# ...
7.13.2.2.2. 링크 복사링크가 클립보드에 복사되었습니다!
metadata:
name: example-vm-pod-affinity
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
template:
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: example-key-1
operator: In
values:
- example-value-1
topologyKey: kubernetes.io/hostname
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: example-key-2
operator: In
values:
- example-value-2
topologyKey: kubernetes.io/hostname
# ...
7.13.2.2.3. 링크 복사링크가 클립보드에 복사되었습니다!
metadata:
name: example-vm-node-affinity
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: example.io/example-key
operator: In
values:
- example-value-1
- example-value-2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: example-node-label-key
operator: In
values:
- example-node-label-value
# ...
7.13.2.2.4. 링크 복사링크가 클립보드에 복사되었습니다!
metadata:
name: example-vm-tolerations
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
tolerations:
- key: "key"
operator: "Equal"
value: "virtualization"
effect: "NoSchedule"
# ...
7.13.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.2.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.3.4. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: {} # ...apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: ksmConfiguration: nodeLabelSelector: matchLabels: <first_example_key>: "true" <second_example_key>: "true" # ...apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: configuration: # ...
7.13.4. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.4.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: certConfig: ca: duration: 48h0m0s renewBefore: 24h0m0s1 server: duration: 24h0m0s2 renewBefore: 12h0m0s3
7.13.4.2. 링크 복사링크가 클립보드에 복사되었습니다!
certConfig:
ca:
duration: 4h0m0s
renewBefore: 1h0m0s
server:
duration: 4h0m0s
renewBefore: 4h0m0s
error: hyperconvergeds.hco.kubevirt.io "kubevirt-hyperconverged" could not be patched: admission webhook "validate-hco.kubevirt.io" denied the request: spec.certConfig: ca.duration is smaller than server.duration
7.13.5. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.5.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: defaultCPUModel: "EPYC"
7.13.6. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.6.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.6.2. 링크 복사링크가 클립보드에 복사되었습니다!
apiversion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: special: vm-secureboot name: vm-secureboot spec: template: metadata: labels: special: vm-secureboot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk features: acpi: {} smm: enabled: true1 firmware: bootloader: efi: secureBoot: true2 # ...$ oc create -f <file_name>.yaml
7.13.7. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.7.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.7.2. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: pxe-net-conf1 spec: config: | { "cniVersion": "0.3.1", "name": "pxe-net-conf",2 "type": "bridge",3 "bridge": "bridge-interface",4 "macspoofchk": false,5 "vlan": 100,6 "preserveDefaultVlan": false7 }
$ oc create -f pxe-net-conf.yamlinterfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1참고devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2networks: - name: default pod: {} - name: pxe-net multus: networkName: pxe-net-conf
$ oc create -f vmi-pxe-boot.yamlvirtualmachineinstance.kubevirt.io "vmi-pxe-boot" created$ oc get vmi vmi-pxe-boot -o yaml | grep -i phase phase: Running$ virtctl vnc vmi-pxe-boot$ virtctl console vmi-pxe-boot
$ ip addr... 3. eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether de:00:00:00:00:de brd ff:ff:ff:ff:ff:ff
7.13.7.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.8. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.8.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.8.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.8.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.9. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.9.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.9.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.9.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.10. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.10.1. 링크 복사링크가 클립보드에 복사되었습니다!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
7.13.10.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.10.3. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: myvm spec: template: spec: domain: cpu: model: Conroe1
7.13.10.4. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt/v1alpha3 kind: VirtualMachine metadata: name: myvm spec: template: spec: domain: cpu: model: host-model1
7.13.10.5. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: vm-fedora spec: running: true template: spec: schedulerName: my-scheduler1 domain: devices: disks: - name: containerdisk disk: bus: virtio # ...
$ oc get podsNAME READY STATUS RESTARTS AGE virt-launcher-vm-fedora-dpc87 2/2 Running 0 24m$ oc describe pod virt-launcher-vm-fedora-dpc87[...] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 21m my-scheduler Successfully assigned default/virt-launcher-vm-fedora-dpc87 to node01 [...]
7.13.11. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.11.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.11.1.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc label node <node_name> nvidia.com/gpu.deploy.operands=false1
$ oc describe node <node_name>$ oc get pods -n nvidia-gpu-operatorNAME READY STATUS RESTARTS AGE gpu-operator-59469b8c5c-hw9wj 1/1 Running 0 8d nvidia-sandbox-validator-7hx98 1/1 Running 0 8d nvidia-sandbox-validator-hdb7p 1/1 Running 0 8d nvidia-sandbox-validator-kxwj7 1/1 Terminating 0 9d nvidia-vfio-manager-7w9fs 1/1 Running 0 8d nvidia-vfio-manager-866pz 1/1 Running 0 8d nvidia-vfio-manager-zqtck 1/1 Terminating 0 9d$ oc get pods -n nvidia-gpu-operatorNAME READY STATUS RESTARTS AGE gpu-operator-59469b8c5c-hw9wj 1/1 Running 0 8d nvidia-sandbox-validator-7hx98 1/1 Running 0 8d nvidia-sandbox-validator-hdb7p 1/1 Running 0 8d nvidia-vfio-manager-7w9fs 1/1 Running 0 8d nvidia-vfio-manager-866pz 1/1 Running 0 8d
7.13.11.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.11.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.11.2.2. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker1 name: 100-worker-iommu2 spec: config: ignition: version: 3.2.0 kernelArguments: - intel_iommu=on3 # ...$ oc create -f 100-worker-kernel-arg-iommu.yaml
$ oc get MachineConfig
7.13.11.2.3. 링크 복사링크가 클립보드에 복사되었습니다!
$ lspci -nnv | grep -i nvidia02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)- 참고
variant: openshift version: 4.15.0 metadata: name: 100-worker-vfiopci labels: machineconfiguration.openshift.io/role: worker1 storage: files: - path: /etc/modprobe.d/vfio.conf mode: 0644 overwrite: true contents: inline: | options vfio-pci ids=10de:1eb82 - path: /etc/modules-load.d/vfio-pci.conf3 mode: 0644 overwrite: true contents: inline: vfio-pci $ butane 100-worker-vfiopci.bu -o 100-worker-vfiopci.yaml$ oc apply -f 100-worker-vfiopci.yaml$ oc get MachineConfigNAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 00-worker d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 01-master-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 01-master-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 01-worker-container-runtime d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 01-worker-kubelet d3da910bfa9f4b599af4ed7f5ac270d55950a3a1 3.2.0 25h 100-worker-iommu 3.2.0 30s 100-worker-vfiopci-configuration 3.2.0 30s
$ lspci -nnk -d 10de:04:00.0 3D controller [0302]: NVIDIA Corporation GP102GL [Tesla P40] [10de:1eb8] (rev a1) Subsystem: NVIDIA Corporation Device [10de:1eb8] Kernel driver in use: vfio-pci Kernel modules: nouveau
7.13.11.2.4. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: permittedHostDevices:1 pciHostDevices:2 - pciDeviceSelector: "10DE:1DB6"3 resourceName: "nvidia.com/GV100GL_Tesla_V100"4 - pciDeviceSelector: "10DE:1EB8" resourceName: "nvidia.com/TU104GL_Tesla_T4" - pciDeviceSelector: "8086:6F54" resourceName: "intel.com/qat" externalResourceProvider: true5 # ...참고
$ oc describe node <node_name>Capacity: cpu: 64 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 915128Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 131395264Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 1 pods: 250 Allocatable: cpu: 63500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 863623130526 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 130244288Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 1 pods: 250
7.13.11.2.5. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: permittedHostDevices: pciHostDevices: - pciDeviceSelector: "10DE:1DB6" resourceName: "nvidia.com/GV100GL_Tesla_V100" - pciDeviceSelector: "10DE:1EB8" resourceName: "nvidia.com/TU104GL_Tesla_T4" # ...
$ oc describe node <node_name>Capacity: cpu: 64 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 915128Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 131395264Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 0 pods: 250 Allocatable: cpu: 63500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 863623130526 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 130244288Ki nvidia.com/GV100GL_Tesla_V100 1 nvidia.com/TU104GL_Tesla_T4 1 intel.com/qat: 0 pods: 250
7.13.11.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.11.3.1. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: domain: devices: hostDevices: - deviceName: nvidia.com/TU104GL_Tesla_T41 name: hostdevices1
$ lspci -nnk | grep NVIDIA$ 02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)
7.13.12. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker1 name: 100-worker-iommu2 spec: config: ignition: version: 3.2.0 kernelArguments: - intel_iommu=on3 # ...$ oc create -f 100-worker-kernel-arg-iommu.yaml
$ oc get MachineConfig
7.13.12.3. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.3.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.3.2. 링크 복사링크가 클립보드에 복사되었습니다!
- 중요
kind: ClusterPolicy apiVersion: nvidia.com/v1 metadata: name: gpu-cluster-policy spec: operator: defaultRuntime: crio use_ocp_driver_toolkit: true initContainer: {} sandboxWorkloads: enabled: true defaultWorkload: vm-vgpu driver: enabled: false1 dcgmExporter: {} dcgm: enabled: true daemonsets: {} devicePlugin: {} gfd: {} migManager: enabled: true nodeStatusExporter: enabled: true mig: strategy: single toolkit: enabled: true validator: plugin: env: - name: WITH_WORKLOAD value: "true" vgpuManager: enabled: true2 repository: <vgpu_container_registry>3 image: <vgpu_image_name> version: nvidia-vgpu-manager vgpuDeviceManager: enabled: false4 config: name: vgpu-devices-config default: default sandboxDevicePlugin: enabled: false5 vfioManager: enabled: false6
7.13.12.4. 링크 복사링크가 클립보드에 복사되었습니다!
# ... mediatedDevicesConfiguration: mediatedDeviceTypes: - nvidia-222 - nvidia-228 - nvidia-105 - nvidia-108 # ...nvidia-105 # ... nvidia-108 nvidia-217 nvidia-299 # ...# ... mediatedDevicesConfiguration: mediatedDeviceTypes: - nvidia-22 - nvidia-223 - nvidia-224 # ...
7.13.12.5. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.5.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv예 7.1.
apiVersion: hco.kubevirt.io/v1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: mediatedDevicesConfiguration: mediatedDeviceTypes: - nvidia-231 nodeMediatedDeviceTypes: - mediatedDeviceTypes: - nvidia-233 nodeSelector: kubernetes.io/hostname: node-11.redhat.com permittedHostDevices: mediatedDevices: - mdevNameSelector: GRID T4-2Q resourceName: nvidia.com/GRID_T4-2Q - mdevNameSelector: GRID T4-8Q resourceName: nvidia.com/GRID_T4-8Q # ...# ... spec: mediatedDevicesConfiguration: mediatedDeviceTypes:1 - <device_type> nodeMediatedDeviceTypes:2 - mediatedDeviceTypes:3 - <device_type> nodeSelector:4 <node_selector_key>: <node_selector_value> # ...중요$ oc get $NODE -o json \ | jq '.status.allocatable \ | with_entries(select(.key | startswith("nvidia.com/"))) \ | with_entries(select(.value != "0"))'
# ... permittedHostDevices: mediatedDevices: - mdevNameSelector: GRID T4-2Q1 resourceName: nvidia.com/GRID_T4-2Q2 # ...
$ oc describe node <node_name>
7.13.12.5.2. 링크 복사링크가 클립보드에 복사되었습니다!
- 참고
7.13.12.5.3. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: mediatedDevicesConfiguration: mediatedDeviceTypes:1 - nvidia-231 permittedHostDevices: mediatedDevices:2 - mdevNameSelector: GRID T4-2Q resourceName: nvidia.com/GRID_T4-2Q
7.13.12.6. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.12.6.1. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: domain: devices: gpus: - deviceName: nvidia.com/TU104GL_Tesla_T41 name: gpu12 - deviceName: nvidia.com/GRID_T4-2Q name: gpu2
$ lspci -nnk | grep <device_name>
7.13.12.6.2. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.13. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.13.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.13.2. 링크 복사링크가 클립보드에 복사되었습니다!
- 중요
7.13.13.3. 링크 복사링크가 클립보드에 복사되었습니다!
apiVersion: kubevirt.io/v1 kind: VirtualMachine spec: template: metadata: annotations: descheduler.alpha.kubernetes.io/evict: "true"apiVersion: operator.openshift.io/v1 kind: KubeDescheduler metadata: name: cluster namespace: openshift-kube-descheduler-operator spec: deschedulingIntervalSeconds: 3600 profiles: - DevPreviewLongLifecycle mode: Predictive1
7.13.14. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.15. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.15.1. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \ --type=json -p='[{"op": "add", "path": "/spec/tuningPolicy", \ "value": "highBurst"}]'
$ oc get kubevirt.kubevirt.io/kubevirt-kubevirt-hyperconverged \ -n openshift-cnv -o go-template --template='{{range $config, \ $value := .spec.configuration}} {{if eq $config "apiConfiguration" \ "webhookConfiguration" "controllerConfiguration" "handlerConfiguration"}} \ {{"\n"}} {{$config}} = {{$value}} {{end}} {{end}} {{"\n"}}
7.13.16. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.16.1. 링크 복사링크가 클립보드에 복사되었습니다!
7.13.16.2. 링크 복사링크가 클립보드에 복사되었습니다!
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv... spec: resourceRequirements: vmiCPUAllocationRatio: 11 # ...