Red Hat Summit Red Hat Summit지원콘솔개발자평가판 시작연락처

가상화

OpenShift Container Platform 4.17

OpenShift Virtualization 설치, 사용법, 릴리스 정보

Red Hat OpenShift Documentation Team

초록

이 문서에서는 OpenShift Container Platform에서 OpenShift Virtualization을 사용하는 방법에 대한 정보를 제공합니다.

1장. 정보
링크 복사

1.1. OpenShift Virtualization 정보
링크 복사

OpenShift Virtualization의 기능 및 지원 범위에 대해 알아보십시오.

1.1.1. OpenShift Virtualization으로 수행할 수 있는 작업
링크 복사

OpenShift Virtualization은 Red Hat OpenShift에서 확장 가능한 엔터프라이즈급 가상화 기능을 제공합니다. 컨테이너 워크로드와 함께 또는 컨테이너 워크로드와 함께 VM(가상 머신)을 관리하는 데 사용할 수 있습니다.

참고

Red Hat OpenShift Virtualization Engine 서브스크립션이 있는 경우 서브스크립션 호스트에서 무제한 VM을 실행할 수 있지만 컨테이너에서 애플리케이션 인스턴스를 실행할 수 없습니다. 자세한 내용은 Red Hat OpenShift Virtualization Engine 및 관련 제품에 대한 서브스크립션 가이드 섹션을 참조하십시오.

OpenShift Virtualization은 Kubernetes 사용자 지정 리소스를 사용하여 가상화 작업을 활성화하여 OpenShift Container Platform 클러스터에 새 개체를 추가합니다. 다음과 같은 가상화 작업이 지원됩니다.

  • Linux 및 Windows VM 생성 및 관리
  • 클러스터에서 서로 함께 Pod 및 VM 워크로드 실행
  • 다양한 콘솔 및 CLI 툴을 통해 VM에 연결
  • 기존 VM 가져오기 및 복제
  • VM에 연결된 네트워크 인터페이스 컨트롤러 및 스토리지 디스크 관리
  • 노드 간 VM 실시간 마이그레이션

OpenShift Container Platform 웹 콘솔의 가상화 화면을 사용하고 OpenShift CLI(oc)를 사용하여 클러스터 및 가상화 리소스를 관리할 수 있습니다.

OpenShift Virtualization은 Red Hat OpenShift Data Foundation 기능과 원활하게 작동하도록 설계 및 테스트되었습니다.

중요

OpenShift Data Foundation을 사용하여 OpenShift Virtualization을 배포할 때 Windows 가상 머신 디스크용 전용 스토리지 클래스를 생성해야 합니다. 자세한 내용은 Windows VM용 ODF PersistentVolume 최적화 를 참조하십시오.

OVN-Kubernetes 또는 인증된 OpenShift CNI 플러그인에 나열된 다른 인증 네트워크 플러그인 중 하나와 함께 OpenShift Virtualization을 사용할 수 있습니다.

Compliance Operator 를 설치하고 ocp4-moderate 및 ocp4-moderate-node 프로필 을 사용하여 검사를 실행하여 OpenShift Virtualization 클러스터에서 규정 준수 문제를 확인할 수 있습니다. Compliance Operator는 NIST 인증 툴 인 OpenSCAP을 사용하여 보안 정책을 검사하고 적용합니다.

1.1.2. OpenShift Virtualization과 VMware vSphere 비교
링크 복사

VMware vSphere에 익숙한 경우 다음 표에는 유사한 작업을 수행하는 데 사용할 수 있는 OpenShift Virtualization 구성 요소가 나열되어 있습니다. 그러나 OpenShift Virtualization은 vSphere와 개념적으로 다르며 기본 OpenShift Container Platform과 많은 기능이 제공되기 때문에 OpenShift Virtualization에는 모든 vSphere 개념 또는 구성 요소에 대한 직접적인 대안이 없습니다.

Expand
표 1.1. vSphere 개념의 가장 가까운 OpenShift Virtualization에 매핑
vSphere 개념OpenShift Virtualization설명

데이터 저장소

영구 볼륨(PV) +
영구 볼륨 클레임(PVC)

VM 디스크를 저장합니다. PV는 기존 스토리지를 나타내며 PVC를 통해 VM에 연결됩니다. RWX( ReadWriteMany ) 액세스 모드로 생성되면 여러 VM에서 동시에 PVC를 마운트할 수 있습니다.

DRS(Dynamic Resource Scheduling)

Pod 제거 정책 +
Descheduler

활성 리소스 밸런싱을 제공합니다. Pod 제거 정책과 Descheduler의 조합을 사용하면 더 적절한 노드로 VM을 실시간 마이그레이션하여 노드 리소스 사용률을 관리할 수 있습니다.

NSX

Multus +
OVN-Kubernetes +
타사 컨테이너 네트워크 인터페이스(CNI) 플러그인

오버레이 네트워크 구성을 제공합니다. OpenShift Virtualization에서 NSX에 직접 해당하는 것은 없지만 OVN-Kubernetes 네트워크 공급자를 사용하거나 인증된 타사 CNI 플러그인을 설치할 수 있습니다.

스토리지 정책 기반 관리(SPBM)

스토리지 클래스

정책 기반 스토리지 선택을 제공합니다. 스토리지 클래스는 다양한 스토리지 유형을 나타내며 서비스 품질, 백업 정책, 회수 정책, 볼륨 확장이 허용되는지 여부와 같은 스토리지 기능을 설명합니다. PVC는 애플리케이션 요구 사항을 충족하기 위해 특정 스토리지 클래스를 요청할 수 있습니다.

vCenter
Cryostat 작업

OpenShift 지표 및 모니터링

호스트 및 VM 메트릭을 제공합니다. OpenShift Container Platform 웹 콘솔을 사용하여 클러스터 및 VM의 전반적인 상태를 보고 모니터링할 수 있습니다.

vMotion

실시간 마이그레이션

실행 중인 VM을 중단 없이 다른 노드로 이동합니다. 실시간 마이그레이션을 사용하려면 VM에 연결된 PVC에 RWX( ReadWriteMany ) 액세스 모드가 있어야 합니다.

vSwitch
DvSwitch

NMState Operator +
Multus

물리적 네트워크 구성을 제공합니다. NMState Operator를 사용하여 상태 중심 네트워크 구성을 적용하고 Linux 브리지 및 네트워크 본딩을 비롯한 다양한 네트워크 인터페이스 유형을 관리할 수 있습니다. Multus를 사용하면 여러 네트워크 인터페이스를 연결하고 VM을 외부 네트워크에 연결할 수 있습니다.

1.1.3. OpenShift Virtualization에서 지원되는 클러스터 버전
링크 복사

OpenShift Container Platform 4.17 클러스터에서 사용할 수 있도록 OpenShift Virtualization 4.17이 지원됩니다. OpenShift Virtualization의 최신 z-stream 릴리스를 사용하려면 먼저 OpenShift Container Platform의 최신 버전으로 업그레이드해야 합니다.

1.1.4. 가상 머신 디스크의 볼륨 및 액세스 모드 정보
링크 복사

알려진 스토리지 공급자와 스토리지 API를 사용하는 경우 볼륨 및 액세스 모드가 자동으로 선택됩니다. 그러나 스토리지 프로필이 없는 스토리지 클래스를 사용하는 경우 볼륨 및 액세스 모드를 구성해야 합니다.

최상의 결과를 얻으려면 RWX( ReadWriteMany ) 액세스 모드와 Block 볼륨 모드를 사용합니다. 이는 다음과 같은 이유로 중요합니다.

  • 실시간 마이그레이션에는 RWX( ReadWriteMany ) 액세스 모드가 필요합니다.

  • 블록 볼륨 모드는 Filesystem 볼륨 모드보다 훨씬 더 잘 작동합니다. 이는 Filesystem 볼륨 모드가 파일 시스템 계층 및 디스크 이미지 파일을 포함하여 더 많은 스토리지 계층을 사용하기 때문입니다. 이러한 계층은 VM 디스크 스토리지에 필요하지 않습니다.

    예를 들어 Red Hat OpenShift Data Foundation을 사용하는 경우 CephFS 볼륨에 Ceph RBD 볼륨을 사용하는 것이 좋습니다.

중요

다음 구성으로 가상 머신을 실시간 마이그레이션할 수 없습니다.

  • RWO( ReadWriteOnce ) 액세스 모드가 있는 스토리지 볼륨
  • GPU와 같은 패스스루 기능

이러한 가상 머신에 대해 evictionStrategy 필드를 None 으로 설정합니다. None 전략은 노드를 재부팅하는 동안 VM의 전원을 끕니다.

1.1.5. 단일 노드 OpenShift 차이점
링크 복사

단일 노드 OpenShift에 OpenShift Virtualization을 설치할 수 있습니다.

그러나 Single-node OpenShift는 다음 기능을 지원하지 않습니다.

  • 고가용성
  • Pod 중단
  • 실시간 마이그레이션
  • 제거 전략이 구성된 가상 머신 또는 템플릿

1.2. 지원되는 제한
링크 복사

OpenShift Virtualization에 대한 OpenShift Container Platform 환경을 계획할 때 테스트된 오브젝트 최대값을 참조할 수 있습니다. 그러나 최대값에 접근하면 성능을 줄이고 대기 시간을 높일 수 있습니다. 특정 사용 사례를 계획하고 클러스터 확장에 영향을 줄 수 있는 모든 요소를 고려해야 합니다.

성능에 영향을 미치는 클러스터 구성 및 옵션에 대한 자세한 내용은 Red Hat 지식 베이스의 OpenShift Virtualization - 튜닝 및 확장 가이드를 참조하십시오.

1.2.1. OpenShift Virtualization에서 테스트된 최대값
링크 복사

다음 제한은 대규모 OpenShift Virtualization 4.x 환경에 적용됩니다. 이는 가능한 가장 큰 크기의 단일 클러스터를 기반으로 합니다. 환경을 계획할 때는 여러 개의 작은 클러스터가 사용 사례에 가장 적합한 옵션이 될 수 있습니다.

1.2.1.1. 가상 머신 최대값
링크 복사

다음 최대값은 OpenShift Virtualization에서 실행되는 VM(가상 머신)에 적용됩니다. 이러한 값은 KVM을 사용하는 Red Hat Enterprise Linux의 가상화 제한에 지정된 제한의 적용을 받습니다.

Expand
목표(VM당)테스트된 제한이론적 제한

가상 CPU

참고

1.2.1.2.
링크 복사

Expand
   

1.2.1.3.
링크 복사

Expand
   

1.3.
링크 복사

1.3.1.
링크 복사

1.3.2.
링크 복사

1.3.3.
링크 복사

1.3.3.1.
링크 복사

Expand
표 1.2.
   

1.3.3.2.
링크 복사

1.3.3.2.1.
링크 복사
Expand
표 1.3.
   

Expand
표 1.4.
   

Expand
표 1.5.
   

1.3.3.2.2.
링크 복사
Expand
표 1.6.
   

Expand
표 1.7.
   

1.3.3.3.
링크 복사

$ oc get scc kubevirt-controller -o yaml
Copy to Clipboard Toggle word wrap 

$ oc get clusterrole kubevirt-controller -o yaml
Copy to Clipboard Toggle word wrap

1.4.
링크 복사

View larger image

[D]

1.4.1.
링크 복사

View larger image

[D]
Expand
표 1.8.
  

1.4.2.
링크 복사

View larger image

[D]
Expand
표 1.9.
  

1.4.3.
링크 복사

View larger image

[D]
Expand
표 1.10.
  

1.4.4.
링크 복사

View larger image

[D]
Expand
표 1.11.
  

1.4.5.
링크 복사

1.4.6.
링크 복사

View larger image

[D]
Expand
표 1.12.
  

2장.
링크 복사

2.1.
링크 복사

2.1.1.
링크 복사

2.1.2.
링크 복사

2.1.2.1.
링크 복사

2.1.2.2.
링크 복사

2.1.2.3.
링크 복사

2.1.3.
링크 복사

2.1.4.
링크 복사

2.1.4.1.
링크 복사
2.1.4.2.
링크 복사

  • 참고

2.1.4.3.
링크 복사
2.1.4.4.
링크 복사

2.1.5.
링크 복사

2.1.5.1.
링크 복사

2.1.5.2.
링크 복사

2.1.6.
링크 복사

  • 참고

2.1.7.
링크 복사

3장.
링크 복사

3.1.
링크 복사

참고

3.1.1.
링크 복사

3.1.2.
링크 복사

3.1.3.
링크 복사

  • 참고

3.1.4.
링크 복사

참고

3.1.5.
링크 복사

3.2.
링크 복사

3.2.1.
링크 복사

3.2.1.1.
링크 복사

      1. $ tar -xvf <virtctl-version-distribution.arch>.tar.gz
        Copy to Clipboard Toggle word wrap 

      2. $ chmod +x <path/virtctl-file-name>
        Copy to Clipboard Toggle word wrap 

      3. $ echo $PATH
        Copy to Clipboard Toggle word wrap 

      4. $ export KUBECONFIG=/home/<user>/clusters/current/auth/kubeconfig
        Copy to Clipboard Toggle word wrap 

      3. C:\> path
        Copy to Clipboard Toggle word wrap 

      2. echo $PATH
        Copy to Clipboard Toggle word wrap
3.2.1.2.
링크 복사

  1. # subscription-manager repos --enable cnv-4.17-for-rhel-8-x86_64-rpms
    Copy to Clipboard Toggle word wrap 

  2. # yum install kubevirt-virtctl
    Copy to Clipboard Toggle word wrap

3.2.2.
링크 복사

참고

3.2.2.1.
링크 복사

Expand
표 3.1.
  

3.2.2.2.
링크 복사

Expand
표 3.2.
  

3.2.2.3.
링크 복사

Expand
표 3.3.
  

3.2.2.4.
링크 복사

Expand
표 3.4.
  

3.2.2.5.
링크 복사

Expand
표 3.5.
  

3.2.2.6.
링크 복사

Expand
표 3.6.
  

3.2.2.7.
링크 복사

  • $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op": "add", "path": "/spec/featureGates", \
  "value": "HotplugVolumes"}]'
    Copy to Clipboard Toggle word wrap 

$ virtctl vmexport download <vmexport_name> --vm|pvc=<object_name> \
  --volume=<volume_name> --output=<output_file>
Copy to Clipboard Toggle word wrap
Expand
표 3.7.
  

3.2.2.8.
링크 복사

Expand
표 3.8.
  

3.2.2.9.
링크 복사

Expand
표 3.9.
  

3.2.3.
링크 복사

  • $ virtctl guestfs -n <namespace> <pvc_name>
    1
    Copy to Clipboard Toggle word wrap
    1
3.2.3.1.
링크 복사

Expand
  

Expand
  

참고

3.2.4.
링크 복사

4장.
링크 복사

4.1.
링크 복사

중요

4.1.1.
링크 복사

  • 중요

4.1.1.1.
링크 복사

참고

  • 중요

  • Expand
    표 4.1.
        
     

    		  

4.1.2.
링크 복사

4.1.2.1.
링크 복사

  • 참고

4.1.2.2.
링크 복사

  • 참고

4.1.2.3.
링크 복사
참고

4.1.2.3.1.
링크 복사

중요

4.1.3.
링크 복사

  • 참고

    Product of (Maximum number of nodes that can drain in parallel) and (Highest total VM memory request allocations across nodes)
    Copy to Clipboard Toggle word wrap

4.1.4.
링크 복사

중요

Memory overhead per infrastructure node ≈ 150 MiB
Copy to Clipboard Toggle word wrap 

Memory overhead per worker node ≈ 360 MiB
Copy to Clipboard Toggle word wrap 

Memory overhead per virtual machine ≈ (1.002 × requested memory) \
              + 218 MiB \
1 

              + 8 MiB × (number of vCPUs) \
2 

              + 16 MiB × (number of graphics devices) \
3 

              + (additional memory overhead)
4
Copy to Clipboard Toggle word wrap

1
2
3
4 

CPU overhead for infrastructure nodes ≈ 4 cores
Copy to Clipboard Toggle word wrap 

CPU overhead for worker nodes ≈ 2 cores + CPU overhead per virtual machine
Copy to Clipboard Toggle word wrap 

Aggregated storage overhead per node ≈ 10 GiB
Copy to Clipboard Toggle word wrap

4.1.5.
링크 복사

4.1.6.
링크 복사

4.1.7.
링크 복사

  • 참고

  • 참고

4.2.
링크 복사

중요

4.2.1.
링크 복사

4.2.1.1.
링크 복사

    2. 주의

    3. 주의

4.2.1.2.
링크 복사

4.2.1.2.1.
링크 복사

  1. apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.17.7
  channel: "stable"
    1
    Copy to Clipboard Toggle word wrap
    1 

  2. $ oc apply -f <file name>.yaml
    Copy to Clipboard Toggle word wrap
참고

4.2.1.2.2.
링크 복사

  1. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
    Copy to Clipboard Toggle word wrap 

  2. $ oc apply -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap 

  • $ watch oc get csv -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

    NAME                                      DISPLAY                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v4.17.7   OpenShift Virtualization   4.17.7                Succeeded
    Copy to Clipboard Toggle word wrap

4.2.2.
링크 복사

4.3.
링크 복사

4.3.1.
링크 복사

중요

4.3.1.1.
링크 복사

4.3.1.2.
링크 복사

  4. 참고

4.3.1.3.
링크 복사

4.3.1.4.
링크 복사

4.3.2.
링크 복사

  1. $ oc delete HyperConverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. $ oc delete subscription kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  3. $ oc delete csv -n openshift-cnv -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv
    Copy to Clipboard Toggle word wrap 

  4. $ oc delete namespace openshift-cnv
    Copy to Clipboard Toggle word wrap 

  5. $ oc delete crd --dry-run=client -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv
    Copy to Clipboard Toggle word wrap 

    customresourcedefinition.apiextensions.k8s.io "cdis.cdi.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "hostpathprovisioners.hostpathprovisioner.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "hyperconvergeds.hco.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "kubevirts.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "ssps.ssp.kubevirt.io" deleted (dry run)
customresourcedefinition.apiextensions.k8s.io "tektontasks.tektontasks.kubevirt.io" deleted (dry run)
    Copy to Clipboard Toggle word wrap 

  6. $ oc delete crd -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv
    Copy to Clipboard Toggle word wrap

5장.
링크 복사

5.1.
링크 복사

5.2.
링크 복사

참고

5.2.1.
링크 복사

5.2.2.
링크 복사

  1. $ oc edit <resource_type> <resource_name> -n openshift-cnv
    Copy to Clipboard Toggle word wrap

5.2.3.
링크 복사

5.2.3.1.
링크 복사

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.17.7
  channel: "stable"
  config:
    nodeSelector:
      example.io/example-infra-key: example-infra-value
1
Copy to Clipboard Toggle word wrap

1 

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source:  redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.17.7
  channel: "stable"
  config:
    tolerations:
    - key: "key"
      operator: "Equal"
      value: "virtualization"
1 

      effect: "NoSchedule"
Copy to Clipboard Toggle word wrap

1
5.2.3.2.
링크 복사

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      nodeSelector:
        example.io/example-infra-key: example-infra-value
1 

  workloads:
    nodePlacement:
      nodeSelector:
        example.io/example-workloads-key: example-workloads-value
2
Copy to Clipboard Toggle word wrap

1
2 

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-infra-key
                operator: In
                values:
                - example-infra-value
1 

  workloads:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-workloads-key
2 

                operator: In
                values:
                - example-workloads-value
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: example.io/num-cpus
                operator: Gt
                values:
                - 8
3
Copy to Clipboard Toggle word wrap

1
2
3 

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  workloads:
    nodePlacement:
      tolerations:
1 

      - key: "key"
        operator: "Equal"
        value: "virtualization"
        effect: "NoSchedule"
Copy to Clipboard Toggle word wrap

1
5.2.3.3.
링크 복사

주의

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  pathConfig:
    path: "</path/to/backing/directory>"
    useNamingPrefix: false
  workload:
    nodeSelector:
      example.io/example-workloads-key: example-workloads-value
1
Copy to Clipboard Toggle word wrap

1

5.3.
링크 복사

5.3.1.
링크 복사

5.3.2.
링크 복사

5.3.2.1.
링크 복사

  • apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br1-eth1-policy
    1 
    
spec:
  desiredState:
    interfaces:
      - name: br1
    2 
    
        description: Linux bridge with eth1 as a port
    3 
    
        type: linux-bridge
    4 
    
        state: up
    5 
    
        ipv4:
          enabled: false
    6 
    
        bridge:
          options:
            stp:
              enabled: false
    7 
    
          port:
            - name: eth1
    8
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8
5.3.2.2.
링크 복사

주의

  2. 참고

5.3.3.
링크 복사

5.3.3.1.
링크 복사

  1. apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: my-secondary-network
    1 
    
  namespace: openshift-cnv
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "migration-bridge",
    "type": "macvlan",
    "master": "eth1",
    2 
    
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
    3 
    
      "range": "10.200.5.0/24"
    4 
    
    }
  }'
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4 

  2. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  3. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  liveMigrationConfig:
    completionTimeoutPerGiB: 800
    network: <network>
    1 
    
    parallelMigrationsPerCluster: 5
    parallelOutboundMigrationsPerNode: 2
    progressTimeout: 150
# ...
    Copy to Clipboard Toggle word wrap

    1 

  • $ oc get vmi <vmi_name> -o jsonpath='{.status.migrationState.targetNodeAddress}'
    Copy to Clipboard Toggle word wrap
5.3.3.2.
링크 복사

5.3.4.
링크 복사

5.3.4.1.
링크 복사

참고

  1. apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: <name>
    1 
    
  namespace: openshift-sriov-network-operator
    2 
    
spec:
  resourceName: <sriov_resource_name>
    3 
    
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
    4 
    
  priority: <priority>
    5 
    
  mtu: <mtu>
    6 
    
  numVfs: <num>
    7 
    
  nicSelector:
    8 
    
    vendor: "<vendor_code>"
    9 
    
    deviceID: "<device_id>"
    10 
    
    pfNames: ["<pf_name>", ...]
    11 
    
    rootDevices: ["<pci_bus_id>", "..."]
    12 
    
  deviceType: vfio-pci
    13 
    
  isRdma: false
    14
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    참고

  3. $ oc create -f <name>-sriov-node-network.yaml
    Copy to Clipboard Toggle word wrap 

  4. $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'
    Copy to Clipboard Toggle word wrap

5.3.5.
링크 복사

5.4.
링크 복사

5.4.1.
링크 복사

중요

5.4.1.1.
링크 복사

참고

  1. apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-csi
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Delete
    1 
    
volumeBindingMode: WaitForFirstConsumer
    2 
    
parameters:
  storagePool: my-storage-pool
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  3. $ oc create -f storageclass_csi.yaml
    Copy to Clipboard Toggle word wrap

5.5.
링크 복사

참고

5.5.1.
링크 복사

중요

참고

    1. apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: custom-config
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ''  # MCP
      #machine.openshift.io/cluster-api-machine-role: worker # machine
      #node-role.kubernetes.io/worker: '' # node
  kubeletConfig:
    failSwapOn: false
      Copy to Clipboard Toggle word wrap 

    2. $ oc wait mcp worker --for condition=Updated=True --timeout=-1s
      Copy to Clipboard Toggle word wrap 

  2. apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 90-worker-swap
spec:
  config:
    ignition:
      version: 3.4.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Provision and enable swap
            ConditionFirstBoot=no
            ConditionPathExists=!/var/tmp/swapfile

            [Service]
            Type=oneshot
            Environment=SWAP_SIZE_MB=5000
            ExecStart=/bin/sh -c "sudo dd if=/dev/zero of=/var/tmp/swapfile count=${SWAP_SIZE_MB} bs=1M && \
            sudo chmod 600 /var/tmp/swapfile && \
            sudo mkswap /var/tmp/swapfile && \
            sudo swapon /var/tmp/swapfile && \
            free -h"

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: swap-provision.service
        - contents: |
            [Unit]
            Description=Restrict swap for system slice
            ConditionFirstBoot=no

            [Service]
            Type=oneshot
            ExecStart=/bin/sh -c "sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: cgroup-system-slice-config.service
    Copy to Clipboard Toggle word wrap 

    NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)
    Copy to Clipboard Toggle word wrap 

    NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
               = 16 GB * (1.5 - 1)
               = 16 GB * (0.5)
               =  8 GB
    Copy to Clipboard Toggle word wrap 

  3. $ oc adm new-project wasp
    Copy to Clipboard Toggle word wrap 
    $ oc create sa -n wasp wasp
    Copy to Clipboard Toggle word wrap 
    $ oc create clusterrolebinding wasp --clusterrole=cluster-admin --serviceaccount=wasp:wasp
    Copy to Clipboard Toggle word wrap 
    $ oc adm policy add-scc-to-user -n wasp privileged -z wasp
    Copy to Clipboard Toggle word wrap 

  4. $ oc wait mcp worker --for condition=Updated=True --timeout=-1s
    Copy to Clipboard Toggle word wrap 

  5. $ oc get csv -n openshift-cnv -l=operators.coreos.com/kubevirt-hyperconverged.openshift-cnv -ojson | jq '.items[0].spec.relatedImages[] | select(.name|test(".*wasp-agent.*")) | .image'
    Copy to Clipboard Toggle word wrap 

  6. kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: wasp-agent
  namespace: wasp
  labels:
    app: wasp
    tier: node
spec:
  selector:
    matchLabels:
      name: wasp
  template:
    metadata:
      annotations:
        description: >-
          Configures swap for workloads
      labels:
        name: wasp
    spec:
      containers:
        - env:
            - name: SWAP_UTILIZATION_THRESHOLD_FACTOR
              value: "0.8"
            - name: MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND
              value: "1000000000"
            - name: MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND
              value: "1000000000"
            - name: AVERAGE_WINDOW_SIZE_SECONDS
              value: "30"
            - name: VERBOSITY
              value: "1"
            - name: FSROOT
              value: /host
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          image: >-
            quay.io/openshift-virtualization/wasp-agent:v4.17
    1 
    
          imagePullPolicy: Always
          name: wasp-agent
          resources:
            requests:
              cpu: 100m
              memory: 50M
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /host
              name: host
            - mountPath: /rootfs
              name: rootfs
      hostPID: true
      hostUsers: true
      priorityClassName: system-node-critical
      serviceAccountName: wasp
      terminationGracePeriodSeconds: 5
      volumes:
        - hostPath:
            path: /
          name: host
        - hostPath:
            path: /
          name: rootfs
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
      maxSurge: 0
    Copy to Clipboard Toggle word wrap
    1 

  7. apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    tier: node
    wasp.io: ""
  name: wasp-rules
  namespace: wasp
spec:
  groups:
    - name: alerts.rules
      rules:
        - alert: NodeHighSwapActivity
          annotations:
            description: High swap activity detected at {{ $labels.instance }}. The rate
              of swap out and swap in exceeds 200 in both operations in the last minute.
              This could indicate memory pressure and may affect system performance.
            runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/docs/runbooks/NodeHighSwapActivity.md
            summary: High swap activity detected at {{ $labels.instance }}.
          expr: rate(node_vmstat_pswpout[1m]) > 200 and rate(node_vmstat_pswpin[1m]) >
            200
          for: 1m
          labels:
            kubernetes_operator_component: kubevirt
            kubernetes_operator_part_of: kubevirt
            operator_health_impact: warning
            severity: warning
    Copy to Clipboard Toggle word wrap 

  8. $ oc label namespace wasp openshift.io/cluster-monitoring="true"
    Copy to Clipboard Toggle word wrap 

      • $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \
  { \
  "op": "replace", \
  "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \
  "value": 150 \
  } \
]'
        Copy to Clipboard Toggle word wrap 

        hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched
        Copy to Clipboard Toggle word wrap 

  1. $ oc rollout status ds wasp-agent -n wasp
    Copy to Clipboard Toggle word wrap 

    daemon set "wasp-agent" successfully rolled out
    Copy to Clipboard Toggle word wrap 

    1. $ oc get nodes -l node-role.kubernetes.io/worker
      Copy to Clipboard Toggle word wrap 

    2. $ oc debug node/<selected_node> -- free -m
      1
      Copy to Clipboard Toggle word wrap
      1

      Expand
      표 5.1.
       

      		   

  3. $ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'
    Copy to Clipboard Toggle word wrap 

    {"memoryOvercommitPercentage":150}
    Copy to Clipboard Toggle word wrap

5.5.2.
링크 복사

averageSwapInPerSecond > maxAverageSwapInPagesPerSecond
&&
averageSwapOutPerSecond > maxAverageSwapOutPagesPerSecond
Copy to Clipboard Toggle word wrap 

nodeWorkingSet + nodeSwapUsage < totalNodeMemory + totalSwapMemory × thresholdFactor
Copy to Clipboard Toggle word wrap

5.5.2.1.
링크 복사

Expand

6장.
링크 복사

6.1.
링크 복사

6.1.1.
링크 복사

6.1.1.2.
링크 복사
중요

6.1.1.3.
링크 복사
6.1.1.4.
링크 복사

6.1.1.4.1.
링크 복사

중요

6.1.2.
링크 복사

참고

  1. $ oc get csv -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. VERSION  REPLACES                                        PHASE
4.9.0    kubevirt-hyperconverged-operator.v4.8.2         Installing
4.9.0    kubevirt-hyperconverged-operator.v4.9.0         Replacing
    Copy to Clipboard Toggle word wrap 

  3. $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'
    Copy to Clipboard Toggle word wrap 

    ReconcileComplete  True  Reconcile completed successfully
Available          True  Reconcile completed successfully
Progressing        False Reconcile completed successfully
Degraded           False Reconcile completed successfully
Upgradeable        True  Reconcile completed successfully
    Copy to Clipboard Toggle word wrap

6.1.3.
링크 복사

참고

참고

6.1.3.1.
링크 복사

  • 참고

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  workloadUpdateStrategy:
    workloadUpdateMethods:
    1 
    
    - LiveMigrate
    2 
    
    - Evict
    3 
    
    batchEvictionSize: 10
    4 
    
    batchEvictionInterval: "1m0s"
    5 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    참고

6.1.3.2.
링크 복사

참고

  • $ oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces
    Copy to Clipboard Toggle word wrap
참고

6.1.4.
링크 복사

6.1.4.1.
링크 복사

참고

6.1.4.2.
링크 복사

중요

$ oc get kv kubevirt-kubevirt-hyperconverged -o json -n openshift-cnv | jq .status.outdatedVirtualMachineInstanceWorkloads
Copy to Clipboard Toggle word wrap 

$ oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces
Copy to Clipboard Toggle word wrap 

  1. $ oc get kv kubevirt-kubevirt-hyperconverged \
  -n openshift-cnv -o jsonpath='{.spec.workloadUpdateStrategy.workloadUpdateMethods}'
    Copy to Clipboard Toggle word wrap 

  2. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op":"replace","path":"/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]'
    Copy to Clipboard Toggle word wrap 

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched
    Copy to Clipboard Toggle word wrap 

  3. $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
    Copy to Clipboard Toggle word wrap

    예 6.1. 

    [
  {
    "lastTransitionTime": "2022-12-09T16:29:11Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 3,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "ReconcileComplete"
  },
  {
    "lastTransitionTime": "2022-12-09T20:30:10Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 3,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2022-12-09T20:30:10Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 3,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2022-12-09T16:39:11Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 3,
    "reason": "ReconcileCompleted",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2022-12-09T20:30:10Z",
    "message": "Reconcile completed successfully",
    "observedGeneration": 3,
    "reason": "ReconcileCompleted",
    "status": "True",
    "type": "Upgradeable"
    1 
    
  }
]
    Copy to Clipboard Toggle word wrap
    1 

  4. $ oc adm upgrade
    Copy to Clipboard Toggle word wrap 

    • $ oc get clusterversion
      Copy to Clipboard Toggle word wrap
      참고

  6. $ oc get csv -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  8. $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
    Copy to Clipboard Toggle word wrap 

    [
  {
    "name": "operator",
    "version": "4.17.7"
  }
]
    Copy to Clipboard Toggle word wrap 

  9. $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
    Copy to Clipboard Toggle word wrap 

  11. $ oc get clusterversion
    Copy to Clipboard Toggle word wrap 

  13. $ oc get csv -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  14. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p \
  "[{\"op\":\"add\",\"path\":\"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\":{WorkloadUpdateMethodConfig}}]"
    Copy to Clipboard Toggle word wrap 

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched
    Copy to Clipboard Toggle word wrap 

    • $ oc get vmim -A
      Copy to Clipboard Toggle word wrap

6.1.5.
링크 복사

6.1.5.1.
링크 복사

6.1.5.2.
링크 복사

6.1.5.3.
링크 복사

6.1.6.
링크 복사

중요

7장.
링크 복사

7.1.
링크 복사

7.1.1.
링크 복사

중요

7.1.1.1.
링크 복사

7.1.1.1.1.
링크 복사

7.1.1.1.2.
링크 복사

7.1.1.2.
링크 복사

7.1.1.3.
링크 복사

7.1.1.3.1.
링크 복사

7.1.1.3.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  commonBootImageNamespace: <custom_namespace>
    1 
    
# ...
    Copy to Clipboard Toggle word wrap

    1

7.1.2.
링크 복사

7.1.2.1.
링크 복사

7.1.2.1.1.
링크 복사

참고

apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineInstancetype
metadata:
  name: example-instancetype
spec:
  cpu:
    guest: 1
1 

  memory:
    guest: 128Mi
2
Copy to Clipboard Toggle word wrap

1
2 

$ virtctl create instancetype --cpu 2 --memory 256Mi
Copy to Clipboard Toggle word wrap

작은 정보

$ virtctl create instancetype --cpu 2 --memory 256Mi | oc apply -f -
Copy to Clipboard Toggle word wrap
7.1.2.1.2.
링크 복사

7.1.2.2.
링크 복사

Expand
표 7.1.
     

7.1.2.3.
링크 복사

7.1.2.4.
링크 복사

    • 참고

7.1.3.
링크 복사

7.1.3.1.
링크 복사

참고

7.1.3.2.
링크 복사

7.1.3.2.1.
링크 복사
Expand
표 7.2.
  

참고

7.1.3.2.2.
링크 복사
Expand
  

Expand
   

참고
7.1.3.2.3.
링크 복사

7.1.3.2.4.
링크 복사

7.1.4.
링크 복사

참고

7.1.4.1.
링크 복사

  1. $ virtctl create vm --name rhel-9-minimal --volume-datasource src:openshift-virtualization-os-images/rhel9
    Copy to Clipboard Toggle word wrap

  2. 참고

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rhel-9-minimal
    1 
    
spec:
  dataVolumeTemplates:
  - metadata:
      name: imported-volume-mk4lj
    spec:
      sourceRef:
        kind: DataSource
        name: rhel9
    2 
    
        namespace: openshift-virtualization-os-images
    3 
    
      storage:
        resources: {}
  instancetype:
    inferFromVolume: imported-volume-mk4lj
    4 
    
    inferFromVolumeFailurePolicy: Ignore
  preference:
    inferFromVolume: imported-volume-mk4lj
    5 
    
    inferFromVolumeFailurePolicy: Ignore
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
        memory:
          guest: 512Mi
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - dataVolume:
          name: imported-volume-mk4lj
        name: imported-volume-mk4lj
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5 

  3. $ oc create -f <vm_manifest_file>.yaml
    Copy to Clipboard Toggle word wrap 

  4. $ virtctl start <vm_name>
    Copy to Clipboard Toggle word wrap

7.2.
링크 복사

7.2.1.
링크 복사

중요

7.2.2.
링크 복사

중요

중요

7.2.2.1.
링크 복사

참고

  1. $ cat > Dockerfile << EOF
FROM registry.access.redhat.com/ubi8/ubi:latest AS builder
ADD --chown=107:107 <vm_image>.qcow2 /disk/
    1 
    
RUN chmod 0440 /disk/*

FROM scratch
COPY --from=builder /disk/* /disk/
EOF
    Copy to Clipboard Toggle word wrap
    1 

  2. $ podman build -t <registry>/<container_disk_name>:latest .
    Copy to Clipboard Toggle word wrap 

  3. $ podman push <registry>/<container_disk_name>:latest
    Copy to Clipboard Toggle word wrap
7.2.2.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  storageImport:
    insecureRegistries:
    1 
    
      - "private-registry-example-1:5000"
      - "private-registry-example-2:5000"
    Copy to Clipboard Toggle word wrap

    1
7.2.2.3.
링크 복사

7.2.2.4.
링크 복사

  1. $ virtctl create vm --name vm-rhel-9 --instancetype u1.small --preference rhel.9 --volume-containerdisk src:registry.redhat.io/rhel9/rhel-guest-image:9.5
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-rhel-9
    1 
    
spec:
  instancetype:
    name: u1.small
    2 
    
  preference:
    name: rhel.9
    3 
    
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      domain:
        devices: {}
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - containerDisk:
          image: registry.redhat.io/rhel9/rhel-guest-image:9.5
    4 
    
        name: vm-rhel-9-containerdisk-0
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4 

  3. $ oc create -f <vm_manifest_file>.yaml
    Copy to Clipboard Toggle word wrap 

  1. $ oc get vm <vm_name>
    Copy to Clipboard Toggle word wrap 

    NAME        AGE   STATUS    READY
vm-rhel-9   18s   Running   True
    Copy to Clipboard Toggle word wrap 

  2. $ virtctl console <vm_name>
    Copy to Clipboard Toggle word wrap 

    Successfully connected to vm-rhel-9 console. The escape sequence is ^]
    Copy to Clipboard Toggle word wrap

7.2.3.
링크 복사

중요

7.2.3.1.
링크 복사

7.2.3.2.
링크 복사

  1. $ virtctl create vm --name vm-rhel-9 --instancetype u1.small --preference rhel.9 --volume-import type:http,url:https://example.com/rhel9.qcow2,size:10Gi
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-rhel-9
    1 
    
spec:
  dataVolumeTemplates:
  - metadata:
      name: imported-volume-6dcpf
    2 
    
    spec:
      source:
        http:
          url: https://example.com/rhel9.qcow2
    3 
    
      storage:
        resources:
          requests:
            storage: 10Gi
    4 
    
  instancetype:
    name: u1.small
    5 
    
  preference:
    name: rhel.9
    6 
    
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - dataVolume:
          name: imported-volume-6dcpf
        name: imported-volume-6dcpf
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6 

  3. $ oc create -f <vm_manifest_file>.yaml
    Copy to Clipboard Toggle word wrap 

  1. $ oc get pods
    Copy to Clipboard Toggle word wrap 

  2. $ oc get dv <data_volume_name>
    Copy to Clipboard Toggle word wrap 

    NAME                    PHASE       PROGRESS   RESTARTS   AGE
imported-volume-6dcpf   Succeeded   100.0%                18s
    Copy to Clipboard Toggle word wrap 

  3. $ virtctl console <vm_name>
    Copy to Clipboard Toggle word wrap 

    Successfully connected to vm-rhel-9 console. The escape sequence is ^]
    Copy to Clipboard Toggle word wrap

7.2.4.
링크 복사

중요

7.2.4.1.
링크 복사

7.2.4.1.1.
링크 복사

  1. $ virtctl stop <my_vm_name>
    Copy to Clipboard Toggle word wrap 

  3. $ oc get vm <my_vm_name> -o jsonpath="{.spec.template.spec.volumes}{'\n'}"
    Copy to Clipboard Toggle word wrap 

    [{"dataVolume":{"name":"<my_vm_volume>"},"name":"rootdisk"},{"cloudInitNoCloud":{...}]
    Copy to Clipboard Toggle word wrap 

  4. $ oc get pvc
    Copy to Clipboard Toggle word wrap 

    NAME            STATUS   VOLUME  CAPACITY   ACCESS MODES  STORAGECLASS     AGE
<my_vm_volume> Bound  …
    Copy to Clipboard Toggle word wrap

    참고

  5. $ virtctl guestfs <my-vm-volume> --uid 107
    Copy to Clipboard Toggle word wrap 

  6. $ virt-sysprep -a disk.img
    Copy to Clipboard Toggle word wrap

7.2.4.2.
링크 복사

7.2.4.2.1.
링크 복사

  7. %WINDIR%\System32\Sysprep\sysprep.exe /generalize /shutdown /oobe /mode:vm
    Copy to Clipboard Toggle word wrap

7.2.4.2.2.
링크 복사

7.2.4.3.
링크 복사

  1. $ virtctl image-upload dv <datavolume_name> \
    1 
    
  --size=<datavolume_size> \
    2 
    
  --image-path=</path/to/image> \
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    참고

  2. $ oc get dvs
    Copy to Clipboard Toggle word wrap

7.2.5.
링크 복사

7.2.5.1.
링크 복사
7.2.5.1.1.
링크 복사

  2. $ yum install -y qemu-guest-agent
    Copy to Clipboard Toggle word wrap 

  3. $ systemctl enable --now qemu-guest-agent
    Copy to Clipboard Toggle word wrap 

  • $ oc get vm <vm_name>
    Copy to Clipboard Toggle word wrap
7.2.5.1.2.
링크 복사

  1. $ net start
    Copy to Clipboard Toggle word wrap
7.2.5.2.
링크 복사

Expand
표 7.3.
   

7.2.5.2.1.
링크 복사

7.2.5.2.2.
링크 복사

7.2.5.2.3.
링크 복사

참고

7.2.5.2.4.
링크 복사

참고

7.2.5.2.5.
링크 복사

작은 정보

  1. # ...
spec:
  domain:
    devices:
      disks:
        - name: virtiocontainerdisk
          bootOrder: 2
    1 
    
          cdrom:
            bus: sata
volumes:
  - containerDisk:
      image: container-native-virtualization/virtio-win
    name: virtiocontainerdisk
    Copy to Clipboard Toggle word wrap
    1 

    • $ virtctl start <vm> -n <namespace>
      Copy to Clipboard Toggle word wrap 

    • $ oc apply -f <vm.yaml>
      Copy to Clipboard Toggle word wrap
7.2.5.3.
링크 복사
7.2.5.3.1.
링크 복사

7.2.6.
링크 복사

중요

7.2.6.1.
링크 복사

7.2.6.2.
링크 복사

7.2.7.
링크 복사

7.2.7.1.
링크 복사

7.2.7.1.1.
링크 복사

7.2.7.1.2.
링크 복사

7.2.7.1.3.
링크 복사

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/cloneFallbackReason: The volume modes of source and target are incompatible
    cdi.kubevirt.io/clonePhase: Succeeded
    cdi.kubevirt.io/cloneType: copy
Copy to Clipboard Toggle word wrap 

NAMESPACE   LAST SEEN   TYPE      REASON                    OBJECT                              MESSAGE
test-ns     0s          Warning   IncompatibleVolumeModes   persistentvolumeclaim/test-target   The volume modes of source and target are incompatible
Copy to Clipboard Toggle word wrap

7.2.7.2.
링크 복사

7.2.7.3.
링크 복사

7.2.7.3.1.
링크 복사

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: golden-volumesnapshot
  namespace: golden-ns
spec:
  volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass
  source:
    persistentVolumeClaimName: golden-snap-source
Copy to Clipboard Toggle word wrap 
spec:
  source:
    snapshot:
      namespace: golden-ns
      name: golden-volumesnapshot
Copy to Clipboard Toggle word wrap
7.2.7.3.2.
링크 복사

참고

    • kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
driver: openshift-storage.rbd.csi.ceph.com
# ...
      Copy to Clipboard Toggle word wrap 

      kind: StorageClass
apiVersion: storage.k8s.io/v1
# ...
provisioner: openshift-storage.rbd.csi.ceph.com
      Copy to Clipboard Toggle word wrap 

  1. apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: <datavolume>
    1 
    
spec:
  source:
    pvc:
      namespace: "<source_namespace>"
    2 
    
      name: "<my_vm_disk>"
    3 
    
  storage: {}
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  2. $ oc create -f <datavolume>.yaml
    Copy to Clipboard Toggle word wrap
    참고

7.2.7.3.3.
링크 복사

  1. $ virtctl create vm --name rhel-9-clone --volume-import type:pvc,src:my-project/imported-volume-q5pr9
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rhel-9-clone
    1 
    
spec:
  dataVolumeTemplates:
  - metadata:
      name: imported-volume-h4qn8
    spec:
      source:
        pvc:
          name: imported-volume-q5pr9
    2 
    
          namespace: my-project
    3 
    
      storage:
        resources: {}
  instancetype:
    inferFromVolume: imported-volume-h4qn8
    4 
    
    inferFromVolumeFailurePolicy: Ignore
  preference:
    inferFromVolume: imported-volume-h4qn8
    5 
    
    inferFromVolumeFailurePolicy: Ignore
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
        memory:
          guest: 512Mi
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - dataVolume:
          name: imported-volume-h4qn8
        name: imported-volume-h4qn8
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5 

  3. $ oc create -f <vm_manifest_file>.yaml
    Copy to Clipboard Toggle word wrap

7.3.
링크 복사

7.3.1.
링크 복사

7.3.1.1.
링크 복사

참고

7.3.1.2.
링크 복사

참고

  1. $ virtctl vnc <vm_name>
    Copy to Clipboard Toggle word wrap 

  2. $ virtctl vnc <vm_name> -v 4
    Copy to Clipboard Toggle word wrap
7.3.1.3.
링크 복사

참고

  1. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p '[{"op": "replace", "path": "/spec/featureGates/deployVmConsoleProxy", "value": true}]'
    Copy to Clipboard Toggle word wrap 

  2. $ curl --header "Authorization: Bearer ${TOKEN}" \
     "https://api.<cluster_fqdn>/apis/token.kubevirt.io/v1alpha1/namespaces/<namespace>/virtualmachines/<vm_name>/vnc?duration=<duration>"
    Copy to Clipboard Toggle word wrap 

    { "token": "eyJhb..." }
    Copy to Clipboard Toggle word wrap 

  3. $ export VNC_TOKEN="<token>"
    Copy to Clipboard Toggle word wrap 

  1. $ oc login --token ${VNC_TOKEN}
    Copy to Clipboard Toggle word wrap 

  2. $ virtctl vnc <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap
주의

$ virtctl delete serviceaccount --namespace "<namespace>" "<vm_name>-vnc-access"
Copy to Clipboard Toggle word wrap
7.3.1.3.1.
링크 복사

    • $ kubectl create rolebinding "${ROLE_BINDING_NAME}" --clusterrole="token.kubevirt.io:generate" --user="${USER_NAME}"
      Copy to Clipboard Toggle word wrap 

    • $ kubectl create rolebinding "${ROLE_BINDING_NAME}" --clusterrole="token.kubevirt.io:generate" --serviceaccount="${SERVICE_ACCOUNT_NAME}"
      Copy to Clipboard Toggle word wrap

7.3.2.
링크 복사

참고

7.3.2.1.
링크 복사

참고

7.3.2.2.
링크 복사

참고

  1. $ virtctl console <vm_name>
    Copy to Clipboard Toggle word wrap 

  2. $ virtctl vnc <vm_name>
    Copy to Clipboard Toggle word wrap 

  3. $ virtctl vnc <vm_name> -v 4
    Copy to Clipboard Toggle word wrap

7.3.3.
링크 복사

7.3.3.1.
링크 복사

참고

7.4.
링크 복사

7.4.1.
링크 복사

  1. $ virtctl create vm --instancetype <my_instancetype> --preference <my_preference>
    Copy to Clipboard Toggle word wrap 

  2. $ virtctl create vm --instancetype virtualmachineinstancetype/<my_instancetype> --preference virtualmachinepreference/<my_preference>
    Copy to Clipboard Toggle word wrap

7.4.2.
링크 복사

  • $ virtctl create vm --volume-import type:pvc,src:my-ns/my-pvc --infer-instancetype --infer-preference
    Copy to Clipboard Toggle word wrap

7.4.3.
링크 복사

  • $ oc label DataSource foo instancetype.kubevirt.io/default-instancetype=<my_instancetype>
    Copy to Clipboard Toggle word wrap

7.5.
링크 복사

7.5.1.
링크 복사

7.5.2.
링크 복사

7.5.2.1.
링크 복사

참고

7.5.2.2.
링크 복사

참고

7.5.2.2.1.
링크 복사

7.5.2.2.2.
링크 복사

    • 참고

7.5.2.2.3.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  dataVolumeTemplates:
    - metadata:
        name: example-vm-volume
      spec:
        sourceRef:
          kind: DataSource
          name: rhel9
          namespace: openshift-virtualization-os-images
        storage:
          resources: {}
  instancetype:
    name: u1.medium
  preference:
    name: rhel.9
  running: true
  template:
    spec:
      domain:
        devices: {}
      volumes:
        - dataVolume:
            name: example-vm-volume
          name: rootdisk
        - cloudInitNoCloud:
    1 
    
            userData: |-
              #cloud-config
              user: cloud-user
          name: cloudinitdisk
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              noCloud: {}
            source:
              secret:
                secretName: authorized-keys
    2 
    
---
apiVersion: v1
kind: Secret
metadata:
  name: authorized-keys
data:
  key: c3NoLXJzYSB...
    3
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  2. $ oc create -f <manifest_file>.yaml
    Copy to Clipboard Toggle word wrap 

  3. $ virtctl start vm example-vm -n example-namespace
    Copy to Clipboard Toggle word wrap 

  • $ oc describe vm example-vm -n example-namespace
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  template:
    spec:
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              noCloud: {}
            source:
              secret:
                secretName: authorized-keys
# ...
    Copy to Clipboard Toggle word wrap

7.5.2.3.
링크 복사

참고

7.5.2.3.1.
링크 복사

참고

7.5.2.3.2.
링크 복사

참고

    • 참고

7.5.2.3.3.
링크 복사

7.5.2.3.4.
링크 복사

참고

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  dataVolumeTemplates:
    - metadata:
        name: example-vm-volume
      spec:
        sourceRef:
          kind: DataSource
          name: rhel9
          namespace: openshift-virtualization-os-images
        storage:
          resources: {}
  instancetype:
    name: u1.medium
  preference:
    name: rhel.9
  running: true
  template:
    spec:
      domain:
        devices: {}
      volumes:
        - dataVolume:
            name: example-vm-volume
          name: rootdisk
        - cloudInitNoCloud:
    1 
    
            userData: |-
              #cloud-config
              runcmd:
              - [ setsebool, -P, virt_qemu_ga_manage_ssh, on ]
          name: cloudinitdisk
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              qemuGuestAgent:
                users: ["cloud-user"]
            source:
              secret:
                secretName: authorized-keys
    2 
    
---
apiVersion: v1
kind: Secret
metadata:
  name: authorized-keys
data:
  key: c3NoLXJzYSB...
    3
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  2. $ oc create -f <manifest_file>.yaml
    Copy to Clipboard Toggle word wrap 

  3. $ virtctl start vm example-vm -n example-namespace
    Copy to Clipboard Toggle word wrap 

  • $ oc describe vm example-vm -n example-namespace
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  template:
    spec:
      accessCredentials:
        - sshPublicKey:
            propagationMethod:
              qemuGuestAgent:
                users: ["cloud-user"]
            source:
              secret:
                secretName: authorized-keys
# ...
    Copy to Clipboard Toggle word wrap

7.5.2.4.
링크 복사

  • $ virtctl -n <namespace> ssh <username>@example-vm -i <ssh_key>
    1
    Copy to Clipboard Toggle word wrap
    1 

    $ virtctl -n my-namespace ssh cloud-user@example-vm -i my-key
    Copy to Clipboard Toggle word wrap

작은 정보

7.5.3.
링크 복사

  1. Host vm/*
  ProxyCommand virtctl port-forward --stdio=true %h %p
    Copy to Clipboard Toggle word wrap 

  2. $ ssh <user>@vm/<vm_name>.<namespace>
    Copy to Clipboard Toggle word wrap

7.5.4.
링크 복사

7.5.4.1.
링크 복사

참고

7.5.4.2.
링크 복사

7.5.4.2.1.
링크 복사

7.5.4.2.2.
링크 복사

7.5.4.2.3.
링크 복사

  • $ virtctl expose vm <vm_name> --name <service_name> --type <service_type> --port <port>
    1
    Copy to Clipboard Toggle word wrap
    1 

    $ virtctl expose vm example-vm --name example-service --type NodePort --port 22
    Copy to Clipboard Toggle word wrap 

  • $ oc get service
    Copy to Clipboard Toggle word wrap

7.5.4.2.4.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  running: false
  template:
    metadata:
      labels:
        special: key
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    참고

  3. apiVersion: v1
kind: Service
metadata:
  name: example-service
  namespace: example-namespace
spec:
# ...
  selector:
    special: key
    1 
    
  type: NodePort
    2 
    
  ports:
    3 
    
    protocol: TCP
    port: 80
    targetPort: 9376
    nodePort: 30000
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  5. $ oc create -f example-service.yaml
    Copy to Clipboard Toggle word wrap 

  • $ oc get service -n example-namespace
    Copy to Clipboard Toggle word wrap
7.5.4.3.
링크 복사

  • $ ssh <user_name>@<ip_address> -p <port>
    1
    Copy to Clipboard Toggle word wrap
    1

7.5.5.
링크 복사

중요

7.5.5.1.
링크 복사

7.5.5.2.
링크 복사

  1. $ oc describe vm <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

    # ...
Interfaces:
  Interface Name:  eth0
  Ip Address:      10.244.0.37/24
  Ip Addresses:
    10.244.0.37/24
    fe80::858:aff:fef4:25/64
  Mac:             0a:58:0a:f4:00:25
  Name:            default
# ...
    Copy to Clipboard Toggle word wrap 

  2. $ ssh <user_name>@<ip_address> -i <ssh_key>
    Copy to Clipboard Toggle word wrap 

    $ ssh cloud-user@10.244.0.37 -i ~/.ssh/id_rsa_cloud-user
    Copy to Clipboard Toggle word wrap

참고

7.6.
링크 복사

7.6.1.
링크 복사

참고

7.6.2.
링크 복사

7.6.3.
링크 복사

  1. $ oc edit vm <vm_name>
    Copy to Clipboard Toggle word wrap 

    • $ oc apply vm <vm_name> -n <namespace>
      Copy to Clipboard Toggle word wrap

7.6.4.
링크 복사

참고

7.6.4.1.
링크 복사
Expand
  

Expand
   

참고

7.6.5.
링크 복사

7.6.6.
링크 복사

7.7.
링크 복사

7.7.1.
링크 복사

참고

7.7.2.
링크 복사

참고

7.7.3.
링크 복사

  1. $ oc edit vm <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

  2. disks:
  - bootOrder: 1
    1 
    
    disk:
      bus: virtio
    name: containerdisk
  - disk:
      bus: virtio
    name: cloudinitdisk
  - cdrom:
      bus: virtio
    name: cd-drive-1
interfaces:
  - boot Order: 2
    2 
    
    macAddress: '02:96:c4:00:00'
    masquerade: {}
    name: default
    Copy to Clipboard Toggle word wrap
    1
    2

7.7.4.
링크 복사

참고

7.8.
링크 복사

7.8.1.
링크 복사

7.8.2.
링크 복사

  • $ oc delete vm <vm_name>
    Copy to Clipboard Toggle word wrap
    참고

7.9.
링크 복사

참고

7.9.1.
링크 복사

  1. apiVersion: export.kubevirt.io/v1beta1
kind: VirtualMachineExport
metadata:
  name: example-export
spec:
  source:
    apiGroup: "kubevirt.io"
    1 
    
    kind: VirtualMachine
    2 
    
    name: example-vm
  ttlDuration: 1h
    3
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  2. $ oc create -f example-export.yaml
    Copy to Clipboard Toggle word wrap 

  3. $ oc get vmexport example-export -o yaml
    Copy to Clipboard Toggle word wrap 

    apiVersion: export.kubevirt.io/v1beta1
kind: VirtualMachineExport
metadata:
  name: example-export
  namespace: example
spec:
  source:
    apiGroup: ""
    kind: PersistentVolumeClaim
    name: example-pvc
  tokenSecretRef: example-token
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-06-21T14:10:09Z"
    reason: podReady
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-06-21T14:09:02Z"
    reason: pvcBound
    status: "True"
    type: PVCReady
  links:
    external:
    1 
    
      cert: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      volumes:
      - formats:
        - format: raw
          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img
        - format: gzip
          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img.gz
        name: example-disk
    internal:
    2 
    
      cert: |-
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      volumes:
      - formats:
        - format: raw
          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img
        - format: gzip
          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img.gz
        name: example-disk
  phase: Ready
  serviceName: virt-export-example-export
    Copy to Clipboard Toggle word wrap

    1
    2

7.9.2.
링크 복사

  • 참고

    2. $ oc get vmexport <export_name> -o jsonpath={.status.links.external.cert} > cacert.crt
      1
      Copy to Clipboard Toggle word wrap
      1 

  2. $ oc get secret export-token-<export_name> -o jsonpath={.data.token} | base64 --decode > token_decode
    1
    Copy to Clipboard Toggle word wrap
    1 

  4. $ oc get vmexport <export_name> -o yaml
    Copy to Clipboard Toggle word wrap 

  5. apiVersion: export.kubevirt.io/v1beta1
kind: VirtualMachineExport
metadata:
  name: example-export
spec:
  source:
    apiGroup: "kubevirt.io"
    kind: VirtualMachine
    name: example-vm
  tokenSecretRef: example-token
status:
#...
  links:
    external:
#...
      manifests:
      - type: all
        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/all
    1 
    
      - type: auth-header-secret
        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret
    2 
    
    internal:
#...
      manifests:
      - type: all
        url: https://virt-export-export-pvc.default.svc/internal/manifests/all
    3 
    
      - type: auth-header-secret
        url: https://virt-export-export-pvc.default.svc/internal/manifests/secret
  phase: Ready
  serviceName: virt-export-example-export
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  7. $ curl --cacert cacert.crt <secret_manifest_url> -H \
    1 
    
"x-kubevirt-export-token:token_decode" -H \
    2 
    
"Accept:application/yaml"
    Copy to Clipboard Toggle word wrap
    1
    2 

    $ curl --cacert cacert.crt https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret -H "x-kubevirt-export-token:token_decode" -H "Accept:application/yaml"
    Copy to Clipboard Toggle word wrap 

  8. $ curl --cacert cacert.crt <all_manifest_url> -H \
    1 
    
"x-kubevirt-export-token:token_decode" -H \
    2 
    
"Accept:application/yaml"
    Copy to Clipboard Toggle word wrap
    1
    2 

    $ curl --cacert cacert.crt https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/all -H "x-kubevirt-export-token:token_decode" -H "Accept:application/yaml"
    Copy to Clipboard Toggle word wrap

7.10.
링크 복사

7.10.1.
링크 복사

참고

7.10.2.
링크 복사

  • $ oc get vmis -A
    Copy to Clipboard Toggle word wrap

7.10.3.
링크 복사

참고

7.10.4.
링크 복사

7.10.5.
링크 복사

  • $ oc delete vmi <vmi_name>
    Copy to Clipboard Toggle word wrap

7.10.6.
링크 복사

7.11.
링크 복사

7.11.1.
링크 복사

참고

7.11.2.
링크 복사

7.11.3.
링크 복사

중요

7.11.4.
링크 복사

7.11.5.
링크 복사

7.12.
링크 복사

중요

7.12.1.
링크 복사

kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  vmStateStorageClass: <storage_class_name>

# ...
Copy to Clipboard Toggle word wrap
참고

7.12.2.
링크 복사

  1. $ oc edit vm <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
    name: example-vm
spec:
  template:
    spec:
      domain:
        devices:
          tpm:
    1 
    
            persistent: true
    2 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2

7.13.
링크 복사

7.13.1.
링크 복사

7.13.2.
링크 복사

Expand
표 7.4.
  

참고

7.13.3.
링크 복사

참고

7.13.3.1.
링크 복사

7.13.3.2.
링크 복사

  1. apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: windows11-installer-run-
  labels:
    pipelinerun: windows11-installer-run
spec:
    params:
    -   name: winImageDownloadURL
        value: <windows_image_download_url>
    1 
    
    -   name: acceptEula
        value: false
    2 
    
    pipelineRef:
        params:
        -   name: catalog
            value: redhat-pipelines
        -   name: type
            value: artifact
        -   name: kind
            value: pipeline
        -   name: name
            value: windows-efi-installer
        -   name: version
            value: 4.17
        resolver: hub
    taskRunSpecs:
    -   pipelineTaskName: modify-windows-iso-file
        PodTemplate:
            securityContext:
                fsGroup: 107
                runAsUser: 107
    Copy to Clipboard Toggle word wrap
    1
    2 

  2. $ oc apply -f windows11-customize-run.yaml
    Copy to Clipboard Toggle word wrap

7.14.
링크 복사

7.14.1.
링크 복사

7.14.1.1.
링크 복사

참고

$ oc annotate namespace my-virtualization-project alpha.kubevirt.io/auto-memory-limits-ratio=1.2
Copy to Clipboard Toggle word wrap 

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. spec:
  featureGates:
    autoResourceLimits: true
    Copy to Clipboard Toggle word wrap
7.14.1.1.1.
링크 복사

주의

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: with-limits
spec:
  running: false
  template:
    spec:
      domain:
# ...
        resources:
          requests:
            memory: 128Mi
          limits:
            memory: 256Mi
    1
    Copy to Clipboard Toggle word wrap
    1

7.14.2.
링크 복사

7.14.2.1.
링크 복사

7.14.2.1.1.
링크 복사

  • apiVersion: aaq.kubevirt.io/v1alpha1
kind: ApplicationAwareResourceQuota
metadata:
  name: example-resource-quota
spec:
  hard:
    requests.memory: 1Gi
    limits.memory: 1Gi
    requests.cpu/vmi: "1"
    1 
    
    requests.memory/vmi: 1Gi
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2 

  • apiVersion: aaq.kubevirt.io/v1alpha1
kind: ApplicationAwareClusterResourceQuota
    1 
    
metadata:
  name: example-resource-quota
spec:
  quota:
    hard:
      requests.memory: 1Gi
      limits.memory: 1Gi
      requests.cpu/vmi: "1"
      requests.memory/vmi: 1Gi
  selector:
    annotations: null
    labels:
      matchLabels:
        kubernetes.io/metadata.name: default
# ...
    Copy to Clipboard Toggle word wrap

    1
    참고

중요

7.14.2.2.
링크 복사

  • $ oc patch hco kubevirt-hyperconverged -n openshift-cnv \
 --type json -p '[{"op": "add", "path": "/spec/featureGates/enableApplicationAwareQuota", "value": true}]'
    Copy to Clipboard Toggle word wrap
7.14.2.3.
링크 복사

  • $ oc patch hco kubevirt-hyperconverged -n openshift-cnv --type merge -p '{
  "spec": {
    "applicationAwareConfig": {
      "vmiCalcConfigName": "DedicatedVirtualResources",
      "namespaceSelector": {
        "matchLabels": {
          "app": "my-app"
        }
      },
      "allowApplicationAwareClusterResourceQuota": true
    }
  }
}'
    Copy to Clipboard Toggle word wrap

7.14.3.
링크 복사

7.14.3.1.
링크 복사

참고

참고

7.14.3.2.
링크 복사

7.14.3.2.1.
링크 복사

주의

metadata:
  name: example-vm-node-selector
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    spec:
      nodeSelector:
        example-key-1: example-value-1
        example-key-2: example-value-2
# ...
Copy to Clipboard Toggle word wrap

7.14.3.2.2.
링크 복사

metadata:
  name: example-vm-pod-affinity
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
1 

          - labelSelector:
              matchExpressions:
              - key: example-key-1
                operator: In
                values:
                - example-value-1
            topologyKey: kubernetes.io/hostname
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
2 

          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: example-key-2
                  operator: In
                  values:
                  - example-value-2
              topologyKey: kubernetes.io/hostname
# ...
Copy to Clipboard Toggle word wrap

1
2
7.14.3.2.3.
링크 복사

metadata:
  name: example-vm-node-affinity
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
1 

            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-key
                operator: In
                values:
                - example-value-1
                - example-value-2
          preferredDuringSchedulingIgnoredDuringExecution:
2 

          - weight: 1
            preference:
              matchExpressions:
              - key: example-node-label-key
                operator: In
                values:
                - example-node-label-value
# ...
Copy to Clipboard Toggle word wrap

1
2
7.14.3.2.4.
링크 복사

참고

metadata:
  name: example-vm-tolerations
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  tolerations:
  - key: "key"
    operator: "Equal"
    value: "virtualization"
    effect: "NoSchedule"
# ...
Copy to Clipboard Toggle word wrap

7.14.4.
링크 복사

중요

7.14.4.1.
링크 복사
7.14.4.2.
링크 복사

7.14.4.2.1.
링크 복사

참고

7.14.4.2.2.
링크 복사

7.14.4.3.
링크 복사

7.14.4.4.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

    • apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  configuration:
    ksmConfiguration:
      nodeLabelSelector: {}
# ...
      Copy to Clipboard Toggle word wrap 

    • apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  configuration:
    ksmConfiguration:
      nodeLabelSelector:
        matchLabels:
          <first_example_key>: "true"
          <second_example_key>: "true"
# ...
      Copy to Clipboard Toggle word wrap 

    • apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  configuration:
# ...
      Copy to Clipboard Toggle word wrap

7.14.5.
링크 복사

7.14.5.1.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  certConfig:
    ca:
      duration: 48h0m0s
      renewBefore: 24h0m0s
    1 
    
    server:
      duration: 24h0m0s
    2 
    
      renewBefore: 12h0m0s
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3
7.14.5.2.
링크 복사

certConfig:
   ca:
     duration: 4h0m0s
     renewBefore: 1h0m0s
   server:
     duration: 4h0m0s
     renewBefore: 4h0m0s
Copy to Clipboard Toggle word wrap 

error: hyperconvergeds.hco.kubevirt.io "kubevirt-hyperconverged" could not be patched: admission webhook "validate-hco.kubevirt.io" denied the request: spec.certConfig: ca.duration is smaller than server.duration
Copy to Clipboard Toggle word wrap

7.14.6.
링크 복사

7.14.6.1.
링크 복사

참고

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
 name: kubevirt-hyperconverged
 namespace: openshift-cnv
spec:
  defaultCPUModel: "EPYC"
    Copy to Clipboard Toggle word wrap

7.14.7.
링크 복사

7.14.7.1.
링크 복사

7.14.7.2.
링크 복사

  1. apiversion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    special: vm-secureboot
  name: vm-secureboot
spec:
  template:
    metadata:
      labels:
        special: vm-secureboot
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
        features:
          acpi: {}
          smm:
            enabled: true
    1 
    
        firmware:
          bootloader:
            efi:
              secureBoot: true
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2 

  2. $ oc create -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap
7.14.7.3.
링크 복사

  • $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op":"replace","path":"/spec/featureGates/VMPersistentState", "value": true}]'
    Copy to Clipboard Toggle word wrap
7.14.7.4.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm
spec:
  template:
    spec:
      domain:
        firmware:
          bootloader:
            efi:
              persistent: true
# ...
    Copy to Clipboard Toggle word wrap

7.14.8.
링크 복사

7.14.8.1.
링크 복사
7.14.8.2.
링크 복사

    1. apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: pxe-net-conf
      1 
      
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "pxe-net-conf",
      2 
      
      "type": "bridge",
      3 
      
      "bridge": "bridge-interface",
      4 
      
      "macspoofchk": false,
      5 
      
      "vlan": 100,
      6 
      
      "disableContainerInterface": true,
      "preserveDefaultVlan": false
      7 
      
    }
      Copy to Clipboard Toggle word wrap
      1
      2
      3
      4
      5
      6
      7 

  2. $ oc create -f pxe-net-conf.yaml
    Copy to Clipboard Toggle word wrap 

    1. interfaces:
- masquerade: {}
  name: default
- bridge: {}
  name: pxe-net
  macAddress: de:00:00:00:00:de
  bootOrder: 1
      Copy to Clipboard Toggle word wrap
      참고

    2. devices:
  disks:
  - disk:
      bus: virtio
    name: containerdisk
    bootOrder: 2
      Copy to Clipboard Toggle word wrap 

    3. networks:
- name: default
  pod: {}
- name: pxe-net
  multus:
    networkName: pxe-net-conf
      Copy to Clipboard Toggle word wrap 

  4. $ oc create -f vmi-pxe-boot.yaml
    Copy to Clipboard Toggle word wrap 

      virtualmachineinstance.kubevirt.io "vmi-pxe-boot" created
    Copy to Clipboard Toggle word wrap 

  5. $ oc get vmi vmi-pxe-boot -o yaml | grep -i phase
  phase: Running
    Copy to Clipboard Toggle word wrap 

  6. $ virtctl vnc vmi-pxe-boot
    Copy to Clipboard Toggle word wrap 

  8. $ virtctl console vmi-pxe-boot
    Copy to Clipboard Toggle word wrap 

  1. $ ip addr
    Copy to Clipboard Toggle word wrap 

    ...
3. eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether de:00:00:00:00:de brd ff:ff:ff:ff:ff:ff
    Copy to Clipboard Toggle word wrap

7.14.8.3.
링크 복사

7.14.9.
링크 복사

7.14.9.1.
링크 복사
7.14.9.2.
링크 복사

7.14.9.3.
링크 복사

참고

  1. kind: VirtualMachine
# ...
spec:
  domain:
    resources:
      requests:
        memory: "4Gi"
    1 
    
    memory:
      hugepages:
        pageSize: "1Gi"
    2 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2 

  2. $ oc apply -f <virtual_machine>.yaml
    Copy to Clipboard Toggle word wrap

7.14.10.
링크 복사

7.14.10.1.
링크 복사

7.14.10.2.
링크 복사
7.14.10.3.
링크 복사

7.14.11.
링크 복사

7.14.11.1.
링크 복사

Expand
  

7.14.11.2.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: myvm
spec:
  template:
    spec:
      domain:
        cpu:
          features:
            - name: apic
    1 
    
              policy: require
    2
    Copy to Clipboard Toggle word wrap
    1
    2
7.14.11.3.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: myvm
spec:
  template:
    spec:
      domain:
        cpu:
          model: Conroe
    1
    Copy to Clipboard Toggle word wrap
    1
7.14.11.4.
링크 복사

  • apiVersion: kubevirt/v1alpha3
kind: VirtualMachine
metadata:
  name: myvm
spec:
  template:
    spec:
      domain:
        cpu:
          model: host-model
    1
    Copy to Clipboard Toggle word wrap
    1
7.14.11.5.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-fedora
spec:
  running: true
  template:
    spec:
      schedulerName: my-scheduler
    1 
    
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
# ...
    Copy to Clipboard Toggle word wrap
    1 

    1. $ oc get pods
      Copy to Clipboard Toggle word wrap 

      NAME                             READY   STATUS    RESTARTS   AGE
virt-launcher-vm-fedora-dpc87    2/2     Running   0          24m
      Copy to Clipboard Toggle word wrap 

    2. $ oc describe pod virt-launcher-vm-fedora-dpc87
      Copy to Clipboard Toggle word wrap 

      [...]
Events:
  Type    Reason     Age   From              Message
  ----    ------     ----  ----              -------
  Normal  Scheduled  21m   my-scheduler  Successfully assigned default/virt-launcher-vm-fedora-dpc87 to node01
[...]
      Copy to Clipboard Toggle word wrap

7.14.12.
링크 복사

7.14.12.1.
링크 복사

7.14.12.1.1.
링크 복사

  • $ oc label node <node_name> nvidia.com/gpu.deploy.operands=false
    1
    Copy to Clipboard Toggle word wrap
    1 

  1. $ oc describe node <node_name>
    Copy to Clipboard Toggle word wrap 

    1. $ oc get pods -n nvidia-gpu-operator
      Copy to Clipboard Toggle word wrap 

      NAME                             READY   STATUS        RESTARTS   AGE
gpu-operator-59469b8c5c-hw9wj    1/1     Running       0          8d
nvidia-sandbox-validator-7hx98   1/1     Running       0          8d
nvidia-sandbox-validator-hdb7p   1/1     Running       0          8d
nvidia-sandbox-validator-kxwj7   1/1     Terminating   0          9d
nvidia-vfio-manager-7w9fs        1/1     Running       0          8d
nvidia-vfio-manager-866pz        1/1     Running       0          8d
nvidia-vfio-manager-zqtck        1/1     Terminating   0          9d
      Copy to Clipboard Toggle word wrap 

    2. $ oc get pods -n nvidia-gpu-operator
      Copy to Clipboard Toggle word wrap 

      NAME                             READY   STATUS    RESTARTS   AGE
gpu-operator-59469b8c5c-hw9wj    1/1     Running   0          8d
nvidia-sandbox-validator-7hx98   1/1     Running   0          8d
nvidia-sandbox-validator-hdb7p   1/1     Running   0          8d
nvidia-vfio-manager-7w9fs        1/1     Running   0          8d
nvidia-vfio-manager-866pz        1/1     Running   0          8d
      Copy to Clipboard Toggle word wrap

7.14.12.2.
링크 복사
7.14.12.2.1.
링크 복사

7.14.12.2.2.
링크 복사

  1. apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
    1 
    
  name: 100-worker-iommu
    2 
    
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
      - intel_iommu=on
    3 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  2. $ oc create -f 100-worker-kernel-arg-iommu.yaml
    Copy to Clipboard Toggle word wrap 

  • $ oc get MachineConfig
    Copy to Clipboard Toggle word wrap
7.14.12.2.3.
링크 복사

  1. $ lspci -nnv | grep -i nvidia
    Copy to Clipboard Toggle word wrap 

    02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)
    Copy to Clipboard Toggle word wrap

  2. 참고

    variant: openshift
version: 4.17.0
metadata:
  name: 100-worker-vfiopci
  labels:
    machineconfiguration.openshift.io/role: worker
    1 
    
storage:
  files:
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=10de:1eb8
    2 
    
  - path: /etc/modules-load.d/vfio-pci.conf
    3 
    
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  3. $ butane 100-worker-vfiopci.bu -o 100-worker-vfiopci.yaml
    Copy to Clipboard Toggle word wrap 

  4. $ oc apply -f 100-worker-vfiopci.yaml
    Copy to Clipboard Toggle word wrap 

  5. $ oc get MachineConfig
    Copy to Clipboard Toggle word wrap 

    NAME                             GENERATEDBYCONTROLLER                      IGNITIONVERSION  AGE
00-master                        d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
00-worker                        d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
01-master-container-runtime      d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
01-master-kubelet                d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
01-worker-container-runtime      d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
01-worker-kubelet                d3da910bfa9f4b599af4ed7f5ac270d55950a3a1   3.2.0            25h
100-worker-iommu                                                            3.2.0            30s
100-worker-vfiopci-configuration                                            3.2.0            30s
    Copy to Clipboard Toggle word wrap 

  • $ lspci -nnk -d 10de:
    Copy to Clipboard Toggle word wrap 

    04:00.0 3D controller [0302]: NVIDIA Corporation GP102GL [Tesla P40] [10de:1eb8] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1eb8]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
    Copy to Clipboard Toggle word wrap

7.14.12.2.4.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    1 
    
    pciHostDevices:
    2 
    
    - pciDeviceSelector: "10DE:1DB6"
    3 
    
      resourceName: "nvidia.com/GV100GL_Tesla_V100"
    4 
    
    - pciDeviceSelector: "10DE:1EB8"
      resourceName: "nvidia.com/TU104GL_Tesla_T4"
    - pciDeviceSelector: "8086:6F54"
      resourceName: "intel.com/qat"
      externalResourceProvider: true
    5 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    참고

  • $ oc describe node <node_name>
    Copy to Clipboard Toggle word wrap 

    Capacity:
  cpu:                            64
  devices.kubevirt.io/kvm:        110
  devices.kubevirt.io/tun:        110
  devices.kubevirt.io/vhost-net:  110
  ephemeral-storage:              915128Mi
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         131395264Ki
  nvidia.com/GV100GL_Tesla_V100   1
  nvidia.com/TU104GL_Tesla_T4     1
  intel.com/qat:                  1
  pods:                           250
Allocatable:
  cpu:                            63500m
  devices.kubevirt.io/kvm:        110
  devices.kubevirt.io/tun:        110
  devices.kubevirt.io/vhost-net:  110
  ephemeral-storage:              863623130526
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         130244288Ki
  nvidia.com/GV100GL_Tesla_V100   1
  nvidia.com/TU104GL_Tesla_T4     1
  intel.com/qat:                  1
  pods:                           250
    Copy to Clipboard Toggle word wrap

7.14.12.2.5.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  permittedHostDevices:
    pciHostDevices:
    - pciDeviceSelector: "10DE:1DB6"
      resourceName: "nvidia.com/GV100GL_Tesla_V100"
    - pciDeviceSelector: "10DE:1EB8"
      resourceName: "nvidia.com/TU104GL_Tesla_T4"
# ...
    Copy to Clipboard Toggle word wrap 

  • $ oc describe node <node_name>
    Copy to Clipboard Toggle word wrap 

    Capacity:
  cpu:                            64
  devices.kubevirt.io/kvm:        110
  devices.kubevirt.io/tun:        110
  devices.kubevirt.io/vhost-net:  110
  ephemeral-storage:              915128Mi
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         131395264Ki
  nvidia.com/GV100GL_Tesla_V100   1
  nvidia.com/TU104GL_Tesla_T4     1
  intel.com/qat:                  0
  pods:                           250
Allocatable:
  cpu:                            63500m
  devices.kubevirt.io/kvm:        110
  devices.kubevirt.io/tun:        110
  devices.kubevirt.io/vhost-net:  110
  ephemeral-storage:              863623130526
  hugepages-1Gi:                  0
  hugepages-2Mi:                  0
  memory:                         130244288Ki
  nvidia.com/GV100GL_Tesla_V100   1
  nvidia.com/TU104GL_Tesla_T4     1
  intel.com/qat:                  0
  pods:                           250
    Copy to Clipboard Toggle word wrap

7.14.12.3.
링크 복사

7.14.12.3.1.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  domain:
    devices:
      hostDevices:
      - deviceName: nvidia.com/TU104GL_Tesla_T4
    1 
    
        name: hostdevices1
    Copy to Clipboard Toggle word wrap

    1 

  • $ lspci -nnk | grep NVIDIA
    Copy to Clipboard Toggle word wrap 

    $ 02:01.0 3D controller [0302]: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] [10de:1eb8] (rev a1)
    Copy to Clipboard Toggle word wrap

7.14.13.
링크 복사

7.14.13.1.
링크 복사

참고

7.14.13.2.
링크 복사

7.14.13.2.1.
링크 복사

  1. apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
    1 
    
  name: 100-worker-iommu
    2 
    
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
      - intel_iommu=on
    3 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  2. $ oc create -f 100-worker-kernel-arg-iommu.yaml
    Copy to Clipboard Toggle word wrap 

  • $ oc get MachineConfig
    Copy to Clipboard Toggle word wrap
7.14.13.3.
링크 복사

참고

7.14.13.3.1.
링크 복사

7.14.13.3.2.
링크 복사

  • 중요

  • kind: ClusterPolicy
apiVersion: nvidia.com/v1
metadata:
  name: gpu-cluster-policy
spec:
  operator:
    defaultRuntime: crio
    use_ocp_driver_toolkit: true
    initContainer: {}
  sandboxWorkloads:
    enabled: true
    defaultWorkload: vm-vgpu
  driver:
    enabled: false
    1 
    
  dcgmExporter: {}
  dcgm:
    enabled: true
  daemonsets: {}
  devicePlugin: {}
  gfd: {}
  migManager:
    enabled: true
  nodeStatusExporter:
    enabled: true
  mig:
    strategy: single
  toolkit:
    enabled: true
  validator:
    plugin:
      env:
        - name: WITH_WORKLOAD
          value: "true"
  vgpuManager:
    enabled: true
    2 
    
    repository: <vgpu_container_registry>
    3 
    
    image: <vgpu_image_name>
    version: nvidia-vgpu-manager
  vgpuDeviceManager:
    enabled: false
    4 
    
    config:
      name: vgpu-devices-config
      default: default
  sandboxDevicePlugin:
    enabled: false
    5 
    
  vfioManager:
    enabled: false
    6
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    6
7.14.13.4.
링크 복사

# ...
mediatedDevicesConfiguration:
  mediatedDeviceTypes:
  - nvidia-222
  - nvidia-228
  - nvidia-105
  - nvidia-108
# ...
Copy to Clipboard Toggle word wrap 

nvidia-105
# ...
nvidia-108
nvidia-217
nvidia-299
# ...
Copy to Clipboard Toggle word wrap 

# ...
mediatedDevicesConfiguration:
  mediatedDeviceTypes:
  - nvidia-22
  - nvidia-223
  - nvidia-224
# ...
Copy to Clipboard Toggle word wrap

7.14.13.5.
링크 복사

7.14.13.5.1.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap

    예 7.1. 

    apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  mediatedDevicesConfiguration:
    mediatedDeviceTypes:
    - nvidia-231
    nodeMediatedDeviceTypes:
    - mediatedDeviceTypes:
      - nvidia-233
      nodeSelector:
        kubernetes.io/hostname: node-11.redhat.com
  permittedHostDevices:
    mediatedDevices:
    - mdevNameSelector: GRID T4-2Q
      resourceName: nvidia.com/GRID_T4-2Q
    - mdevNameSelector: GRID T4-8Q
      resourceName: nvidia.com/GRID_T4-8Q
# ...
    Copy to Clipboard Toggle word wrap 

  2. # ...
spec:
  mediatedDevicesConfiguration:
    mediatedDeviceTypes:
    1 
    
    - <device_type>
    nodeMediatedDeviceTypes:
    2 
    
    - mediatedDeviceTypes:
    3 
    
      - <device_type>
      nodeSelector:
    4 
    
        <node_selector_key>: <node_selector_value>
# ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    중요

    1. $ oc get $NODE -o json \
  | jq '.status.allocatable \
    | with_entries(select(.key | startswith("nvidia.com/"))) \
    | with_entries(select(.value != "0"))'
      Copy to Clipboard Toggle word wrap 

  4. # ...
  permittedHostDevices:
    mediatedDevices:
    - mdevNameSelector: GRID T4-2Q
    1 
    
      resourceName: nvidia.com/GRID_T4-2Q
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2 

  • $ oc describe node <node_name>
    Copy to Clipboard Toggle word wrap
7.14.13.5.2.
링크 복사

  • 참고

7.14.13.5.3.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  mediatedDevicesConfiguration:
    mediatedDeviceTypes:
    1 
    
      - nvidia-231
  permittedHostDevices:
    mediatedDevices:
    2 
    
    - mdevNameSelector: GRID T4-2Q
      resourceName: nvidia.com/GRID_T4-2Q
    Copy to Clipboard Toggle word wrap

    1
    2
7.14.13.6.
링크 복사

7.14.13.6.1.
링크 복사

  • apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  domain:
    devices:
      gpus:
      - deviceName: nvidia.com/TU104GL_Tesla_T4
    1 
    
        name: gpu1
    2 
    
      - deviceName: nvidia.com/GRID_T4-2Q
        name: gpu2
    Copy to Clipboard Toggle word wrap

    1
    2 

  • $ lspci -nnk | grep <device_name>
    Copy to Clipboard Toggle word wrap
7.14.13.6.2.
링크 복사

참고

7.14.14.
링크 복사

7.14.14.1.
링크 복사

  1. $ lsusb
    Copy to Clipboard Toggle word wrap 

  2. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  3. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
   name: kubevirt-hyperconverged
   namespace: openshift-cnv
spec:
  configuration:
    permittedHostDevices:
    1 
    
      usbHostDevices:
    2 
    
        - resourceName: kubevirt.io/peripherals
    3 
    
          selectors:
            - vendor: "045e"
              product: "07a5"
            - vendor: "062a"
              product: "4102"
            - vendor: "072f"
              product: "b100"
    Copy to Clipboard Toggle word wrap

    1
    2
    3
7.14.14.2.
링크 복사

  1. $ oc /dev/serial/by-id/usb-VENDOR_device_name
    Copy to Clipboard Toggle word wrap 

  2. $ oc edit vmi vmi-usb
    Copy to Clipboard Toggle word wrap 

  3. apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  labels:
    special: vmi-usb
  name: vmi-usb
    1 
    
spec:
  domain:
    devices:
      hostDevices:
      - deviceName: kubevirt.io/peripherals
        name: local-peripherals
# ...
    Copy to Clipboard Toggle word wrap

    1

7.14.15.
링크 복사

7.14.15.1.
링크 복사

7.14.15.2.
링크 복사

중요

      2. 중요

7.14.15.3.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    metadata:
      annotations:
        descheduler.alpha.kubernetes.io/evict: "true"
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  deschedulingIntervalSeconds: 3600
  profiles:
  - LongLifecycle
    1 
    
  mode: Predictive
    2 
    
  profileCustomizations:
    devEnableEvictionsInBackground: true
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3

7.14.16.
링크 복사

7.14.17.
링크 복사

7.14.17.1.
링크 복사

  • $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type=json -p='[{"op": "add", "path": "/spec/tuningPolicy", \
  "value": "highBurst"}]'
    Copy to Clipboard Toggle word wrap 

  • $ oc get kubevirt.kubevirt.io/kubevirt-kubevirt-hyperconverged \
  -n openshift-cnv -o go-template --template='{{range $config, \
  $value := .spec.configuration}} {{if eq $config "apiConfiguration" \
  "webhookConfiguration" "controllerConfiguration" "handlerConfiguration"}} \
  {{"\n"}} {{$config}} = {{$value}} {{end}} {{end}} {{"\n"}}
    Copy to Clipboard Toggle word wrap

7.14.18.
링크 복사

7.14.18.1.
링크 복사

7.14.18.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. ...
spec:
  resourceRequirements:
    vmiCPUAllocationRatio: 1
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1

7.14.19.
링크 복사

참고

7.14.19.1.
링크 복사
7.14.19.2.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VM
spec:
  domain:
    devices:
      networkInterfaceMultiqueue: true
    Copy to Clipboard Toggle word wrap

7.14.20.
링크 복사

7.15.
링크 복사

7.15.1.
링크 복사

참고

7.15.1.1.
링크 복사

7.15.1.2.
링크 복사

  • $ virtctl addvolume <virtual-machine|virtual-machine-instance> \
  --volume-name=<datavolume|PVC> \
  [--persist] [--serial=<label-name>]
    Copy to Clipboard Toggle word wrap 

  • $ virtctl removevolume <virtual-machine|virtual-machine-instance> \
  --volume-name=<datavolume|PVC>
    Copy to Clipboard Toggle word wrap

7.15.2.
링크 복사

7.15.2.1.
링크 복사

  1. $ oc edit pvc <pvc_name>
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: vm-disk-expand
spec:
  accessModes:
     - ReadWriteMany
  resources:
    requests:
       storage: 3Gi
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
7.15.2.2.
링크 복사

  1. apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: blank-image-datavolume
spec:
  source:
    blank: {}
  storage:
    resources:
      requests:
        storage: <2Gi>
    1 
    
  storageClassName: "<storage_class>"
    2
    Copy to Clipboard Toggle word wrap

    1
    2 

  2. $ oc create -f <blank-image-datavolume>.yaml
    Copy to Clipboard Toggle word wrap

7.15.3.
링크 복사

7.15.3.1.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: <vm_name>
spec:
  template:
# ...
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: rootdisk
            errorPolicy: report
    1 
    
            disk1: disk_one
    2 
    
          - disk:
              bus: virtio
            name: cloudinitdisk
            disk2: disk_two
            shareable: true
    3 
    
          interfaces:
          - masquerade: {}
            name: default
    Copy to Clipboard Toggle word wrap
    1
    2
    3
7.15.3.2.
링크 복사

중요

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-0
spec:
  template:
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: sata
            name: rootdisk
          - errorPolicy: report
    1 
    
            lun:
    2 
    
              bus: scsi
              reservation: true
    3 
    
            name: na-shared
            serial: shared1234
      volumes:
      - dataVolume:
          name: vm-0
        name: rootdisk
      - name: na-shared
        persistentVolumeClaim:
          claimName: pvc-na-share
    Copy to Clipboard Toggle word wrap
    1
    2
    3
7.15.3.2.1.
링크 복사

7.15.3.2.2.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-0
spec:
  template:
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: sata
            name: rootdisk
          - errorPolicy: report
            lun:
    1 
    
              bus: scsi
              reservation: true
    2 
    
            name: na-shared
            serial: shared1234
      volumes:
      - dataVolume:
          name: vm-0
        name: rootdisk
      - name: na-shared
        persistentVolumeClaim:
          claimName: pvc-na-share
    Copy to Clipboard Toggle word wrap
    1
    2
7.15.3.3.
링크 복사

7.15.3.3.1.
링크 복사

7.15.3.3.2.
링크 복사

  1. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p \
'[{"op":"replace","path":"/spec/featureGates/persistentReservation", "value": true}]'
    Copy to Clipboard Toggle word wrap

8장.
링크 복사

8.1.
링크 복사

참고

그림 8.1.

View larger image

[D]

참고

8.1.1.
링크 복사

8.1.2.
링크 복사

8.1.3.
링크 복사

  1. 참고

참고

8.1.3.1.
링크 복사

Expand
표 8.1.
   

8.1.4.
링크 복사

8.1.5.
링크 복사

8.1.6.
링크 복사

8.2.
링크 복사

참고

8.2.1.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - name: default
              masquerade: {}
    1 
    
              ports:
    2 
    
                - port: 80
# ...
      networks:
      - name: default
        pod: {}
    Copy to Clipboard Toggle word wrap
    1
    2
    참고

  2. $ oc create -f <vm-name>.yaml
    Copy to Clipboard Toggle word wrap

8.2.2.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm-ipv6
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - name: default
              masquerade: {}
    1 
    
              ports:
                - port: 80
    2 
    
# ...
      networks:
      - name: default
        pod: {}
      volumes:
      - cloudInitNoCloud:
          networkData: |
            version: 2
            ethernets:
              eth0:
                dhcp4: true
                addresses: [ fd10:0:2::2/120 ]
    3 
    
                gateway6: fd10:0:2::1
    4
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4 

  2. $ oc create -f example-vm-ipv6.yaml
    Copy to Clipboard Toggle word wrap 

$ oc get vmi <vmi-name> -o jsonpath="{.status.interfaces[*].ipAddresses}"
Copy to Clipboard Toggle word wrap

8.2.3.
링크 복사

참고

8.3.
링크 복사

8.3.1.
링크 복사

참고

8.3.2.
링크 복사

8.3.3.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  running: false
  template:
    metadata:
      labels:
        special: key
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    참고

  3. apiVersion: v1
kind: Service
metadata:
  name: example-service
  namespace: example-namespace
spec:
# ...
  selector:
    special: key
    1 
    
  type: NodePort
    2 
    
  ports:
    3 
    
    protocol: TCP
    port: 80
    targetPort: 9376
    nodePort: 30000
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  5. $ oc create -f example-service.yaml
    Copy to Clipboard Toggle word wrap 

  • $ oc get service -n example-namespace
    Copy to Clipboard Toggle word wrap

8.4.
링크 복사

중요

8.4.1.
링크 복사

  1. apiVersion: v1
kind: Service
metadata:
  name: mysubdomain
    1 
    
spec:
  selector:
    expose: me
    2 
    
  clusterIP: None
    3 
    
  ports:
    4 
    
  - protocol: TCP
    port: 1234
    targetPort: 1234
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4 

  3. $ oc create -f headless_service.yaml
    Copy to Clipboard Toggle word wrap

8.4.2.
링크 복사

  1. $ oc edit vm <vm_name>
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-fedora
spec:
  template:
    metadata:
      labels:
        expose: me
    1 
    
    spec:
      hostname: "myvm"
    2 
    
      subdomain: "mysubdomain"
    3 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3

8.4.3.
링크 복사

  1. $ virtctl console vm-fedora
    Copy to Clipboard Toggle word wrap 

  2. $ ping myvm.mysubdomain.<namespace>.svc.cluster.local
    Copy to Clipboard Toggle word wrap 

    PING myvm.mysubdomain.default.svc.cluster.local (10.244.0.57) 56(84) bytes of data.
64 bytes from myvm.mysubdomain.default.svc.cluster.local (10.244.0.57): icmp_seq=1 ttl=64 time=0.029 ms
    Copy to Clipboard Toggle word wrap

8.5.
링크 복사

참고

8.5.1.
링크 복사

  • apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br1-eth1-policy
    1 
    
spec:
  desiredState:
    interfaces:
      - name: br1
    2 
    
        description: Linux bridge with eth1 as a port
    3 
    
        type: linux-bridge
    4 
    
        state: up
    5 
    
        ipv4:
          enabled: false
    6 
    
        bridge:
          options:
            stp:
              enabled: false
    7 
    
          port:
            - name: eth1
    8
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8

8.5.2.
링크 복사

8.5.2.1.
링크 복사

주의

  2. 참고

8.5.2.2.
링크 복사

주의

  1. apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-network
    1 
    
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1
    2 
    
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "bridge-network",
    3 
    
      "type": "bridge",
    4 
    
      "bridge": "br1",
    5 
    
      "macspoofchk": false,
    6 
    
      "vlan": 100,
    7 
    
      "disableContainerInterface": true,
      "preserveDefaultVlan": false
    8 
    
    }
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8
    참고

  2. $ oc create -f network-attachment-definition.yaml
    1
    Copy to Clipboard Toggle word wrap
    1 

  • $ oc get network-attachment-definition bridge-network
    Copy to Clipboard Toggle word wrap

8.5.3.
링크 복사

8.5.3.1.
링크 복사

Expand
  

8.5.3.2.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - bridge: {}
              name: bridge-net
    1 
    
# ...
      networks:
        - name: bridge-net
    2 
    
          multus:
            networkName: bridge-network
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  2. $ oc apply -f example-vm.yaml
    Copy to Clipboard Toggle word wrap

8.6.
링크 복사

8.6.1.
링크 복사

참고

  1. apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: <name>
    1 
    
  namespace: openshift-sriov-network-operator
    2 
    
spec:
  resourceName: <sriov_resource_name>
    3 
    
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
    4 
    
  priority: <priority>
    5 
    
  mtu: <mtu>
    6 
    
  numVfs: <num>
    7 
    
  nicSelector:
    8 
    
    vendor: "<vendor_code>"
    9 
    
    deviceID: "<device_id>"
    10 
    
    pfNames: ["<pf_name>", ...]
    11 
    
    rootDevices: ["<pci_bus_id>", "..."]
    12 
    
  deviceType: vfio-pci
    13 
    
  isRdma: false
    14
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    참고

  3. $ oc create -f <name>-sriov-node-network.yaml
    Copy to Clipboard Toggle word wrap 

  4. $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'
    Copy to Clipboard Toggle word wrap

8.6.2.
링크 복사

참고

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: <name>
1 

  namespace: openshift-sriov-network-operator
2 

spec:
  resourceName: <sriov_resource_name>
3 

  networkNamespace: <target_namespace>
4 

  vlan: <vlan>
5 

  spoofChk: "<spoof_check>"
6 

  linkState: <link_state>
7 

  maxTxRate: <max_tx_rate>
8 

  minTxRate: <min_rx_rate>
9 

  vlanQoS: <vlan_qos>
10 

  trust: "<trust_vf>"
11 

  capabilities: <capabilities>
12
Copy to Clipboard Toggle word wrap
1
2
3
4
5
6
중요

7
8
9
참고

10
11
중요

12 

  1. $ oc create -f <name>-sriov-network.yaml
    Copy to Clipboard Toggle word wrap 

  2. $ oc get net-attach-def -n <namespace>
    Copy to Clipboard Toggle word wrap

8.6.3.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  domain:
    devices:
      interfaces:
      - name: nic1
    1 
    
        sriov: {}
  networks:
  - name: nic1
    2 
    
    multus:
        networkName: sriov-network
    3 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  2. $ oc apply -f <vm_sriov>.yaml
    1
    Copy to Clipboard Toggle word wrap
    1

8.6.4.
링크 복사

8.7.
링크 복사

8.7.1.
링크 복사

    1. $ oc label node <node_name> node-role.kubernetes.io/worker-dpdk=""
      Copy to Clipboard Toggle word wrap 

    2. apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-dpdk
  labels:
    machineconfiguration.openshift.io/role: worker-dpdk
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - worker-dpdk
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-dpdk: ""
      Copy to Clipboard Toggle word wrap 

  3. apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: profile-1
spec:
  cpu:
    isolated: 4-39,44-79
    reserved: 0-3,40-43
  globallyDisableIrqLoadBalancing: true
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 8
      node: 0
      size: 1G
  net:
    userLevelNetworking: true
  nodeSelector:
    node-role.kubernetes.io/worker-dpdk: ""
  numa:
    topologyPolicy: single-numa-node
    Copy to Clipboard Toggle word wrap

    참고

  4. $ oc get performanceprofiles.performance.openshift.io profile-1 -o=jsonpath='{.status.runtimeClass}{"\n"}'
    Copy to Clipboard Toggle word wrap 

  5. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
    --type='json' -p='[{"op": "add", "path": "/spec/defaultRuntimeClass", "value":"<runtimeclass-name>"}]'
    Copy to Clipboard Toggle word wrap
    참고

  6. $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
    --type='json' -p='[{"op": "replace", "path": "/spec/featureGates/alignCPUs", "value": true}]'
    Copy to Clipboard Toggle word wrap
    참고

  7. apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: policy-1
  namespace: openshift-sriov-network-operator
spec:
  resourceName: intel_nics_dpdk
  deviceType: vfio-pci
  mtu: 9000
  numVfs: 4
  priority: 99
  nicSelector:
    vendor: "8086"
    deviceID: "1572"
    pfNames:
      - eno3
    rootDevices:
      - "0000:19:00.2"
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
    Copy to Clipboard Toggle word wrap

8.7.1.1.
링크 복사

  1. $ oc label node <node_name> node-role.kubernetes.io/worker-dpdk-
    Copy to Clipboard Toggle word wrap 

  2. $ oc delete mcp worker-dpdk
    Copy to Clipboard Toggle word wrap

8.7.2.
링크 복사

  1. $ oc create ns dpdk-checkup-ns
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: dpdk-sriovnetwork
  namespace: openshift-sriov-network-operator
spec:
  ipam: |
    {
      "type": "host-local",
      "subnet": "10.56.217.0/24",
      "rangeStart": "10.56.217.171",
      "rangeEnd": "10.56.217.181",
      "routes": [{
        "dst": "0.0.0.0/0"
      }],
      "gateway": "10.56.217.1"
    }
  networkNamespace: dpdk-checkup-ns
    1 
    
  resourceName: intel_nics_dpdk
    2 
    
  spoofChk: "off"
  trust: "on"
  vlan: 1019
    Copy to Clipboard Toggle word wrap

    1
    2

8.7.3.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rhel-dpdk-vm
spec:
  running: true
  template:
    metadata:
      annotations:
        cpu-load-balancing.crio.io: disable
    1 
    
        cpu-quota.crio.io: disable
    2 
    
        irq-load-balancing.crio.io: disable
    3 
    
    spec:
      domain:
        cpu:
          sockets: 1
    4 
    
          cores: 5
    5 
    
          threads: 2
          dedicatedCpuPlacement: true
          isolateEmulatorThread: true
        interfaces:
          - masquerade: {}
            name: default
          - model: virtio
            name: nic-east
            pciAddress: '0000:07:00.0'
            sriov: {}
          networkInterfaceMultiqueue: true
          rng: {}
      memory:
        hugepages:
          pageSize: 1Gi
    6 
    
          guest: 8Gi
      networks:
        - name: default
          pod: {}
        - multus:
            networkName: dpdk-net
    7 
    
          name: nic-east
# ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    6
    7 

  3. $ oc apply -f <file_name>.yaml
    Copy to Clipboard Toggle word wrap 

    1. $ grubby --update-kernel=ALL --args="default_hugepagesz=1GB hugepagesz=1G hugepages=8"
      Copy to Clipboard Toggle word wrap 

    2. $ dnf install -y tuned-profiles-cpu-partitioning
      Copy to Clipboard Toggle word wrap 
      $ echo isolated_cores=2-9 > /etc/tuned/cpu-partitioning-variables.conf
      Copy to Clipboard Toggle word wrap 

      $ tuned-adm profile cpu-partitioning
      Copy to Clipboard Toggle word wrap 

    3. $ dnf install -y driverctl
      Copy to Clipboard Toggle word wrap 
      $ driverctl set-override 0000:07:00.0 vfio-pci
      Copy to Clipboard Toggle word wrap

8.8.
링크 복사

참고

  1. 참고

8.8.1.
링크 복사

참고

8.8.1.1.
링크 복사

  1. apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: l2-network
  namespace: my-namespace
spec:
  config: |-
    {
            "cniVersion": "0.3.1",
    1 
    
            "name": "my-namespace-l2-network",
    2 
    
            "type": "ovn-k8s-cni-overlay",
    3 
    
            "topology":"layer2",
    4 
    
            "mtu": 1300,
    5 
    
            "netAttachDefName": "my-namespace/l2-network"
    6 
    
    }
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    참고

  2. $ oc apply -f <filename>.yaml
    Copy to Clipboard Toggle word wrap
8.8.1.2.
링크 복사

  1. apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: mapping
    1 
    
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
    2 
    
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: localnet-network
    3 
    
        bridge: br-ex
    4 
    
        state: present
    5
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    참고

  2. apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: localnet-network
  namespace: default
spec:
  config: |-
    {
            "cniVersion": "0.3.1",
    1 
    
            "name": "localnet-network",
    2 
    
            "type": "ovn-k8s-cni-overlay",
    3 
    
            "topology": "localnet",
    4 
    
            "netAttachDefName": "default/localnet-network"
    5 
    
    }
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5 

  3. $ oc apply -f <filename>.yaml
    Copy to Clipboard Toggle word wrap
8.8.1.3.
링크 복사

8.8.1.4.
링크 복사

8.8.2.
링크 복사

8.8.2.1.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-server
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          interfaces:
          - name: secondary
    1 
    
            bridge: {}
        resources:
          requests:
            memory: 1024Mi
      networks:
      - name: secondary
    2 
    
        multus:
          networkName: <nad_name>
    3 
    
      nodeSelector:
        node-role.kubernetes.io/worker: ''
    4 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4 

  2. $ oc apply -f <filename>.yaml
    Copy to Clipboard Toggle word wrap

8.9.
링크 복사

참고

8.9.1.
링크 복사

참고

8.9.2.
링크 복사

  1. $ virtctl start <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

  2. $ oc edit vm <vm_name>
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-fedora
template:
  spec:
    domain:
      devices:
        interfaces:
        - name: defaultnetwork
          masquerade: {}
        # new interface
        - name: <secondary_nic>
    1 
    
          bridge: {}
    networks:
    - name: defaultnetwork
      pod: {}
    # new network
    - name: <secondary_nic>
    2 
    
      multus:
        networkName: <nad_name>
    3 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  3. $ virtctl migrate <vm_name>
    Copy to Clipboard Toggle word wrap 

  1. $ oc get VirtualMachineInstanceMigration -w
    Copy to Clipboard Toggle word wrap 

    NAME                        PHASE             VMI
kubevirt-migrate-vm-lj62q   Scheduling        vm-fedora
kubevirt-migrate-vm-lj62q   Scheduled         vm-fedora
kubevirt-migrate-vm-lj62q   PreparingTarget   vm-fedora
kubevirt-migrate-vm-lj62q   TargetReady       vm-fedora
kubevirt-migrate-vm-lj62q   Running           vm-fedora
kubevirt-migrate-vm-lj62q   Succeeded         vm-fedora
    Copy to Clipboard Toggle word wrap 

  2. $ oc get vmi vm-fedora -ojsonpath="{ @.status.interfaces }"
    Copy to Clipboard Toggle word wrap 

    [
  {
    "infoSource": "domain, guest-agent",
    "interfaceName": "eth0",
    "ipAddress": "10.130.0.195",
    "ipAddresses": [
      "10.130.0.195",
      "fd02:0:0:3::43c"
    ],
    "mac": "52:54:00:0e:ab:25",
    "name": "default",
    "queueCount": 1
  },
  {
    "infoSource": "domain, guest-agent, multus-status",
    "interfaceName": "eth1",
    "mac": "02:d8:b8:00:00:2a",
    "name": "bridge-interface",
    1 
    
    "queueCount": 1
  }
]
    Copy to Clipboard Toggle word wrap

    1

8.9.3.
링크 복사

참고

  1. $ oc edit vm <vm_name>
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-fedora
template:
  spec:
    domain:
      devices:
        interfaces:
          - name: defaultnetwork
            masquerade: {}
          # set the interface state to absent
          - name: <secondary_nic>
            state: absent
    1 
    
            bridge: {}
    networks:
      - name: defaultnetwork
        pod: {}
      - name: <secondary_nic>
        multus:
          networkName: <nad_name>
# ...
    Copy to Clipboard Toggle word wrap

    1 

  2. $ virtctl migrate <vm_name>
    Copy to Clipboard Toggle word wrap

8.10.
링크 복사

8.10.1.
링크 복사

중요

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: vm-istio
  name: vm-istio
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: vm-istio
        app: vm-istio
    1 
    
      annotations:
        sidecar.istio.io/inject: "true"
    2 
    
    spec:
      domain:
        devices:
          interfaces:
          - name: default
            masquerade: {}
    3 
    
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        resources:
          requests:
            memory: 1024M
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - containerDisk:
          image: registry:5000/kubevirt/fedora-cloud-container-disk-demo:devel
        name: containerdisk
    Copy to Clipboard Toggle word wrap

    1
    2
    3 

  2. $ oc apply -f <vm_name>.yaml
    1
    Copy to Clipboard Toggle word wrap
    1 

  3. apiVersion: v1
kind: Service
metadata:
  name: vm-istio
spec:
  selector:
    app: vm-istio
    1 
    
  ports:
    - port: 8080
      name: http
      protocol: TCP
    Copy to Clipboard Toggle word wrap
    1 

  4. $ oc create -f <service_name>.yaml
    1
    Copy to Clipboard Toggle word wrap
    1

8.11.
링크 복사

8.11.1.
링크 복사

  1. apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: my-secondary-network
    1 
    
  namespace: openshift-cnv
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "migration-bridge",
    "type": "macvlan",
    "master": "eth1",
    2 
    
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
    3 
    
      "range": "10.200.5.0/24"
    4 
    
    }
  }'
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4 

  2. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  3. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  liveMigrationConfig:
    completionTimeoutPerGiB: 800
    network: <network>
    1 
    
    parallelMigrationsPerCluster: 5
    parallelOutboundMigrationsPerNode: 2
    progressTimeout: 150
# ...
    Copy to Clipboard Toggle word wrap

    1 

  • $ oc get vmi <vmi_name> -o jsonpath='{.status.migrationState.targetNodeAddress}'
    Copy to Clipboard Toggle word wrap

8.11.2.
링크 복사

8.12.
링크 복사

8.12.1.
링크 복사

8.12.1.1.
링크 복사

참고

    • kind: VirtualMachine
spec:
# ...
  template:
  # ...
    spec:
      volumes:
      - cloudInitNoCloud:
          networkData: |
            version: 2
            ethernets:
              eth1:
      1 
      
                dhcp4: true
      Copy to Clipboard Toggle word wrap
      1 

    • kind: VirtualMachine
spec:
# ...
  template:
  # ...
    spec:
      volumes:
      - cloudInitNoCloud:
          networkData: |
            version: 2
            ethernets:
              eth1:
      1 
      
                addresses:
                - 10.10.10.14/24
      2
      Copy to Clipboard Toggle word wrap
      1
      2

8.12.2.
링크 복사

8.12.2.1.
링크 복사

참고

8.12.2.2.
링크 복사

참고

  • $ oc describe vmi <vmi_name>
    Copy to Clipboard Toggle word wrap 

    # ...
Interfaces:
   Interface Name:  eth0
   Ip Address:      10.244.0.37/24
   Ip Addresses:
     10.244.0.37/24
     fe80::858:aff:fef4:25/64
   Mac:             0a:58:0a:f4:00:25
   Name:            default
   Interface Name:  v2
   Ip Address:      1.1.1.7/24
   Ip Addresses:
     1.1.1.7/24
     fe80::f4d9:70ff:fe13:9089/64
   Mac:             f6:d9:70:13:90:89
   Interface Name:  v1
   Ip Address:      1.1.1.1/24
   Ip Addresses:
     1.1.1.1/24
     1.1.1.2/24
     1.1.1.4/24
     2001:de7:0:f101::1/64
     2001:db8:0:f101::1/64
     fe80::1420:84ff:fe10:17aa/64
   Mac:             16:20:84:10:17:aa
    Copy to Clipboard Toggle word wrap

8.13.
링크 복사

중요

8.13.1.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
    featureGates:
      deployKubeSecondaryDNS: true
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1 

  4. $ oc expose -n openshift-cnv deployment/secondary-dns --name=dns-lb \
  --type=LoadBalancer --port=53 --target-port=5353 --protocol='UDP'
    Copy to Clipboard Toggle word wrap 

  5. $ oc get service -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

    NAME       TYPE             CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
dns-lb     LoadBalancer     172.30.27.5    10.46.41.94      53:31829/TCP     5s
    Copy to Clipboard Toggle word wrap 

  6. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  7. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  featureGates:
    deployKubeSecondaryDNS: true
  kubeSecondaryDNSNameServerIP: "10.46.41.94"
    1 
    
# ...
    Copy to Clipboard Toggle word wrap
    1 

  9.  $ oc get dnses.config.openshift.io cluster -o jsonpath='{.spec.baseDomain}'
    Copy to Clipboard Toggle word wrap 

    openshift.example.com
    Copy to Clipboard Toggle word wrap 

  10. vm.<FQDN>. IN NS ns.vm.<FQDN>.
    Copy to Clipboard Toggle word wrap 
    ns.vm.<FQDN>. IN A <kubeSecondaryDNSNameServerIP>
    Copy to Clipboard Toggle word wrap

8.13.2.
링크 복사

  • $ oc get dnses.config.openshift.io cluster -o json | jq .spec.baseDomain
    Copy to Clipboard Toggle word wrap 

  1. $ oc get vm -n <namespace> <vm_name> -o yaml
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
  namespace: example-namespace
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          interfaces:
            - bridge: {}
              name: example-nic
# ...
      networks:
      - multus:
          networkName: bridge-conf
        name: example-nic
    1
    Copy to Clipboard Toggle word wrap

    1 

  2. $ ssh <user_name>@<interface_name>.<vm_name>.<namespace>.vm.<cluster_fqdn>
    Copy to Clipboard Toggle word wrap

8.14.
링크 복사

참고

8.14.1.
링크 복사

  • $ oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io=ignore
    Copy to Clipboard Toggle word wrap 

  • $ oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io-
    Copy to Clipboard Toggle word wrap

9장.
링크 복사

9.1.
링크 복사

9.1.1.
링크 복사

9.1.2.
링크 복사

9.1.3.
링크 복사

9.1.4.
링크 복사

9.2.
링크 복사

중요

9.2.1.
링크 복사

주의

  1. $ oc edit storageprofile <storage_class>
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: <unknown_provisioner_class>
# ...
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    1 
    
    volumeMode: Filesystem
    2 
    
status:
  provisioner: <unknown_provisioner>
  storageClass: <unknown_provisioner_class>
    Copy to Clipboard Toggle word wrap

    1
    2
9.2.1.1.
링크 복사

9.2.1.2.
링크 복사

  • apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: ocs-storagecluster-ceph-rbd-virtualization
spec:
  snapshotClass: ocs-storagecluster-rbdplugin-snapclass
    Copy to Clipboard Toggle word wrap 

  • # oc patch VolumeSnapshotClass ocs-storagecluster-cephfsplugin-snapclass --type=merge -p '{"metadata":{"annotations":{"snapshot.storage.kubernetes.io/is-default-class":"true"}}}'
    Copy to Clipboard Toggle word wrap
9.2.1.3.
링크 복사

  1. $ oc get storageprofile
    Copy to Clipboard Toggle word wrap 

  2. $ oc describe storageprofile <name>
    Copy to Clipboard Toggle word wrap 

    Name:         ocs-storagecluster-ceph-rbd-virtualization
Namespace:
Labels:       app=containerized-data-importer
              app.kubernetes.io/component=storage
              app.kubernetes.io/managed-by=cdi-controller
              app.kubernetes.io/part-of=hyperconverged-cluster
              app.kubernetes.io/version=4.17.2
              cdi.kubevirt.io=
Annotations:  <none>
API Version:  cdi.kubevirt.io/v1beta1
Kind:         StorageProfile
Metadata:
  Creation Timestamp:  2023-11-13T07:58:02Z
  Generation:          2
  Owner References:
    API Version:           cdi.kubevirt.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CDI
    Name:                  cdi-kubevirt-hyperconverged
    UID:                   2d6f169a-382c-4caf-b614-a640f2ef8abb
  Resource Version:        4186799537
  UID:                     14aef804-6688-4f2e-986b-0297fd3aaa68
Spec:
Status:
  Claim Property Sets:
    1 
    
    accessModes:
      ReadWriteMany
    volumeMode:  Block
    accessModes:
      ReadWriteOnce
    volumeMode:  Block
    accessModes:
      ReadWriteOnce
    volumeMode:                   Filesystem
  Clone Strategy:                  csi-clone
    2 
    
  Data Import Cron Source Format:  snapshot
    3 
    
  Provisioner:                     openshift-storage.rbd.csi.ceph.com
  Snapshot Class:                  ocs-storagecluster-rbdplugin-snapclass
  Storage Class:                   ocs-storagecluster-ceph-rbd-virtualization
Events:                            <none>
    Copy to Clipboard Toggle word wrap

    1
    2
    3
9.2.1.4.
링크 복사

참고

apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: <provisioner_class>
# ...
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
1 

    volumeMode:  Filesystem
2 

  cloneStrategy: csi-clone
3 

status:
  provisioner: <provisioner>
  storageClass: <provisioner_class>
Copy to Clipboard Toggle word wrap

1
2
3

9.3.
링크 복사

9.3.1.
링크 복사

9.3.1.1.
링크 복사

참고

    • $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op": "replace", "path": \
  "/spec/featureGates/enableCommonBootImageImport", \
  "value": false}]'
      Copy to Clipboard Toggle word wrap 

    • $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
  --type json -p '[{"op": "replace", "path": \
  "/spec/featureGates/enableCommonBootImageImport", \
  "value": true}]'
      Copy to Clipboard Toggle word wrap

9.3.2.
링크 복사

중요

9.3.2.1.
링크 복사

중요

    1. $ oc get sc -o json| jq '.items[].metadata|select(.annotations."storageclass.kubevirt.io/is-default-virt-class"=="true")|.name'
      Copy to Clipboard Toggle word wrap 

    2. $ oc patch storageclass <storage_class_name> -p '{"metadata": {"annotations": {"storageclass.kubevirt.io/is-default-virt-class": "false"}}}'
      Copy to Clipboard Toggle word wrap 

    3. $ oc get sc -o json| jq '.items[].metadata|select(.annotations."storageclass.kubernetes.io/is-default-class"=="true")|.name'
      Copy to Clipboard Toggle word wrap 

    4. $ oc patch storageclass <storage_class_name> -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'
      Copy to Clipboard Toggle word wrap 

    1. $ oc patch storageclass <storage_class_name> -p '{"metadata": {"annotations": {"storageclass.kubevirt.io/is-default-virt-class": "true"}}}'
      Copy to Clipboard Toggle word wrap 

    2. $ oc patch storageclass <storage_class_name> -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'
      Copy to Clipboard Toggle word wrap 

  3. $ oc delete DataVolume,VolumeSnapshot -n openshift-virtualization-os-images --selector=cdi.kubevirt.io/dataImportCron
    Copy to Clipboard Toggle word wrap 

  4. $ oc get storageprofile <storage_class_name> -o json | jq .status.dataImportCronSourceFormat
    Copy to Clipboard Toggle word wrap
9.3.2.2.
링크 복사

중요

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  dataImportCronTemplates:
  - metadata:
      name: rhel9-image-cron
    spec:
      template:
        spec:
          storage:
            storageClassName: <storage_class>
    1 
    
      schedule: "0 */12 * * *"
    2 
    
      managedDataSource: <data_source>
    3 
    
# ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    참고

  4. $ oc delete DataVolume,VolumeSnapshot -n openshift-virtualization-os-images --selector=cdi.kubevirt.io/dataImportCron
    Copy to Clipboard Toggle word wrap 

  5. $ oc get storageprofile <storage_class_name> -o json | jq .status.dataImportCronSourceFormat
    Copy to Clipboard Toggle word wrap
9.3.2.3.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  dataImportCronTemplates:
  - metadata:
      name: centos-stream9-image-cron
      annotations:
        cdi.kubevirt.io/storage.bind.immediate.requested: "true"
    1 
    
    spec:
      schedule: "0 */12 * * *"
    2 
    
      template:
        spec:
          source:
            registry:
    3 
    
              url: docker://quay.io/containerdisks/centos-stream:9
          storage:
            resources:
              requests:
                storage: 30Gi
      garbageCollect: Outdated
      managedDataSource: centos-stream9
    4
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
9.3.2.4.
링크 복사

참고

  1. $ oc edit storageprofile <storage_class>
    Copy to Clipboard Toggle word wrap 

  3. apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
# ...
spec:
  dataImportCronSourceFormat: snapshot
    Copy to Clipboard Toggle word wrap 

  1. $ oc get storageprofile <storage_class>  -oyaml
    Copy to Clipboard Toggle word wrap

9.3.3.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap

    1. 참고

    2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  dataImportCronTemplates:
  - metadata:
      annotations:
        dataimportcrontemplate.kubevirt.io/enable: 'false'
      name: rhel8-image-cron
# ...
      Copy to Clipboard Toggle word wrap

9.3.4.
링크 복사

  1. $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o yaml
    Copy to Clipboard Toggle word wrap 

    apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
# ...
status:
# ...
  dataImportCronTemplates:
  - metadata:
      annotations:
        cdi.kubevirt.io/storage.bind.immediate.requested: "true"
      name: centos-9-image-cron
    spec:
      garbageCollect: Outdated
      managedDataSource: centos-stream9
      schedule: 55 8/12 * * *
      template:
        metadata: {}
        spec:
          source:
            registry:
              url: docker://quay.io/containerdisks/centos-stream:9
          storage:
            resources:
              requests:
                storage: 30Gi
        status: {}
    status:
      commonTemplate: true
    1 
    
# ...
  - metadata:
      annotations:
        cdi.kubevirt.io/storage.bind.immediate.requested: "true"
      name: user-defined-dic
    spec:
      garbageCollect: Outdated
      managedDataSource: user-defined-centos-stream9
      schedule: 55 8/12 * * *
      template:
        metadata: {}
        spec:
          source:
            registry:
              pullMethod: node
              url: docker://quay.io/containerdisks/centos-stream:9
          storage:
            resources:
              requests:
                storage: 30Gi
        status: {}
    status: {}
    2 
    
# ...
    Copy to Clipboard Toggle word wrap

    1
    2

9.4.
링크 복사

9.4.1.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. # ...
spec:
  filesystemOverhead:
    global: "<new_global_value>"
    1 
    
    storageClass:
      <storage_class_name>: "<new_value_for_this_storage_class>"
    2
    Copy to Clipboard Toggle word wrap
    1
    2 

  • $ oc get cdiconfig -o yaml
    Copy to Clipboard Toggle word wrap 

    $ oc get cdiconfig -o jsonpath='{.items..status.filesystemOverhead}'
    Copy to Clipboard Toggle word wrap

9.5.
링크 복사

9.5.1.
링크 복사

중요

  1. apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  storagePools:
    1 
    
  - name: any_name
    path: "/var/myvolumes"
    2 
    
  workload:
    nodeSelector:
      kubernetes.io/os: linux
    Copy to Clipboard Toggle word wrap
    1
    2 

  3. $ oc create -f hpp_cr.yaml
    Copy to Clipboard Toggle word wrap
9.5.1.1.
링크 복사

참고

9.5.1.2.
링크 복사

참고

  1. apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-csi
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Delete
    1 
    
volumeBindingMode: WaitForFirstConsumer
    2 
    
parameters:
  storagePool: my-storage-pool
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3 

  3. $ oc create -f storageclass_csi.yaml
    Copy to Clipboard Toggle word wrap

9.5.2.
링크 복사

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: iso-pvc
spec:
  volumeMode: Block
1 

  storageClassName: my-storage-class
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
Copy to Clipboard Toggle word wrap

1

9.5.2.1.
링크 복사

중요

  1. apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  storagePools:
    1 
    
  - name: my-storage-pool
    path: "/var/myvolumes"
    2 
    
    pvcTemplate:
      volumeMode: Block
    3 
    
      storageClassName: my-storage-class
    4 
    
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
    5 
    
  workload:
    nodeSelector:
      kubernetes.io/os: linux
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5 

  3. $ oc create -f hpp_pvc_template_pool.yaml
    Copy to Clipboard Toggle word wrap

9.6.
링크 복사

9.6.1.
링크 복사

참고

  1. apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <datavolume-cloner>
    1 
    
rules:
- apiGroups: ["cdi.kubevirt.io"]
  resources: ["datavolumes/source"]
  verbs: ["*"]
    Copy to Clipboard Toggle word wrap
    1 

  2. $ oc create -f <datavolume-cloner.yaml>
    1
    Copy to Clipboard Toggle word wrap
    1 

  3. apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <allow-clone-to-user>
    1 
    
  namespace: <Source namespace>
    2 
    
subjects:
- kind: ServiceAccount
  name: default
  namespace: <Destination namespace>
    3 
    
roleRef:
  kind: ClusterRole
  name: datavolume-cloner
    4 
    
  apiGroup: rbac.authorization.k8s.io
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4 

  4. $ oc create -f <datavolume-cloner.yaml>
    1
    Copy to Clipboard Toggle word wrap
    1

9.7.
링크 복사

9.7.1.
링크 복사

9.7.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  resourceRequirements:
    storageWorkloads:
      limits:
        cpu: "500m"
        memory: "2Gi"
      requests:
        cpu: "250m"
        memory: "1Gi"
    Copy to Clipboard Toggle word wrap

9.8.
링크 복사

9.8.1.
링크 복사

참고

9.8.2.
링크 복사

Expand
  

9.8.3.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  scratchSpaceStorageClass: "<storage_class>"
    1
    Copy to Clipboard Toggle word wrap
    1

9.8.4.
링크 복사

Expand
      

9.9.
링크 복사

9.9.1.
링크 복사

9.9.2.
링크 복사

  • apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: preallocated-datavolume
spec:
  source:
    1 
    
    registry:
      url: <image_url>
    2 
    
  storage:
    resources:
      requests:
        storage: 1Gi
  preallocation: true
# ...
    Copy to Clipboard Toggle word wrap
    1
    2

9.10.
링크 복사

9.10.1.
링크 복사

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: datavolume-example
  annotations:
    v1.multus-cni.io/default-network: bridge-network
1 

# ...
Copy to Clipboard Toggle word wrap

1

9.11.
링크 복사

9.11.1.
링크 복사

10장.
링크 복사

10.1.
링크 복사

10.1.1.
링크 복사

  • 참고

    Product of (Maximum number of nodes that can drain in parallel) and (Highest total VM memory request allocations across nodes)
    Copy to Clipboard Toggle word wrap

10.1.2.
링크 복사

참고

10.1.3.
링크 복사

10.1.4.
링크 복사

10.2.
링크 복사

10.2.1.
링크 복사

  • $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

    apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  liveMigrationConfig:
    bandwidthPerMigration: 64Mi
    1 
    
    completionTimeoutPerGiB: 800
    2 
    
    parallelMigrationsPerCluster: 5
    3 
    
    parallelOutboundMigrationsPerNode: 2
    4 
    
    progressTimeout: 150
    5 
    
    allowPostCopy: false
    6
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    6
참고

10.2.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

    apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  liveMigrationConfig:
    bandwidthPerMigration: 0Mi
    1 
    
    completionTimeoutPerGiB: 150
    2 
    
    parallelMigrationsPerCluster: 5
    3 
    
    parallelOutboundMigrationsPerNode: 1
    4 
    
    progressTimeout: 150
    5 
    
    allowPostCopy: true
    6
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    6
참고

10.2.4.
링크 복사

작은 정보

10.2.4.1.
링크 복사

참고

    1. $ oc edit vm <vm_name>
      Copy to Clipboard Toggle word wrap 

    2. apiVersion: migrations.kubevirt.io/v1alpha1
kind: VirtualMachine
metadata:
  name: <vm_name>
  namespace: default
  labels:
    app: my-app
    environment: production
spec:
  template:
    metadata:
      labels:
        kubevirt.io/domain: <vm_name>
        kubevirt.io/size: large
        kubevirt.io/environment: production
# ...
      Copy to Clipboard Toggle word wrap 

  2. apiVersion: migrations.kubevirt.io/v1alpha1
kind: MigrationPolicy
metadata:
  name: <migration_policy>
spec:
  selectors:
    namespaceSelector:
    1 
    
      hpc-workloads: "True"
      xyz-workloads-type: ""
    virtualMachineInstanceSelector:
    2 
    
      kubevirt.io/environment: "production"
    Copy to Clipboard Toggle word wrap
    1
    2 

  3. $ oc create -f <migration_policy>.yaml
    Copy to Clipboard Toggle word wrap

10.3.
링크 복사

작은 정보

10.3.1.
링크 복사

10.3.1.1.
링크 복사

참고

10.3.1.2.
링크 복사

  1. apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: <migration_name>
spec:
  vmiName: <vm_name>
    Copy to Clipboard Toggle word wrap 

  2. $ oc create -f <migration_name>.yaml
    Copy to Clipboard Toggle word wrap 

  • $ oc describe vmi <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

    # ...
Status:
  Conditions:
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Status:                True
    Type:                  LiveMigratable
  Migration Method:  LiveMigration
  Migration State:
    Completed:                    true
    End Timestamp:                2018-12-24T06:19:42Z
    Migration UID:                d78c8962-0743-11e9-a540-fa163e0c69f1
    Source Node:                  node2.example.com
    Start Timestamp:              2018-12-24T06:19:35Z
    Target Node:                  node1.example.com
    Target Node Address:          10.9.0.18:43891
    Target Node Domain Detected:  true
    Copy to Clipboard Toggle word wrap

10.3.2.
링크 복사

10.3.2.1.
링크 복사

10.3.2.2.
링크 복사

  • $ oc delete vmim migration-job
    Copy to Clipboard Toggle word wrap

11장.
링크 복사

11.1.
링크 복사

참고

중요

참고

11.1.1.
링크 복사

중요

Expand
표 11.1.
    

11.1.1.1.
링크 복사

중요

  1. $ oc edit vm <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: <vm_name>
spec:
  template:
    spec:
      evictionStrategy: LiveMigrateIfPossible
    1 
    
# ...
    Copy to Clipboard Toggle word wrap

    1 

  2. $ virtctl restart <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap
11.1.1.2.
링크 복사

  1. $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
    Copy to Clipboard Toggle word wrap 

  2. apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  evictionStrategy: LiveMigrate
# ...
    Copy to Clipboard Toggle word wrap

11.1.2.
링크 복사

중요

11.1.2.1.
링크 복사

Expand
표 11.2.
    

참고

11.1.2.2.
링크 복사

중요

  • $ oc edit vm <vm_name> -n <namespace>
    Copy to Clipboard Toggle word wrap 

    apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  runStrategy: Always
# ...
    Copy to Clipboard Toggle word wrap

11.1.3.
링크 복사

11.2.
링크 복사

11.2.1.
링크 복사

예 11.1. 

"486"
Conroe
athlon
core2duo
coreduo
kvm32
kvm64
n270
pentium
pentium2
pentium3
pentiumpro
phenom
qemu32
qemu64
Copy to Clipboard Toggle word wrap

11.2.2.
링크 복사

  • 예 11.2. 

    apic
clflush
cmov
cx16
cx8
de
fpu
fxsr
lahf_lm
lm
mca
mce
mmx
msr
mtrr
nx
pae
pat
pge
pni
pse
pse36
sep
sse
sse2
sse4.1
ssse3
syscall
tsc
    Copy to Clipboard Toggle word wrap

    예 11.3. 

    aes
apic
avx
avx2
bmi1
bmi2
clflush
cmov
cx16
cx8
de
erms
fma
fpu
fsgsbase
fxsr
hle
invpcid
lahf_lm
lm
mca
mce
mmx
movbe
msr
mtrr
nx
pae
pat
pcid
pclmuldq
pge
pni
popcnt
pse
pse36
rdtscp
rtm
sep
smep
sse
sse2
sse4.1
sse4.2
ssse3
syscall
tsc
tsc-deadline
x2apic
xsave
    Copy to Clipboard Toggle word wrap

  • 예 11.4. 

    aes
avx
avx2
bmi1
bmi2
erms
fma
fsgsbase
hle
invpcid
movbe
pcid
pclmuldq
popcnt
rdtscp
rtm
sse4.2
tsc-deadline
x2apic
xsave
    Copy to Clipboard Toggle word wrap

11.2.3.
링크 복사

  • apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  obsoleteCPUs:
    cpuModels:
    1 
    
      - "<obsolete_cpu_1>"
      - "<obsolete_cpu_2>"
    minCPUModel: "<minimum_cpu_model>"
    2
    Copy to Clipboard Toggle word wrap
    1
    2

11.3.
링크 복사

11.3.1.
링크 복사

  • $ oc annotate node <node_name> node-labeller.kubevirt.io/skip-node=true
    1
    Copy to Clipboard Toggle word wrap