Chapter 8. Virtual machines

Procedure

1. Click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Click Create and select Virtual Machine With YAML.

4. Write or paste your virtual machine configuration in the editable window.

   1. Alternatively, use the example virtual machine provided by default in the YAML screen.
5. Optional: Click Download to download the YAML configuration file in its present state.
6. Click Create to create the virtual machine.

Procedure

1. In the OpenShift Virtualization console, click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Select a virtual machine to open the Virtual Machine Overview page.
4. Click Console. The VNC console opens by default.
5. Select Disconnect before switching to ensure that only one console session is open at a time. Otherwise, the VNC console session remains active in the background.
6. Click the VNC Console drop-down list and select Serial Console.
7. Click Disconnect to end the console session.
8. Optional: Open the serial console in a separate window by clicking Open Console in New Window.

Procedure

1. In the OpenShift Virtualization console, click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Select a virtual machine to open the Virtual Machine Overview page.
4. Click the Console tab. The VNC console opens by default.
5. Optional: Open the VNC console in a separate window by clicking Open Console in New Window.
6. Optional: Send key combinations to the virtual machine by clicking Send Key.
7. Click outside the console window and then click Disconnect to end the session.

Procedure

1. In the OpenShift Virtualization console, click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Select a Windows virtual machine to open the Virtual Machine Overview screen.
4. Click the Console tab.
5. In the Console list, select Desktop Viewer.
6. In the Network Interface list, select the layer-2 NIC.
7. Click Launch Remote Desktop to download the console.rdp file.

8. Open an RDP client and reference the console.rdp file. For example, using remmina:

   $ remmina --connect /path/to/console.rdp

9. Enter the Administrator user name and password to connect to the Windows virtual machine.

Procedure

1. In the OpenShift Container Platform console, click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Select a virtual machine to open the Virtual Machine Overview page.
4. From the Actions list, select Copy SSH Command. You can now paste this command onto the OpenShift CLI (oc).

Procedure

1. Run the following command to create the fedora-vm-ssh service:

   $ virtctl expose vm <fedora-vm> --port=22 --name=fedora-vm-ssh --type=NodePort 1

   1

   <fedora-vm> is the name of the VM that you run the fedora-vm-ssh service on.

2. Check the service to find out which port the service acquired:

   $ oc get svc

   Example output

   NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
   fedora-vm-ssh   NodePort   127.0.0.1      <none>        22:32551/TCP   6s

   In this example, the service acquired the 32551 port.

3. Log in to the VMI via SSH. Use the ipAddress of any of the cluster nodes and the port that you found in the previous step:

   $ ssh username@<node_IP_address> -p 32551

Procedure

1. In the YAML file for the VM, add the label and a value for exposing the service for SSH connections. Enable the masquerade feature for the interface:

   Example VirtualMachine definition

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     namespace: ssh-ns 1
     name: vm-ssh
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm-ssh
           special: vm-ssh 2
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: containerdisk
             - disk:
                 bus: virtio
               name: cloudinitdisk
             interfaces:
             - masquerade: {} 3
               name: testmasquerade 4
             rng: {}
           machine:
             type: ""
           resources:
             requests:
               memory: 1024M
         networks:
         - name: testmasquerade
           pod: {}
         volumes:
         - name: containerdisk
           containerDisk:
             image: kubevirt/fedora-cloud-container-disk-demo
         - name: cloudinitdisk
           cloudInitNoCloud:
             userData: |
               #cloud-config
               user: fedora
               password: fedora
               chpasswd: {expire: False}
   # ...

   1

   Name of the namespace created by the oc create namespace command.

   2

   Label used by the service to identify the virtual machine instances that are enabled for SSH traffic connections. The label can be any key:value pair that is added as a label to this YAML file and as a selector in the service YAML file.

   3

   The interface type is masquerade.

   4

   The name of this interface is testmasquerade.

2. Create the VM:

   $ oc create -f <path_for_the_VM_YAML_file>

3. Start the VM:

   $ virtctl start vm-ssh

4. In the YAML file for the service, specify the service name, port number, and the target port.

   Example Service definition

   apiVersion: v1
   kind: Service
   metadata:
     name: svc-ssh 1
     namespace: ssh-ns 2
   spec:
     ports:
     - targetPort: 22 3
       protocol: TCP
       port: 27017
     selector:
       special: vm-ssh 4
     type: NodePort
   # ...

   1

   Name of the SSH service.

   2

   Name of the namespace created by the oc create namespace command.

   3

   The target port number for the SSH connection.

   4

   The selector name and value must match the label specified in the YAML file for the VM.

5. Create the service:

   $ oc create -f <path_for_the_service_YAML_file>

6. Verify that the VM is running:

   $ oc get vmi

   Example output

   NAME    AGE     PHASE       IP              NODENAME
   vm-ssh 6s       Running     10.244.196.152  node01

7. Check the service to find out which port the service acquired:

   $ oc get svc

   Example output

   NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
   svc-ssh     NodePort       10.106.236.208 <none>        27017:30093/TCP   22s

   In this example, the service acquired the port number 30093.

8. Run the following command to obtain the IP address for the node:

   $ oc get node <node_name> -o wide

   Example output

   NAME    STATUS   ROLES   AGE    VERSION  INTERNAL-IP      EXTERNAL-IP
   node01  Ready    worker  6d22h  v1.22.1  192.168.55.101   <none>

9. Log in to the VM via SSH by specifying the IP address of the node where the VM is running and the port number. Use the port number displayed by the oc get svc command and the IP address of the node displayed by the oc get node command. The following example shows the ssh command with the username, node’s IP address, and the port number:

   $ ssh fedora@192.168.55.101 -p 30093

Procedure

1. Connect to the graphical interface with the virtctl utility:

   $ virtctl vnc <VMI>

2. If the command failed, try using the -v flag to collect troubleshooting information:

   $ virtctl vnc <VMI> -v 4

Procedure

1. Log in to the OpenShift Virtualization cluster through the oc CLI tool as a user with an access token.

   $ oc login -u <user> https://<cluster.example.com>:8443

2. Use oc describe vmi to display the configuration of the running Windows virtual machine.

   $ oc describe vmi <windows-vmi-name>

   Example output

   ...
   spec:
     networks:
     - name: default
       pod: {}
     - multus:
         networkName: cnv-bridge
       name: bridge-net
   ...
   status:
     interfaces:
     - interfaceName: eth0
       ipAddress: 198.51.100.0/24
       ipAddresses:
         198.51.100.0/24
       mac: a0:36:9f:0f:b1:70
       name: default
     - interfaceName: eth1
       ipAddress: 192.0.2.0/24
       ipAddresses:
         192.0.2.0/24
         2001:db8::/32
       mac: 00:17:a4:77:77:25
       name: bridge-net
   ...

3. Identify and copy the IP address of the layer 2 network interface. This is 192.0.2.0 in the above example, or 2001:db8:: if you prefer IPv6.
4. Open an RDP client and use the IP address copied in the previous step for the connection.
5. Enter the Administrator user name and password to connect to the Windows virtual machine.

Procedure

1. Start the virtual machine and connect to a graphical console.
2. Log in to a Windows user session.

3. Open Device Manager and expand Other devices to list any Unknown device.

   1. Open the Device Properties to identify the unknown device. Right-click the device and select Properties.
   2. Click the Details tab and select Hardware Ids in the Property list.
   3. Compare the Value for the Hardware Ids with the supported VirtIO drivers.
4. Right-click the device and select Update Driver Software.
5. Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
6. Click Next to install the driver.
7. Repeat this process for all the necessary VirtIO drivers.
8. After the driver installs, click Close to close the window.
9. Reboot the virtual machine to complete the driver installation.

Procedure

1. Start the virtual machine and connect to a graphical console.
2. Begin the Windows installation process.
3. Select the Advanced installation.
4. The storage destination will not be recognized until the driver is loaded. Click Load driver.
5. The drivers are attached as a SATA CD drive. Click OK and browse the CD drive for the storage driver to load. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
6. Repeat the previous two steps for all required drivers.
7. Complete the Windows installation.

Procedure

1. Set limits for a VM by editing the VirtualMachine manifest. For example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: with-limits
   spec:
     running: false
     template:
       spec:
         domain:
   # ...
           resources:
             requests:
               memory: 128Mi
             limits:
               memory: 256Mi  1

   1

   This configuration is supported because the limits.memory value is at least 100Mi larger than the requests.memory value.

2. Save the VirtualMachine manifest.

Procedure

1. Open the HyperConverged CR by running the following command:

   $ oc edit hco -n openshift-cnv kubevirt-hyperconverged

2. Edit the spec.certConfig fields as shown in the following example. To avoid overloading the system, ensure that all values are greater than or equal to 10 minutes. Express all values as strings that comply with the golang ParseDuration format.

   apiVersion: hco.kubevirt.io/v1beta1
   kind: HyperConverged
   metadata:
    name: kubevirt-hyperconverged
    namespace: openshift-cnv
   spec:
     certConfig:
       ca:
         duration: 48h0m0s
         renewBefore: 24h0m0s 1
       server:
         duration: 24h0m0s  2
         renewBefore: 12h0m0s  3

   1

   The value of ca.renewBefore must be less than or equal to the value of ca.duration.

   2

   The value of server.duration must be less than or equal to the value of ca.duration.

   3

   The value of server.renewBefore must be less than or equal to the value of server.duration.

3. Apply the YAML file to your cluster.

Procedure

1. Edit an Ansible Playbook YAML file so that it includes the kubevirt_vm task:

     kubevirt_vm:
       namespace:
       name:
       cpu_cores:
       memory:
       disks:
         - name:
           volume:
             containerDisk:
               image:
           disk:
             bus:

   Note

   This snippet only includes the kubevirt_vm portion of the playbook.

2. Edit the values to reflect the virtual machine you want to create, including the namespace, the number of cpu_cores, the memory, and the disks. For example:

     kubevirt_vm:
       namespace: default
       name: vm1
       cpu_cores: 1
       memory: 64Mi
       disks:
         - name: containerdisk
           volume:
             containerDisk:
               image: kubevirt/cirros-container-disk-demo:latest
           disk:
             bus: virtio

3. If you want the virtual machine to boot immediately after creation, add state: running to the YAML file. For example:

     kubevirt_vm:
       namespace: default
       name: vm1
       state: running 1
       cpu_cores: 1

   1

   Changing this value to state: absent deletes the virtual machine, if it already exists.

4. Run the ansible-playbook command, using your playbook’s file name as the only argument:

   $ ansible-playbook create-vm.yaml

5. Review the output to determine if the play was successful:

   Example output

   (...)
   TASK [Create my first VM] ************************************************************************
   changed: [localhost]
   
   PLAY RECAP ********************************************************************************************************
   localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

6. If you did not include state: running in your playbook file and you want to boot the VM now, edit the file so that it includes state: running and run the playbook again:

   $ ansible-playbook create-vm.yaml

Procedure

1. Create a YAML file that defines a VM object. Use the firmware stanza of the example YAML file:

   Booting in EFI mode with secure boot active

   apiversion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       special: vm-secureboot
     name: vm-secureboot
   spec:
     template:
       metadata:
         labels:
           special: vm-secureboot
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: containerdisk
           features:
             acpi: {}
             smm:
               enabled: true 1
           firmware:
             bootloader:
               efi:
                 secureBoot: true 2
   #...

   1

   OpenShift Virtualization requires System Management Mode (SMM) to be enabled for Secure Boot in EFI mode to occur.

   2

   OpenShift Virtualization supports a VM with or without Secure Boot when using EFI mode. If Secure Boot is enabled, then EFI mode is required. However, EFI mode can be enabled without using Secure Boot.

2. Apply the manifest to your cluster by running the following command:

   $ oc create -f <file_name>.yaml

Procedure

1. Configure a PXE network on the cluster:

   1. Create the network attachment definition file for PXE network pxe-net-conf:

      apiVersion: "k8s.cni.cncf.io/v1"
      kind: NetworkAttachmentDefinition
      metadata:
        name: pxe-net-conf
      spec:
        config: '{
          "cniVersion": "0.3.1",
          "name": "pxe-net-conf",
          "plugins": [
            {
              "type": "cnv-bridge",
              "bridge": "br1",
              "vlan": 1 1
            },
            {
              "type": "cnv-tuning" 2
            }
          ]
        }'

      1

      Optional: The VLAN tag.

      2

      The cnv-tuning plugin provides support for custom MAC addresses.

      Note

      The virtual machine instance will be attached to the bridge br1 through an access port with the requested VLAN.

2. Create the network attachment definition by using the file you created in the previous step:

   $ oc create -f pxe-net-conf.yaml

3. Edit the virtual machine instance configuration file to include the details of the interface and network.

   1. Specify the network and MAC address, if required by the PXE server. If the MAC address is not specified, a value is assigned automatically.

      Ensure that bootOrder is set to 1 so that the interface boots first. In this example, the interface is connected to a network called <pxe-net>:

      interfaces:
      - masquerade: {}
        name: default
      - bridge: {}
        name: pxe-net
        macAddress: de:00:00:00:00:de
        bootOrder: 1

      Note

      Boot order is global for interfaces and disks.

   2. Assign a boot device number to the disk to ensure proper booting after operating system provisioning.

      Set the disk bootOrder value to 2:

      devices:
        disks:
        - disk:
            bus: virtio
          name: containerdisk
          bootOrder: 2

   3. Specify that the network is connected to the previously created network attachment definition. In this scenario, <pxe-net> is connected to the network attachment definition called <pxe-net-conf>:

      networks:
      - name: default
        pod: {}
      - name: pxe-net
        multus:
          networkName: pxe-net-conf

4. Create the virtual machine instance:

   $ oc create -f vmi-pxe-boot.yaml

1. Wait for the virtual machine instance to run:

   $ oc get vmi vmi-pxe-boot -o yaml | grep -i phase
     phase: Running

2. View the virtual machine instance using VNC:

   $ virtctl vnc vmi-pxe-boot

3. Watch the boot screen to verify that the PXE boot is successful.

4. Log in to the virtual machine instance:

   $ virtctl console vmi-pxe-boot

5. Verify the interfaces and MAC address on the virtual machine and that the interface connected to the bridge has the specified MAC address. In this case, we used eth1 for the PXE boot, without an IP address. The other interface, eth0, got an IP address from OpenShift Container Platform.

   $ ip addr

Procedure

1. To explicitly tell the virtual machine instance that it has more memory available than was requested from the cluster, edit the virtual machine configuration file and set spec.domain.memory.guest to a higher value than spec.domain.resources.requests.memory. This process is called memory overcommitment.

   In this example, 1024M is requested from the cluster, but the virtual machine instance is told that it has 2048M available. As long as there is enough free memory available on the node, the virtual machine instance will consume up to 2048M.

   kind: VirtualMachine
   spec:
     template:
       domain:
       resources:
           requests:
             memory: 1024M
       memory:
           guest: 2048M

   Note

   The same eviction rules as those for pods apply to the virtual machine instance if the node is under memory pressure.

2. Create the virtual machine:

   $ oc create -f <file_name>.yaml

Procedure

1. To disable guest memory overhead accounting, edit the YAML configuration file and set the overcommitGuestOverhead value to true. This parameter is disabled by default.

   kind: VirtualMachine
   spec:
     template:
       domain:
       resources:
           overcommitGuestOverhead: true
           requests:
             memory: 1024M

   Note

   If overcommitGuestOverhead is enabled, it adds the guest overhead to memory limits, if present.

2. Create the virtual machine:

   $ oc create -f <file_name>.yaml

Procedure

1. In your virtual machine configuration, add the resources.requests.memory and memory.hugepages.pageSize parameters to the spec.domain. The following configuration snippet is for a virtual machine that requests a total of 4Gi memory with a page size of 1Gi:

   kind: VirtualMachine
   ...
   spec:
     domain:
       resources:
         requests:
           memory: "4Gi" 1
       memory:
         hugepages:
           pageSize: "1Gi" 2
   ...

   1

   The total amount of memory requested for the virtual machine. This value must be divisible by the page size.

   2

   The size of each huge page. Valid values for x86_64 architecture are 1Gi and 2Mi. The page size must be smaller than the requested memory.

2. Apply the virtual machine configuration:

   $ oc apply -f <virtual_machine>.yaml

Procedure

1. Click Workloads Virtual Machines from the side menu.
2. Select a virtual machine to open the Virtual Machine tab.
3. Click the Details tab.
4. Click the pencil icon to the right of the Dedicated Resources field to open the Dedicated Resources window.
5. Select Schedule this workload with dedicated resources (guaranteed policy).
6. Click Save.

Procedure

1. Create a YAML file with the following contents:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm2-rhel84-watchdog
     name: <vm-name>
   spec:
     running: false
     template:
       metadata:
        labels:
           kubevirt.io/vm: vm2-rhel84-watchdog
       spec:
         domain:
           devices:
             watchdog:
               name: <watchdog>
               i6300esb:
                 action: "poweroff" 1
   ...

   1

   Specify the watchdog action (poweroff, reset, or shutdown).

   The example above configures the i6300esb watchdog device on a RHEL8 VM with the poweroff action and exposes the device as /dev/watchdog.

   This device can now be used by the watchdog binary.

2. Apply the YAML file to your cluster by running the following command:

   $ oc apply -f <file_name>.yaml

1. Run the following command to verify that the VM is connected to the watchdog device:

   $ lspci | grep watchdog -i

2. Run one of the following commands to confirm the watchdog is active:

   • Trigger a kernel panic:

     # echo c > /proc/sysrq-trigger

   • Terminate the watchdog service:

     # pkill -9 watchdog

Procedure

1. As a root user, install the watchdog package and dependencies:

   # yum install watchdog

2. Uncomment the following line in the /etc/watchdog.conf file, and save the changes:

   #watchdog-device = /dev/watchdog

3. Enable the watchdog service to start on boot:

   # systemctl enable --now watchdog.service

Procedure

1. Ensure you are in the correct namespace. The config map can only be referenced by data volumes if it is in the same namespace.

   $ oc get ns

2. Create the config map:

   $ oc create configmap <configmap-name> --from-file=</path/to/file/ca.pem>

Procedure

1. If your data source requires authentication, create a Secret manifest, specifying the data source credentials, and save it as endpoint-secret.yaml:

   apiVersion: v1
   kind: Secret
   metadata:
     name: endpoint-secret 1
     labels:
       app: containerized-data-importer
   type: Opaque
   data:
     accessKeyId: "" 2
     secretKey:   "" 3

   1

   Specify the name of the Secret.

   2

   Specify the Base64-encoded key ID or user name.

   3

   Specify the Base64-encoded secret key or password.

2. Apply the Secret manifest:

   $ oc apply -f endpoint-secret.yaml

3. Edit the VirtualMachine manifest, specifying the data source for the virtual machine image you want to import, and save it as vm-fedora-datavolume.yaml:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     creationTimestamp: null
     labels:
       kubevirt.io/vm: vm-fedora-datavolume
     name: vm-fedora-datavolume 1
   spec:
     dataVolumeTemplates:
     - metadata:
         creationTimestamp: null
         name: fedora-dv 2
       spec:
         storage:
           resources:
             requests:
               storage: 10Gi
           storageClassName: local
         source:
           http: 3
             url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 4
             secretRef: endpoint-secret 5
             certConfigMap: "" 6
       status: {}
     running: true
     template:
       metadata:
         creationTimestamp: null
         labels:
           kubevirt.io/vm: vm-fedora-datavolume
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: datavolumedisk1
           machine:
             type: ""
           resources:
             requests:
               memory: 1.5Gi
         terminationGracePeriodSeconds: 180
         volumes:
         - dataVolume:
             name: fedora-dv
           name: datavolumedisk1
   status: {}

   1

   Specify the name of the virtual machine.

   2

   Specify the name of the data volume.

   3

   Specify http for an HTTP or HTTPS endpoint. Specify registry for a container disk image imported from a registry.

   4

   The source of the virtual machine image you want to import. This example references a virtual machine image at an HTTPS endpoint. An example of a container registry endpoint is url: "docker://kubevirt/fedora-cloud-container-disk-demo:latest".

   5

   Required if you created a Secret for the data source.

   6

   Optional: Specify a CA certificate config map.

4. Create the virtual machine:

   $ oc create -f vm-fedora-datavolume.yaml

   Note

   The oc create command creates the data volume and the virtual machine. The CDI controller creates an underlying PVC with the correct annotation and the import process begins. When the import is complete, the data volume status changes to Succeeded. You can start the virtual machine.

   Data volume provisioning happens in the background, so there is no need to monitor the process.

Verification

1. The importer pod downloads the virtual machine image or container disk from the specified URL and stores it on the provisioned PV. View the status of the importer pod by running the following command:

   $ oc get pods

2. Monitor the data volume until its status is Succeeded by running the following command:

   $ oc describe dv fedora-dv 1

   1

   Specify the data volume name that you defined in the VirtualMachine manifest.

3. Verify that provisioning is complete and that the virtual machine has started by accessing its serial console:

   $ virtctl console vm-fedora-datavolume

Procedure

1. Log in as root to the node on which to create the local PV. This procedure uses node01 for its examples.

2. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file loop10 with a size of 2Gb (20 100Mb blocks):

   $ dd if=/dev/zero of=<loop10> bs=100M count=20

3. Mount the loop10 file as a loop device.

   $ losetup </dev/loop10>d3 <loop10> 1 2

   1

   File path where the loop device is mounted.

   2

   The file created in the previous step to be mounted as the loop device.

4. Create a PersistentVolume manifest that references the mounted loop device.

   kind: PersistentVolume
   apiVersion: v1
   metadata:
     name: <local-block-pv10>
     annotations:
   spec:
     local:
       path: </dev/loop10> 1
     capacity:
       storage: <2Gi>
     volumeMode: Block 2
     storageClassName: local 3
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - <node01> 4

   1

   The path of the loop device on the node.

   2

   Specifies it is a block PV.

   3

   Optional: Set a storage class for the PV. If you omit it, the cluster default is used.

   4

   The node on which the block device was mounted.

5. Create the block PV.

   # oc create -f <local-block-pv10.yaml>1

   1

   The file name of the persistent volume created in the previous step.

Procedure

1. If your data source requires authentication, create a Secret manifest, specifying the data source credentials, and save it as endpoint-secret.yaml:

   apiVersion: v1
   kind: Secret
   metadata:
     name: endpoint-secret 1
     labels:
       app: containerized-data-importer
   type: Opaque
   data:
     accessKeyId: "" 2
     secretKey:   "" 3

   1

   Specify the name of the Secret.

   2

   Specify the Base64-encoded key ID or user name.

   3

   Specify the Base64-encoded secret key or password.

2. Apply the Secret manifest:

   $ oc apply -f endpoint-secret.yaml

3. Create a DataVolume manifest, specifying the data source for the virtual machine image and Block for storage.volumeMode.

   apiVersion: cdi.kubevirt.io/v1beta1
   kind: DataVolume
   metadata:
     name: import-pv-datavolume 1
   spec:
     storageClassName: local 2
       source:
         http:
           url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 3
           secretRef: endpoint-secret 4
     storage:
       volumeMode: Block 5
       resources:
         requests:
           storage: 10Gi

   1

   Specify the name of the data volume.

   2

   Optional: Set the storage class or omit it to accept the cluster default.

   3

   Specify the HTTP or HTTPS URL of the image to import.

   4

   Required if you created a Secret for the data source.

   5

   The volume mode and access mode are detected automatically for known storage provisioners. Otherwise, specify Block.

4. Create the data volume to import the virtual machine image:

   $ oc create -f import-pv-datavolume.yaml

Procedure

1. Create a ClusterRole manifest:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: <datavolume-cloner> 1
   rules:
   - apiGroups: ["cdi.kubevirt.io"]
     resources: ["datavolumes/source"]
     verbs: ["*"]

   1

   Unique name for the cluster role.

2. Create the cluster role in the cluster:

   $ oc create -f <datavolume-cloner.yaml> 1

   1

   The file name of the ClusterRole manifest created in the previous step.

3. Create a RoleBinding manifest that applies to both the source and destination namespaces and references the cluster role created in the previous step.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: <allow-clone-to-user> 1
     namespace: <Source namespace> 2
   subjects:
   - kind: ServiceAccount
     name: default
     namespace: <Destination namespace> 3
   roleRef:
     kind: ClusterRole
     name: datavolume-cloner 4
     apiGroup: rbac.authorization.k8s.io

   1

   Unique name for the role binding.

   2

   The namespace for the source data volume.

   3

   The namespace to which the data volume is cloned.

   4

   The name of the cluster role created in the previous step.

4. Create the role binding in the cluster:

   $ oc create -f <datavolume-cloner.yaml> 1

   1

   The file name of the RoleBinding manifest created in the previous step.

Procedure

1. Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC.

2. Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC, and the size of the new data volume.

   For example:

   apiVersion: cdi.kubevirt.io/v1beta1
   kind: DataVolume
   metadata:
     name: <cloner-datavolume> 1
   spec:
     source:
       pvc:
         namespace: "<source-namespace>" 2
         name: "<my-favorite-vm-disk>" 3
     pvc:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
           storage: <2Gi> 4

   1

   The name of the new data volume.

   2

   The namespace where the source PVC exists.

   3

   The name of the source PVC.

   4

   The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC.

3. Start cloning the PVC by creating the data volume:

   $ oc create -f <cloner-datavolume>.yaml

   Note

   Data volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones.

Procedure

1. Examine the virtual machine you want to clone to identify the name and namespace of the associated PVC.

2. Create a YAML file for a VirtualMachine object. The following virtual machine example clones my-favorite-vm-disk, which is located in the source-namespace namespace. The 2Gi data volume called favorite-clone is created from my-favorite-vm-disk.

   For example:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     labels:
       kubevirt.io/vm: vm-dv-clone
     name: vm-dv-clone 1
   spec:
     running: false
     template:
       metadata:
         labels:
           kubevirt.io/vm: vm-dv-clone
       spec:
         domain:
           devices:
             disks:
             - disk:
                 bus: virtio
               name: root-disk
           resources:
             requests:
               memory: 64M
         volumes:
         - dataVolume:
             name: favorite-clone
           name: root-disk
     dataVolumeTemplates:
     - metadata:
         name: favorite-clone
       spec:
         storage:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 2Gi
         source:
           pvc:
             namespace: "source-namespace"
             name: "my-favorite-vm-disk"

   1

   The virtual machine to create.

3. Create the virtual machine with the PVC-cloned data volume:

   $ oc create -f <vm-clone-datavolumetemplate>.yaml

Procedure

1. Log in as root to the node on which to create the local PV. This procedure uses node01 for its examples.

2. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file loop10 with a size of 2Gb (20 100Mb blocks):

   $ dd if=/dev/zero of=<loop10> bs=100M count=20

3. Mount the loop10 file as a loop device.

   $ losetup </dev/loop10>d3 <loop10> 1 2

   1

   File path where the loop device is mounted.

   2

   The file created in the previous step to be mounted as the loop device.

4. Create a PersistentVolume manifest that references the mounted loop device.

   kind: PersistentVolume
   apiVersion: v1
   metadata:
     name: <local-block-pv10>
     annotations:
   spec:
     local:
       path: </dev/loop10> 1
     capacity:
       storage: <2Gi>
     volumeMode: Block 2
     storageClassName: local 3
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - <node01> 4

   1

   The path of the loop device on the node.

   2

   Specifies it is a block PV.

   3

   Optional: Set a storage class for the PV. If you omit it, the cluster default is used.

   4

   The node on which the block device was mounted.

5. Create the block PV.

   # oc create -f <local-block-pv10.yaml>1

   1

   The file name of the persistent volume created in the previous step.

Procedure

1. Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC.

2. Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC, volumeMode: Block so that an available block PV is used, and the size of the new data volume.

   For example:

   apiVersion: cdi.kubevirt.io/v1beta1
   kind: DataVolume
   metadata:
     name: <cloner-datavolume> 1
   spec:
     source:
       pvc:
         namespace: "<source-namespace>" 2
         name: "<my-favorite-vm-disk>" 3
     pvc:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
           storage: <2Gi> 4
       volumeMode: Block 5

   1

   The name of the new data volume.

   2

   The namespace where the source PVC exists.

   3

   The name of the source PVC.

   4

   The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC.

   5

   Specifies that the destination is a block PV

3. Start cloning the PVC by creating the data volume:

   $ oc create -f <cloner-datavolume>.yaml

   Note

   Data volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones.

Procedure

1. Edit the interfaces spec of your virtual machine configuration file:

   kind: VirtualMachine
   spec:
     domain:
       devices:
         interfaces:
           - name: default
             masquerade: {} 1
             ports: 2
               - port: 80
     networks:
     - name: default
       pod: {}

   1

   Connect using masquerade mode.

   2

   Optional: List the ports that you want to expose from the virtual machine, each specified by the port field. The port value must be a number between 0 and 65536. When the ports array is not used, all ports in the valid range are open to incoming traffic. In this example, incoming traffic is allowed on port 80.

   Note

   Ports 49152 and 49153 are reserved for use by the libvirt platform and all other incoming traffic to these ports is dropped.

2. Create the virtual machine:

   $ oc create -f <vm-name>.yaml

Procedure

1. In a new virtual machine configuration, include an interface with masquerade and configure the IPv6 address and default gateway by using cloud-init.

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: example-vm-ipv6
   ...
             interfaces:
               - name: default
                 masquerade: {} 1
                 ports:
                   - port: 80 2
         networks:
         - name: default
           pod: {}
         volumes:
         - cloudInitNoCloud:
             networkData: |
               version: 2
               ethernets:
                 eth0:
                   dhcp4: true
                   addresses: [ fd10:0:2::2/120 ] 3
                   gateway6: fd10:0:2::1 4

   1

   Connect using masquerade mode.

   2

   Allows incoming traffic on port 80 to the virtual machine.

   3

   The static IPv6 address as determined by the Network.pod.vmIPv6NetworkCIDR field in the virtual machine instance configuration. The default value is fd10:0:2::2/120.

   4

   The gateway IP address as determined by the Network.pod.vmIPv6NetworkCIDR field in the virtual machine instance configuration. The default value is fd10:0:2::1.

2. Create the virtual machine in the namespace:

   $ oc create -f example-vm-ipv6.yaml

Procedure

1. Edit the VirtualMachine manifest to add the label for service creation:

   apiVersion: kubevirt.io/v1
   kind: VirtualMachine
   metadata:
     name: vm-ephemeral
     namespace: example-namespace
   spec:
     running: false
     template:
       metadata:
         labels:
           special: key 1
   # ...

   1

   Add the label special: key in the spec.template.metadata.labels section.

   Note

   Labels on a virtual machine are passed through to the pod. The special: key label must match the label in the spec.selector attribute of the Service manifest.

2. Save the VirtualMachine manifest file to apply your changes.

3. Create a Service manifest to expose the VM:

   apiVersion: v1
   kind: Service
   metadata:
     name: vmservice 1
     namespace: example-namespace 2
   spec:
     externalTrafficPolicy: Cluster 3
     ports:
     - nodePort: 30000 4
       port: 27017
       protocol: TCP
       targetPort: 22 5
     selector:
       special: key 6
     type: NodePort 7

   1

   The name of the Service object.

   2

   The namespace where the Service object resides. This must match the metadata.namespace field of the VirtualMachine manifest.

   3

   Optional: Specifies how the nodes distribute service traffic that is received on external IP addresses. This only applies to NodePort and LoadBalancer service types. The default value is Cluster which routes traffic evenly to all cluster endpoints.

   4

   Optional: When set, the nodePort value must be unique across all services. If not specified, a value in the range above 30000 is dynamically allocated.

   5

   Optional: The VM port to be exposed by the service. It must reference an open port if a port list is defined in the VM manifest. If targetPort is not specified, it takes the same value as port.

   6

   The reference to the label that you added in the spec.template.metadata.labels stanza of the VirtualMachine manifest.

   7

   The type of service. Possible values are ClusterIP, NodePort and LoadBalancer.

4. Save the Service manifest file.

5. Create the service by running the following command:

   $ oc create -f <service_name>.yaml

6. Start the VM. If the VM is already running, restart it.

Verification

1. Query the Service object to verify that it is available:

   $ oc get service -n example-namespace

   Example output for ClusterIP service

   NAME        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
   vmservice   ClusterIP   172.30.3.149   <none>        27017/TCP   2m

   Example output for NodePort service

   NAME        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
   vmservice   NodePort    172.30.232.73   <none>       27017:30000/TCP    5m

   Example output for LoadBalancer service

   NAME        TYPE            CLUSTER-IP     EXTERNAL-IP                    PORT(S)           AGE
   vmservice   LoadBalancer    172.30.27.5   172.29.10.235,172.29.10.235     27017:31829/TCP   5s

2. Choose the appropriate method to connect to the virtual machine:

   • For a ClusterIP service, connect to the VM from within the cluster by using the service IP address and the service port. For example:

     $ ssh fedora@172.30.3.149 -p 27017

   • For a NodePort service, connect to the VM by specifying the node IP address and the node port outside the cluster network. For example:

     $ ssh fedora@$NODE_IP -p 30000

   • For a LoadBalancer service, use the vinagre client to connect to your virtual machine by using the public IP address and port. External ports are dynamically allocated.

Procedure

1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: <name> 1
     namespace: openshift-sriov-network-operator 2
   spec:
     resourceName: <sriov_resource_name> 3
     nodeSelector:
       feature.node.kubernetes.io/network-sriov.capable: "true" 4
     priority: <priority> 5
     mtu: <mtu> 6
     numVfs: <num> 7
     nicSelector: 8
       vendor: "<vendor_code>" 9
       deviceID: "<device_id>" 10
       pfNames: ["<pf_name>", ...] 11
       rootDevices: ["<pci_bus_id>", "..."] 12
     deviceType: vfio-pci 13
     isRdma: false 14

   1

   Specify a name for the CR object.

   2

   Specify the namespace where the SR-IOV Operator is installed.

   3

   Specify the resource name of the SR-IOV device plugin. You can create multiple SriovNetworkNodePolicy objects for a resource name.

   4

   Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed only on selected nodes.

   5

   Optional: Specify an integer value between 0 and 99. A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99. The default value is 99.

   6

   Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.

   7

   Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 128.

   8

   The nicSelector mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device.

   9

   Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3.

   10

   Optional: Specify the device hex code of SR-IOV network device. The only allowed values are 158b, 1015, 1017.

   11

   Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device.

   12

   The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1.

   13

   The vfio-pci driver type is required for virtual functions in OpenShift Virtualization.

   14

   Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false. The default value is false.

   Note

   If isRDMA flag is set to true, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.

2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".

3. Create the SriovNetworkNodePolicy object:

   $ oc create -f <name>-sriov-node-network.yaml

   where <name> specifies the name for this configuration.

   After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

   $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

Procedure

1. Create the following SriovNetwork object, and then save the YAML in the <name>-sriov-network.yaml file. Replace <name> with a name for this additional network.

2. To create the object, enter the following command. Replace <name> with a name for this additional network.

   $ oc create -f <name>-sriov-network.yaml

3. Optional: To confirm that the NetworkAttachmentDefinition object associated with the SriovNetwork object that you created in the previous step exists, enter the following command. Replace <namespace> with the namespace you specified in the SriovNetwork object.

   $ oc get net-attach-def -n <namespace>

Procedure

1. Include the SR-IOV network details in the spec.domain.devices.interfaces and spec.networks of the VM configuration:

   kind: VirtualMachine
   ...
   spec:
     domain:
       devices:
         interfaces:
         - name: <default> 1
           masquerade: {} 2
         - name: <nic1> 3
           sriov: {}
     networks:
     - name: <default> 4
       pod: {}
     - name: <nic1> 5
       multus:
           networkName: <sriov-network> 6
   ...

   1

   A unique name for the interface that is connected to the pod network.

   2

   The masquerade binding to the default pod network.

   3

   A unique name for the SR-IOV interface.

   4

   The name of the pod network interface. This must be the same as the interfaces.name that you defined earlier.

   5

   The name of the SR-IOV interface. This must be the same as the interfaces.name that you defined earlier.

   6

   The name of the SR-IOV network attachment definition.

2. Apply the virtual machine configuration:

   $ oc apply -f <vm-sriov.yaml> 1

   1

   The name of the virtual machine YAML file.

Procedure

1. In the OpenShift Virtualization console, click Workloads Virtualization from the side menu.
2. Click the Virtual Machines tab.
3. Select a virtual machine name to open the Virtual Machine Overview screen.

1. PVCs must request a ReadWriteMany access mode.
2. Storage provider must support both Kubernetes and CSI snapshot APIs

Procedure

1. Create the MachineConfig file. For example:

   $ touch machineconfig.yaml

2. Edit the file, ensuring that you include the directory where you want the hostpath provisioner to create PVs. For example:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: 50-set-selinux-for-hostpath-provisioner
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
           - contents: |
               [Unit]
               Description=Set SELinux chcon for hostpath provisioner
               Before=kubelet.service
   
               [Service]
               ExecStart=/usr/bin/chcon -Rt container_file_t <backing_directory_path> 1
   
               [Install]
               WantedBy=multi-user.target
             enabled: true
             name: hostpath-provisioner.service

   1

   Specify the backing directory where you want the provisioner to create PVs. This directory must not be located in the filesystem’s root directory (/).

3. Create the MachineConfig object:

   $ oc create -f machineconfig.yaml -n <namespace>

Procedure

1. Create the HostPathProvisioner custom resource file. For example:

   $ touch hostpathprovisioner_cr.yaml

2. Edit the file, ensuring that the spec.pathConfig.path value is the directory where you want the hostpath provisioner to create PVs. For example:

   apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
   kind: HostPathProvisioner
   metadata:
     name: hostpath-provisioner
   spec:
     imagePullPolicy: IfNotPresent
     pathConfig:
       path: "<backing_directory_path>" 1
       useNamingPrefix: false 2
     workload: 3

   1

   Specify the backing directory where you want the provisioner to create PVs. This directory must not be located in the filesystem’s root directory (/).

   2

   Change this value to true if you want to use the name of the persistent volume claim (PVC) that is bound to the created PV as the prefix of the directory name.

   3

   Optional: You can use the spec.workload field to configure node placement rules for the hostpath provisioner.

   Note

   If you did not create the backing directory, the provisioner attempts to create it for you. If you did not apply the container_file_t SELinux context, this can cause Permission denied errors.

3. Create the custom resource in the openshift-cnv namespace:

   $ oc create -f hostpathprovisioner_cr.yaml -n openshift-cnv

Procedure

1. Create a YAML file for defining the storage class. For example:

   $ touch storageclass.yaml

2. Edit the file. For example:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: hostpath-provisioner 1
   provisioner: kubevirt.io/hostpath-provisioner
   reclaimPolicy: Delete 2
   volumeBindingMode: WaitForFirstConsumer 3

   1

   You can optionally rename the storage class by changing this value.

   2

   The two possible reclaimPolicy values are Delete and Retain. If you do not specify a value, the storage class defaults to Delete.

   3

   The volumeBindingMode value determines when dynamic provisioning and volume binding occur. Specify WaitForFirstConsumer to delay the binding and provisioning of a PV until after a pod that uses the persistent volume claim (PVC) is created. This ensures that the PV meets the pod’s scheduling requirements.

   Note

   Virtual machines use data volumes that are based on local PVs. Local PVs are bound to specific nodes. While the disk image is prepared for consumption by the virtual machine, it is possible that the virtual machine cannot be scheduled to the node where the local storage PV was previously pinned.

   To solve this problem, use the Kubernetes pod scheduler to bind the PVC to a PV on the correct node. By using StorageClass with volumeBindingMode set to WaitForFirstConsumer, the binding and provisioning of the PV is delayed until a Pod is created using the PVC.

3. Create the StorageClass object:

   $ oc create -f storageclass.yaml