Virtualization


OpenShift Container Platform 4.17

OpenShift Virtualization installation, usage, and release notes

Red Hat OpenShift Documentation Team

Abstract

This document provides information about how to use OpenShift Virtualization in OpenShift Container Platform.

Chapter 1. About

1.1. About OpenShift Virtualization

Learn about OpenShift Virtualization’s capabilities and support scope.

1.1.1. What you can do with OpenShift Virtualization

OpenShift Virtualization is an add-on to OpenShift Container Platform that allows you to run and manage virtual machine workloads alongside container workloads.

OpenShift Virtualization adds new objects into your OpenShift Container Platform cluster by using Kubernetes custom resources to enable virtualization tasks. These tasks include:

  • Creating and managing Linux and Windows virtual machines (VMs)
  • Running pod and VM workloads alongside each other in a cluster
  • Connecting to virtual machines through a variety of consoles and CLI tools
  • Importing and cloning existing virtual machines
  • Managing network interface controllers and storage disks attached to virtual machines
  • Live migrating virtual machines between nodes

An enhanced web console provides a graphical portal to manage these virtualized resources alongside the OpenShift Container Platform cluster containers and infrastructure.

OpenShift Virtualization is designed and tested to work well with Red Hat OpenShift Data Foundation features.

Important

When you deploy OpenShift Virtualization with OpenShift Data Foundation, you must create a dedicated storage class for Windows virtual machine disks. See Optimizing ODF PersistentVolumes for Windows VMs for details.

You can use OpenShift Virtualization with OVN-Kubernetes or one of the other certified network plugins listed in Certified OpenShift CNI Plug-ins.

You can check your OpenShift Virtualization cluster for compliance issues by installing the Compliance Operator and running a scan with the ocp4-moderate and ocp4-moderate-node profiles. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies.

1.1.1.1. OpenShift Virtualization supported cluster version

OpenShift Virtualization 4.17 is supported for use on OpenShift Container Platform 4.17 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.

1.1.2. About volume and access modes for virtual machine disks

If you use the storage API with known storage providers, the volume and access modes are selected automatically. However, if you use a storage class that does not have a storage profile, you must configure the volume and access mode.

For best results, use the ReadWriteMany (RWX) access mode and the Block volume mode. This is important for the following reasons:

  • ReadWriteMany (RWX) access mode is required for live migration.
  • The Block volume mode performs significantly better than the Filesystem volume mode. This is because the Filesystem volume mode uses more storage layers, including a file system layer and a disk image file. These layers are not necessary for VM disk storage.

    For example, if you use Red Hat OpenShift Data Foundation, Ceph RBD volumes are preferable to CephFS volumes.

Important

You cannot live migrate virtual machines with the following configurations:

  • Storage volume with ReadWriteOnce (RWO) access mode
  • Passthrough features such as GPUs

Set the evictionStrategy field to None for these virtual machines. The None strategy powers down VMs during node reboots.

1.1.3. Single-node OpenShift differences

You can install OpenShift Virtualization on single-node OpenShift.

However, you should be aware that Single-node OpenShift does not support the following features:

  • High availability
  • Pod disruption
  • Live migration
  • Virtual machines or templates that have an eviction strategy configured

1.1.4. Additional resources

1.2. Security policies

Learn about OpenShift Virtualization security and authorization.

Key points

  • OpenShift Virtualization adheres to the restricted Kubernetes pod security standards profile, which aims to enforce the current best practices for pod security.
  • Virtual machine (VM) workloads run as unprivileged pods.
  • Security context constraints (SCCs) are defined for the kubevirt-controller service account.
  • TLS certificates for OpenShift Virtualization components are renewed and rotated automatically.

1.2.1. About workload security

By default, virtual machine (VM) workloads do not run with root privileges in OpenShift Virtualization, and there are no supported OpenShift Virtualization features that require root privileges.

For each VM, a virt-launcher pod runs an instance of libvirt in session mode to manage the VM process. In session mode, the libvirt daemon runs as a non-root user account and only permits connections from clients that are running under the same user identifier (UID). Therefore, VMs run as unprivileged pods, adhering to the security principle of least privilege.

1.2.2. TLS certificates

TLS certificates for OpenShift Virtualization components are renewed and rotated automatically. You are not required to refresh them manually.

Automatic renewal schedules

TLS certificates are automatically deleted and replaced according to the following schedule:

  • KubeVirt certificates are renewed daily.
  • Containerized Data Importer controller (CDI) certificates are renewed every 15 days.
  • MAC pool certificates are renewed every year.

Automatic TLS certificate rotation does not disrupt any operations. For example, the following operations continue to function without any disruption:

  • Migrations
  • Image uploads
  • VNC and console connections

1.2.3. Authorization

OpenShift Virtualization uses role-based access control (RBAC) to define permissions for human users and service accounts. The permissions defined for service accounts control the actions that OpenShift Virtualization components can perform.

You can also use RBAC roles to manage user access to virtualization features. For example, an administrator can create an RBAC role that provides the permissions required to launch a virtual machine. The administrator can then restrict access by binding the role to specific users.

1.2.3.1. Default cluster roles for OpenShift Virtualization

By using cluster role aggregation, OpenShift Virtualization extends the default OpenShift Container Platform cluster roles to include permissions for accessing virtualization objects.

Table 1.1. OpenShift Virtualization cluster roles
Default cluster roleOpenShift Virtualization cluster roleOpenShift Virtualization cluster role description

view

kubevirt.io:view

A user that can view all OpenShift Virtualization resources in the cluster but cannot create, delete, modify, or access them. For example, the user can see that a virtual machine (VM) is running but cannot shut it down or gain access to its console.

edit

kubevirt.io:edit

A user that can modify all OpenShift Virtualization resources in the cluster. For example, the user can create VMs, access VM consoles, and delete VMs.

admin

kubevirt.io:admin

A user that has full permissions to all OpenShift Virtualization resources, including the ability to delete collections of resources. The user can also view and modify the OpenShift Virtualization runtime configuration, which is located in the HyperConverged custom resource in the openshift-cnv namespace.

1.2.3.2. RBAC roles for storage features in OpenShift Virtualization

The following permissions are granted to the Containerized Data Importer (CDI), including the cdi-operator and cdi-controller service accounts.

1.2.3.2.1. Cluster-wide RBAC roles
Table 1.2. Aggregated cluster roles for the cdi.kubevirt.io API group
CDI cluster roleResourcesVerbs

cdi.kubevirt.io:admin

datavolumes, uploadtokenrequests

* (all)

datavolumes/source

create

cdi.kubevirt.io:edit

datavolumes, uploadtokenrequests

*

datavolumes/source

create

cdi.kubevirt.io:view

cdiconfigs, dataimportcrons, datasources, datavolumes, objecttransfers, storageprofiles, volumeimportsources, volumeuploadsources, volumeclonesources

get, list, watch

datavolumes/source

create

cdi.kubevirt.io:config-reader

cdiconfigs, storageprofiles

get, list, watch

Table 1.3. Cluster-wide roles for the cdi-operator service account
API groupResourcesVerbs

rbac.authorization.k8s.io

clusterrolebindings, clusterroles

get, list, watch, create, update, delete

security.openshift.io

securitycontextconstraints

get, list, watch, update, create

apiextensions.k8s.io

customresourcedefinitions, customresourcedefinitions/status

get, list, watch, create, update, delete

cdi.kubevirt.io

*

*

upload.cdi.kubevirt.io

*

*

admissionregistration.k8s.io

validatingwebhookconfigurations, mutatingwebhookconfigurations

create, list, watch

admissionregistration.k8s.io

validatingwebhookconfigurations

Allow list: cdi-api-dataimportcron-validate, cdi-api-populator-validate, cdi-api-datavolume-validate, cdi-api-validate, objecttransfer-api-validate

get, update, delete

admissionregistration.k8s.io

mutatingwebhookconfigurations

Allow list: cdi-api-datavolume-mutate

get, update, delete

apiregistration.k8s.io

apiservices

get, list, watch, create, update, delete

Table 1.4. Cluster-wide roles for the cdi-controller service account
API groupResourcesVerbs

"" (core)

events

create, patch

"" (core)

persistentvolumeclaims

get, list, watch, create, update, delete, deletecollection, patch

"" (core)

persistentvolumes

get, list, watch, update

"" (core)

persistentvolumeclaims/finalizers, pods/finalizers

update

"" (core)

pods, services

get, list, watch, create, delete

"" (core)

configmaps

get, create

storage.k8s.io

storageclasses, csidrivers

get, list, watch

config.openshift.io

proxies

get, list, watch

cdi.kubevirt.io

*

*

snapshot.storage.k8s.io

volumesnapshots, volumesnapshotclasses, volumesnapshotcontents

get, list, watch, create, delete

snapshot.storage.k8s.io

volumesnapshots

update, deletecollection

apiextensions.k8s.io

customresourcedefinitions

get, list, watch

scheduling.k8s.io

priorityclasses

get, list, watch

image.openshift.io

imagestreams

get, list, watch

"" (core)

secrets

create

kubevirt.io

virtualmachines/finalizers

update

1.2.3.2.2. Namespaced RBAC roles
Table 1.5. Namespaced roles for the cdi-operator service account
API groupResourcesVerbs

rbac.authorization.k8s.io

rolebindings, roles

get, list, watch, create, update, delete

"" (core)

serviceaccounts, configmaps, events, secrets, services

get, list, watch, create, update, patch, delete

apps

deployments, deployments/finalizers

get, list, watch, create, update, delete

route.openshift.io

routes, routes/custom-host

get, list, watch, create, update

config.openshift.io

proxies

get, list, watch

monitoring.coreos.com

servicemonitors, prometheusrules

get, list, watch, create, delete, update, patch

coordination.k8s.io

leases

get, create, update

Table 1.6. Namespaced roles for the cdi-controller service account
API groupResourcesVerbs

"" (core)

configmaps

get, list, watch, create, update, delete

"" (core)

secrets

get, list, watch

batch

cronjobs

get, list, watch, create, update, delete

batch

jobs

create, delete, list, watch

coordination.k8s.io

leases

get, create, update

networking.k8s.io

ingresses

get, list, watch

route.openshift.io

routes

get, list, watch

1.2.3.3. Additional SCCs and permissions for the kubevirt-controller service account

Security context constraints (SCCs) control permissions for pods. These permissions include actions that a pod, a collection of containers, can perform and what resources it can access. You can use SCCs to define a set of conditions that a pod must run with to be accepted into the system.

The virt-controller is a cluster controller that creates the virt-launcher pods for virtual machines in the cluster. These pods are granted permissions by the kubevirt-controller service account.

The kubevirt-controller service account is granted additional SCCs and Linux capabilities so that it can create virt-launcher pods with the appropriate permissions. These extended permissions allow virtual machines to use OpenShift Virtualization features that are beyond the scope of typical pods.

The kubevirt-controller service account is granted the following SCCs:

  • scc.AllowHostDirVolumePlugin = true
    This allows virtual machines to use the hostpath volume plugin.
  • scc.AllowPrivilegedContainer = false
    This ensures the virt-launcher pod is not run as a privileged container.
  • scc.AllowedCapabilities = []corev1.Capability{"SYS_NICE", "NET_BIND_SERVICE"}

    • SYS_NICE allows setting the CPU affinity.
    • NET_BIND_SERVICE allows DHCP and Slirp operations.

Viewing the SCC and RBAC definitions for the kubevirt-controller

You can view the SecurityContextConstraints definition for the kubevirt-controller by using the oc tool:

$ oc get scc kubevirt-controller -o yaml

You can view the RBAC definition for the kubevirt-controller clusterrole by using the oc tool:

$ oc get clusterrole kubevirt-controller -o yaml

1.2.4. Additional resources

1.3. OpenShift Virtualization Architecture

The Operator Lifecycle Manager (OLM) deploys operator pods for each component of OpenShift Virtualization:

  • Compute: virt-operator
  • Storage: cdi-operator
  • Network: cluster-network-addons-operator
  • Scaling: ssp-operator

OLM also deploys the hyperconverged-cluster-operator pod, which is responsible for the deployment, configuration, and life cycle of other components, and several helper pods: hco-webhook, and hyperconverged-cluster-cli-download.

After all operator pods are successfully deployed, you should create the HyperConverged custom resource (CR). The configurations set in the HyperConverged CR serve as the single source of truth and the entrypoint for OpenShift Virtualization, and guide the behavior of the CRs.

The HyperConverged CR creates corresponding CRs for the operators of all other components within its reconciliation loop. Each operator then creates resources such as daemon sets, config maps, and additional components for the OpenShift Virtualization control plane. For example, when the HyperConverged Operator (HCO) creates the KubeVirt CR, the OpenShift Virtualization Operator reconciles it and creates additional resources such as virt-controller, virt-handler, and virt-api.

The OLM deploys the Hostpath Provisioner (HPP) Operator, but it is not functional until you create a hostpath-provisioner CR.

Deployments

1.3.1. About the HyperConverged Operator (HCO)

The HCO, hco-operator, provides a single entry point for deploying and managing OpenShift Virtualization and several helper operators with opinionated defaults. It also creates custom resources (CRs) for those operators.

hco-operator components
Table 1.7. HyperConverged Operator components
ComponentDescription

deployment/hco-webhook

Validates the HyperConverged custom resource contents.

deployment/hyperconverged-cluster-cli-download

Provides the virtctl tool binaries to the cluster so that you can download them directly from the cluster.

KubeVirt/kubevirt-kubevirt-hyperconverged

Contains all operators, CRs, and objects needed by OpenShift Virtualization.

SSP/ssp-kubevirt-hyperconverged

A Scheduling, Scale, and Performance (SSP) CR. This is automatically created by the HCO.

CDI/cdi-kubevirt-hyperconverged

A Containerized Data Importer (CDI) CR. This is automatically created by the HCO.

NetworkAddonsConfig/cluster

A CR that instructs and is managed by the cluster-network-addons-operator.

1.3.2. About the Containerized Data Importer (CDI) Operator

The CDI Operator, cdi-operator, manages CDI and its related resources, which imports a virtual machine (VM) image into a persistent volume claim (PVC) by using a data volume.

cdi-operator components
Table 1.8. CDI Operator components
ComponentDescription

deployment/cdi-apiserver

Manages the authorization to upload VM disks into PVCs by issuing secure upload tokens.

deployment/cdi-uploadproxy

Directs external disk upload traffic to the appropriate upload server pod so that it can be written to the correct PVC. Requires a valid upload token.

pod/cdi-importer

Helper pod that imports a virtual machine image into a PVC when creating a data volume.

1.3.3. About the Cluster Network Addons Operator

The Cluster Network Addons Operator, cluster-network-addons-operator, deploys networking components on a cluster and manages the related resources for extended network functionality.

cluster-network-addons-operator components
Table 1.9. Cluster Network Addons Operator components
ComponentDescription

deployment/kubemacpool-cert-manager

Manages TLS certificates of Kubemacpool’s webhooks.

deployment/kubemacpool-mac-controller-manager

Provides a MAC address pooling service for virtual machine (VM) network interface cards (NICs).

daemonset/bridge-marker

Marks network bridges available on nodes as node resources.

daemonset/kube-cni-linux-bridge-plugin

Installs Container Network Interface (CNI) plugins on cluster nodes, enabling the attachment of VMs to Linux bridges through network attachment definitions.

1.3.4. About the Hostpath Provisioner (HPP) Operator

The HPP Operator, hostpath-provisioner-operator, deploys and manages the multi-node HPP and related resources.

hpp-operator components
Table 1.10. HPP Operator components
ComponentDescription

deployment/hpp-pool-hpp-csi-pvc-block-<worker_node_name>

Provides a worker for each node where the HPP is designated to run. The pods mount the specified backing storage on the node.

daemonset/hostpath-provisioner-csi

Implements the Container Storage Interface (CSI) driver interface of the HPP.

daemonset/hostpath-provisioner

Implements the legacy driver interface of the HPP.

1.3.5. About the Scheduling, Scale, and Performance (SSP) Operator

The SSP Operator, ssp-operator, deploys the common templates, the related default boot sources, the pipeline tasks, and the template validator.

1.3.6. About the OpenShift Virtualization Operator

The OpenShift Virtualization Operator, virt-operator, deploys, upgrades, and manages OpenShift Virtualization without disrupting current virtual machine (VM) workloads. In addition, the OpenShift Virtualization Operator deploys the common instance types and common preferences.

virt-operator components
Table 1.11. virt-operator components
ComponentDescription

deployment/virt-api

HTTP API server that serves as the entry point for all virtualization-related flows.

deployment/virt-controller

Observes the creation of a new VM instance object and creates a corresponding pod. When the pod is scheduled on a node, virt-controller updates the VM with the node name.

daemonset/virt-handler

Monitors any changes to a VM and instructs virt-launcher to perform the required operations. This component is node-specific.

pod/virt-launcher

Contains the VM that was created by the user as implemented by libvirt and qemu.

Chapter 2. Release notes

2.1. OpenShift Virtualization release notes

2.1.1. Providing documentation feedback

To report an error or to improve our documentation, log in to your Red Hat Jira account and submit a Jira issue.

2.1.2. About Red Hat OpenShift Virtualization

With Red Hat OpenShift Virtualization, you can bring traditional virtual machines (VMs) into OpenShift Container Platform and run them alongside containers. In OpenShift Virtualization, VMs are native Kubernetes objects that you can manage by using the OpenShift Container Platform web console or the command line.

OpenShift Virtualization is represented by the OpenShift Virtualization icon.

You can use OpenShift Virtualization the OVN-Kubernetes Container Network Interface (CNI) network provider.

Learn more about what you can do with OpenShift Virtualization.

Learn more about OpenShift Virtualization architecture and deployments.

Prepare your cluster for OpenShift Virtualization.

2.1.2.1. OpenShift Virtualization supported cluster version

OpenShift Virtualization 4.17 is supported for use on OpenShift Container Platform 4.17 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.

2.1.2.2. Supported guest operating systems

To view the supported guest operating systems for OpenShift Virtualization, see Certified Guest Operating Systems in Red Hat OpenStack Platform, Red Hat Virtualization, OpenShift Virtualization and Red Hat Enterprise Linux with KVM.

2.1.2.3. Microsoft Windows SVVP certification

OpenShift Virtualization is certified in Microsoft’s Windows Server Virtualization Validation Program (SVVP) to run Windows Server workloads.

The SVVP certification applies to:

  • Red Hat Enterprise Linux CoreOS workers. In the Microsoft SVVP Catalog, they are named Red Hat OpenShift Container Platform 4 on RHEL CoreOS 9.
  • Intel and AMD CPUs.

2.1.3. Quick starts

Quick start tours are available for several OpenShift Virtualization features. To view the tours, click the Help icon ? in the menu bar on the header of the OpenShift Container Platform web console and then select Quick Starts. You can filter the available tours by entering the keyword virtualization in the Filter field.

2.1.4. New and changed features

This release adds new features and enhancements related to the following components and concepts:

2.1.4.1. Infrastructure
  • The inferFromVolume attribute is now supported for use with imported container disks. When requested, OpenShift Virtualization can copy the labels instancetype.kubevirt.io/default-instancetype and instancetype.kubevirt.io/default-preference from a source container disk to the boot volume of a new VM.
  • You can now select a custom namespace for Red Hat golden images instead of using the default openshift-virtualization-os-images namespace. By using a custom namespace, cluster administrators can restrict user access to the default boot sources. To update this setting by using the web console, go to VirtualizationOverviewSettingsClusterGeneral settingsBootable volumes project.
2.1.4.2. Virtualization
  • You can now increase VM workload density on nodes by overcommitting memory (RAM) with the wasp-agent. The wasp agent assigns swap resources to worker nodes and manages pod evictions when nodes are at risk.

    Note

    Overcommitting memory on a highly utilized system can decrease workload performance.

  • Enabling post-copy live migration for VM workloads is now generally available.
  • As a cluster administrator, you can expose USB devices in a cluster, making them available for virtual machine (VM) owners to assign to VMs. You expose a USB device by first enabling host passthrough and then configuring the VM to access the USB device.
  • You can now use the Application-Aware Quota (AAQ) Operator to customize and manage resource quotas for individual components in an OpenShift Container Platform cluster. The AAQ Operator provides the ApplicationAwareResourceQuota and ApplicationAwareClusterResourceQuota custom resource definitions (CRDs) that can be used to allocate resources without interfering with cluster-level activities such as upgrades and node maintenance.
2.1.4.3. Storage
  • The VirtualMachineSnapshot API version is now v1beta1.
  • The VirtualMachineExport API version is now v1beta1.
2.1.4.4. Web console
  • The OpenShift Container Platform web console includes a new focused view, which presents a condensed navigation menu specific to the OpenShift Virtualization perspective. This view complements but does not replace the existing OpenShift Container Platform web console Virtualization navigation options.

    To access the new view, navigate to AdministratorVirtualization in the web console.

  • An OpenShift Virtualization guided tour is now available. You can access the tour by either clicking Start Tour on the Welcome to OpenShift Virtualization dialog or navigating to VirtualizationOverviewSettingsUserGetting started resourcesGuided tour.
  • Hot plugging memory for VMs from the web console is now generally available.
  • Hot plugging CPUs for VMs from the web console is now generally available.

2.1.5. Deprecated and removed features

2.1.5.1. Deprecated features

Deprecated features are included in the current release and supported. However, they will be removed in a future release and are not recommended for new deployments.

  • The DevPreviewLongLifecycle profile is deprecated. The profile is now LongLifecycle and is generally available.
  • The copy-template, modify-vm-template, and create-vm-from-template tasks are deprecated.
  • Support for Windows Server 2012 R2 templates is deprecated.
  • The alerts KubeVirtComponentExceedsRequestedMemory and KubeVirtComponentExceedsRequestedCPU are deprecated. You can safely silence them.
2.1.5.2. Removed features

Removed features are those that were deprecated in earlier releases. They are now removed from OpenShift Virtualization and are no longer supported.

  • CentOS 7 and CentOS Stream 8 are now in the End of Life phase. As a consequence, the container images for these operating systems have been removed from OpenShift Virtualization and are no longer community supported.
  • The tekton-tasks-operator is removed. The Tekton tasks and example pipelines are now available in the task catalog (ArtifactHub).

2.1.6. Technology Preview features

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use. Note the following scope of support on the Red Hat Customer Portal for these features:

Technology Preview Features Support Scope

  • You can now migrate storage classes for running and stopped VMs.

    Note

    Storage live migration is not enabled by default in the HyperConverged custom resource. To enable the required feature gates, follow the workaround documented in Enable storage live migration in OpenShift Virtualization 4.17 in the Red Hat knowledge base.

2.1.7. Known issues

Nodes
  • Uninstalling OpenShift Virtualization does not remove the feature.node.kubevirt.io node labels created by OpenShift Virtualization. You must remove the labels manually. (CNV-38543)
  • In a heterogeneous cluster with different compute nodes, virtual machines that have HyperV reenlightenment enabled cannot be scheduled on nodes that do not support timestamp-counter scaling (TSC) or have the appropriate TSC frequency. (BZ#2151169)
Storage
  • If you clone more than 100 VMs using the csi-clone cloning strategy, then the Ceph CSI might not purge the clones. Manually deleting the clones might also fail. (CNV-23501)

    • As a workaround, you can restart the ceph-mgr to purge the VM clones.
Virtualization
  • When adding a virtual Trusted Platform Module (vTPM) device to a Windows VM, the BitLocker Drive Encryption system check passes even if the vTPM device is not persistent. This is because a vTPM device that is not persistent stores and recovers encryption keys using ephemeral storage for the lifetime of the virt-launcher pod. When the VM migrates or is shut down and restarts, the vTPM data is lost. (CNV-36448)
  • OpenShift Virtualization links a service account token in use by a pod to that specific pod. OpenShift Virtualization implements a service account volume by creating a disk image that contains a token. If you migrate a VM, then the service account volume becomes invalid. (CNV-33835)

    • As a workaround, use user accounts rather than service accounts because user account tokens are not bound to a specific pod.
Web console
  • When you create a persistent volume claim (PVC) by selecting With Data upload form from the Create PersistentVolumeClaim list in the web console, uploading data to the PVC by using the Upload Data field fails. (CNV-37607)

Chapter 3. Getting started

3.1. Getting started with OpenShift Virtualization

You can explore the features and functionalities of OpenShift Virtualization by installing and configuring a basic environment.

Note

Cluster configuration procedures require cluster-admin privileges.

3.1.1. Tours and quick starts

You can start exploring OpenShift Virtualization by taking tours in the OpenShift Container Platform web console.

Getting started tour

This short guided tour introduces several key aspects of using OpenShift Virtualization. There are two ways to start the tour:

  • On the Welcome to OpenShift Virtualization dialog, click Start Tour.
  • Go to VirtualizationOverviewSettingsUserGetting started resources and click Guided tour.
Quick starts

Quick start tours are available for several OpenShift Virtualization features. To access quick starts, complete the following steps:

  1. Click the Help icon ? in the menu bar on the header of the OpenShift Container Platform web console.
  2. Select Quick Starts.

You can filter the available tours by entering the keyword virtualization in the Filter field.

3.1.2. Planning and installing OpenShift Virtualization

Plan and install OpenShift Virtualization on an OpenShift Container Platform cluster:

Planning and installation resources

3.1.3. Creating and managing virtual machines

Create a virtual machine (VM):

Connect a VM to a secondary network:

Connect to a VM:

Manage a VM:

3.1.4. Next steps

3.2. Using the CLI tools

You can manage OpenShift Virtualization resources by using the virtctl command line tool.

You can access and modify virtual machine (VM) disk images by using the libguestfs command line tool. You deploy libguestfs by using the virtctl libguestfs command.

3.2.1. Installing virtctl

To install virtctl on Red Hat Enterprise Linux (RHEL) 9, Linux, Windows, and MacOS operating systems, you download and install the virtctl binary file.

To install virtctl on RHEL 8, you enable the OpenShift Virtualization repository and then install the kubevirt-virtctl package.

3.2.1.1. Installing the virtctl binary on RHEL 9, Linux, Windows, or macOS

You can download the virtctl binary for your operating system from the OpenShift Container Platform web console and then install it.

Procedure

  1. Navigate to the Virtualization → Overview page in the web console.
  2. Click the Download virtctl link to download the virtctl binary for your operating system.
  3. Install virtctl:

    • For RHEL 9 and other Linux operating systems:

      1. Decompress the archive file:

        $ tar -xvf <virtctl-version-distribution.arch>.tar.gz
      2. Run the following command to make the virtctl binary executable:

        $ chmod +x <path/virtctl-file-name>
      3. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        $ echo $PATH
      4. Set the KUBECONFIG environment variable:

        $ export KUBECONFIG=/home/<user>/clusters/current/auth/kubeconfig
    • For Windows:

      1. Decompress the archive file.
      2. Navigate the extracted folder hierarchy and double-click the virtctl executable file to install the client.
      3. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        C:\> path
    • For macOS:

      1. Decompress the archive file.
      2. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        echo $PATH
3.2.1.2. Installing the virtctl RPM on RHEL 8

You can install the virtctl RPM package on Red Hat Enterprise Linux (RHEL) 8 by enabling the OpenShift Virtualization repository and installing the kubevirt-virtctl package.

Prerequisites

  • Each host in your cluster must be registered with Red Hat Subscription Manager (RHSM) and have an active OpenShift Container Platform subscription.

Procedure

  1. Enable the OpenShift Virtualization repository by using the subscription-manager CLI tool to run the following command:

    # subscription-manager repos --enable cnv-4.17-for-rhel-8-x86_64-rpms
  2. Install the kubevirt-virtctl package by running the following command:

    # yum install kubevirt-virtctl

3.2.2. virtctl commands

The virtctl client is a command-line utility for managing OpenShift Virtualization resources.

Note

The virtual machine (VM) commands also apply to virtual machine instances (VMIs) unless otherwise specified.

3.2.2.1. virtctl information commands

You use virtctl information commands to view information about the virtctl client.

Table 3.1. Information commands
CommandDescription

virtctl version

View the virtctl client and server versions.

virtctl help

View a list of virtctl commands.

virtctl <command> -h|--help

View a list of options for a specific command.

virtctl options

View a list of global command options for any virtctl command.

3.2.2.2. VM information commands

You can use virtctl to view information about virtual machines (VMs) and virtual machine instances (VMIs).

Table 3.2. VM information commands
CommandDescription

virtctl fslist <vm_name>

View the file systems available on a guest machine.

virtctl guestosinfo <vm_name>

View information about the operating systems on a guest machine.

virtctl userlist <vm_name>

View the logged-in users on a guest machine.

3.2.2.3. VM manifest creation commands

You can use virtctl create commands to create manifests for virtual machines, instance types, and preferences.

Table 3.3. VM manifest creation commands
CommandDescription
virtctl create vm

Create a VirtualMachine (VM) manifest.

virtctl create vm --name <vm_name>

Create a VM manifest, specifying a name for the VM.

virtctl create vm --instancetype <instancetype_name>

Create a VM manifest that uses an existing cluster-wide instance type.

virtctl create vm --instancetype=virtualmachineinstancetype/<instancetype_name>

Create a VM manifest that uses an existing namespaced instance type.

virtctl create instancetype --cpu <cpu_value> --memory <memory_value> --name <instancetype_name>

Create a manifest for a cluster-wide instance type.

virtctl create instancetype --cpu <cpu_value> --memory <memory_value> --name <instancetype_name> --namespace <namespace_value>

Create a manifest for a namespaced instance type.

virtctl create preference --name <preference_name>

Create a manifest for a cluster-wide VM preference, specifying a name for the preference.

virtctl create preference --namespace <namespace_value>

Create a manifest for a namespaced VM preference.

3.2.2.4. VM management commands

You use virtctl virtual machine (VM) management commands to manage and migrate virtual machines (VMs) and virtual machine instances (VMIs).

Table 3.4. VM management commands
CommandDescription

virtctl start <vm_name>

Start a VM.

virtctl start --paused <vm_name>

Start a VM in a paused state. This option enables you to interrupt the boot process from the VNC console.

virtctl stop <vm_name>

Stop a VM.

virtctl stop <vm_name> --grace-period 0 --force

Force stop a VM. This option might cause data inconsistency or data loss.

virtctl pause vm <vm_name>

Pause a VM. The machine state is kept in memory.

virtctl unpause vm <vm_name>

Unpause a VM.

virtctl migrate <vm_name>

Migrate a VM.

virtctl migrate-cancel <vm_name>

Cancel a VM migration.

virtctl restart <vm_name>

Restart a VM.

3.2.2.5. VM connection commands

You use virtctl connection commands to expose ports and connect to virtual machines (VMs) and virtual machine instances (VMIs).

Table 3.5. VM connection commands
CommandDescription

virtctl console <vm_name>

Connect to the serial console of a VM.

virtctl expose vm <vm_name> --name <service_name> --type <ClusterIP|NodePort|LoadBalancer> --port <port>

Create a service that forwards a designated port of a VM and expose the service on the specified port of the node.

Example: virtctl expose vm rhel9_vm --name rhel9-ssh --type NodePort --port 22

virtctl scp -i <ssh_key> <file_name> <user_name>@<vm_name>

Copy a file from your machine to a VM. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl scp -i <ssh_key> <user_name@<vm_name>:<file_name> .

Copy a file from a VM to your machine. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl ssh -i <ssh_key> <user_name>@<vm_name>

Open an SSH connection with a VM. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl vnc <vm_name>

Connect to the VNC console of a VM.

You must have virt-viewer installed.

virtctl vnc --proxy-only=true <vm_name>

Display the port number and connect manually to a VM by using any viewer through the VNC connection.

virtctl vnc --port=<port-number> <vm_name>

Specify a port number to run the proxy on the specified port, if that port is available.

If a port number is not specified, the proxy runs on a random port.

3.2.2.6. VM export commands

Use virtctl vmexport commands to create, download, or delete a volume exported from a VM, VM snapshot, or persistent volume claim (PVC). Certain manifests also contain a header secret, which grants access to the endpoint to import a disk image in a format that OpenShift Virtualization can use.

Table 3.6. VM export commands
CommandDescription

virtctl vmexport create <vmexport_name> --vm|snapshot|pvc=<object_name>

Create a VirtualMachineExport custom resource (CR) to export a volume from a VM, VM snapshot, or PVC.

  • --vm: Exports the PVCs of a VM.
  • --snapshot: Exports the PVCs contained in a VirtualMachineSnapshot CR.
  • --pvc: Exports a PVC.
  • Optional: --ttl=1h specifies the time to live. The default duration is 2 hours.

virtctl vmexport delete <vmexport_name>

Delete a VirtualMachineExport CR manually.

virtctl vmexport download <vmexport_name> --output=<output_file> --volume=<volume_name>

Download the volume defined in a VirtualMachineExport CR.

  • --output specifies the file format. Example: disk.img.gz.
  • --volume specifies the volume to download. This flag is optional if only one volume is available.

Optional:

  • --keep-vme retains the VirtualMachineExport CR after download. The default behavior is to delete the VirtualMachineExport CR after download.
  • --insecure enables an insecure HTTP connection.

virtctl vmexport download <vmexport_name> --<vm|snapshot|pvc>=<object_name> --output=<output_file> --volume=<volume_name>

Create a VirtualMachineExport CR and then download the volume defined in the CR.

virtctl vmexport download export --manifest

Retrieve the manifest for an existing export. The manifest does not include the header secret.

virtctl vmexport download export --manifest --vm=example

Create a VM export for a VM example, and retrieve the manifest. The manifest does not include the header secret.

virtctl vmexport download export --manifest --snap=example

Create a VM export for a VM snapshot example, and retrieve the manifest. The manifest does not include the header secret.

virtctl vmexport download export --manifest --include-secret

Retrieve the manifest for an existing export. The manifest includes the header secret.

virtctl vmexport download export --manifest --manifest-output-format=json

Retrieve the manifest for an existing export in json format. The manifest does not include the header secret.

virtctl vmexport download export --manifest --include-secret --output=manifest.yaml

Retrieve the manifest for an existing export. The manifest includes the header secret and writes it to the file specified.

3.2.2.7. VM memory dump commands

You can use the virtctl memory-dump command to output a VM memory dump on a PVC. You can specify an existing PVC or use the --create-claim flag to create a new PVC.

Prerequisites

  • The PVC volume mode must be FileSystem.
  • The PVC must be large enough to contain the memory dump.

    The formula for calculating the PVC size is (VMMemorySize + 100Mi) * FileSystemOverhead, where 100Mi is the memory dump overhead.

  • You must enable the hot plug feature gate in the HyperConverged custom resource by running the following command:

    $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
      --type json -p '[{"op": "add", "path": "/spec/featureGates", \
      "value": "HotplugVolumes"}]'

Downloading the memory dump

You must use the virtctl vmexport download command to download the memory dump:

$ virtctl vmexport download <vmexport_name> --vm|pvc=<object_name> \
  --volume=<volume_name> --output=<output_file>
Table 3.7. VM memory dump commands
CommandDescription

virtctl memory-dump get <vm_name> --claim-name=<pvc_name>

Save the memory dump of a VM on a PVC. The memory dump status is displayed in the status section of the VirtualMachine resource.

Optional:

  • --create-claim creates a new PVC with the appropriate size. This flag has the following options:

    • --storage-class=<storage_class>: Specify a storage class for the PVC.
    • --access-mode=<access_mode>: Specify ReadWriteOnce or ReadWriteMany.

virtctl memory-dump get <vm_name>

Rerun the virtctl memory-dump command with the same PVC.

This command overwrites the previous memory dump.

virtctl memory-dump remove <vm_name>

Remove a memory dump.

You must remove a memory dump manually if you want to change the target PVC.

This command removes the association between the VM and the PVC, so that the memory dump is not displayed in the status section of the VirtualMachine resource. The PVC is not affected.

3.2.2.8. Hot plug and hot unplug commands

You use virtctl to add or remove resources from running virtual machines (VMs) and virtual machine instances (VMIs).

Table 3.8. Hot plug and hot unplug commands
CommandDescription

virtctl addvolume <vm_name> --volume-name=<datavolume_or_PVC> [--persist] [--serial=<label>]

Hot plug a data volume or persistent volume claim (PVC).

Optional:

  • --persist mounts the virtual disk permanently on a VM. This flag does not apply to VMIs.
  • --serial=<label> adds a label to the VM. If you do not specify a label, the default label is the data volume or PVC name.

virtctl removevolume <vm_name> --volume-name=<virtual_disk>

Hot unplug a virtual disk.

virtctl addinterface <vm_name> --network-attachment-definition-name <net_attach_def_name> --name <interface_name>

Hot plug a Linux bridge network interface.

virtctl removeinterface <vm_name> --name <interface_name>

Hot unplug a Linux bridge network interface.

3.2.2.9. Image upload commands

You use the virtctl image-upload commands to upload a VM image to a data volume.

Table 3.9. Image upload commands
CommandDescription

virtctl image-upload dv <datavolume_name> --image-path=</path/to/image> --no-create

Upload a VM image to a data volume that already exists.

virtctl image-upload dv <datavolume_name> --size=<datavolume_size> --image-path=</path/to/image>

Upload a VM image to a new data volume of a specified requested size.

3.2.3. Deploying libguestfs by using virtctl

You can use the virtctl guestfs command to deploy an interactive container with libguestfs-tools and a persistent volume claim (PVC) attached to it.

Procedure

  • To deploy a container with libguestfs-tools, mount the PVC, and attach a shell to it, run the following command:

    $ virtctl guestfs -n <namespace> <pvc_name> 1
    1
    The PVC name is a required argument. If you do not include it, an error message appears.
3.2.3.1. Libguestfs and virtctl guestfs commands

Libguestfs tools help you access and modify virtual machine (VM) disk images. You can use libguestfs tools to view and edit files in a guest, clone and build virtual machines, and format and resize disks.

You can also use the virtctl guestfs command and its sub-commands to modify, inspect, and debug VM disks on a PVC. To see a complete list of possible sub-commands, enter virt- on the command line and press the Tab key. For example:

CommandDescription

virt-edit -a /dev/vda /etc/motd

Edit a file interactively in your terminal.

virt-customize -a /dev/vda --ssh-inject root:string:<public key example>

Inject an ssh key into the guest and create a login.

virt-df -a /dev/vda -h

See how much disk space is used by a VM.

virt-customize -a /dev/vda --run-command 'rpm -qa > /rpm-list'

See the full list of all RPMs installed on a guest by creating an output file containing the full list.

virt-cat -a /dev/vda /rpm-list

Display the output file list of all RPMs created using the virt-customize -a /dev/vda --run-command 'rpm -qa > /rpm-list' command in your terminal.

virt-sysprep -a /dev/vda

Seal a virtual machine disk image to be used as a template.

By default, virtctl guestfs creates a session with everything needed to manage a VM disk. However, the command also supports several flag options if you want to customize the behavior:

Flag OptionDescription

--h or --help

Provides help for guestfs.

-n <namespace> option with a <pvc_name> argument

To use a PVC from a specific namespace.

If you do not use the -n <namespace> option, your current project is used. To change projects, use oc project <namespace>.

If you do not include a <pvc_name> argument, an error message appears.

--image string

Lists the libguestfs-tools container image.

You can configure the container to use a custom image by using the --image option.

--kvm

Indicates that kvm is used by the libguestfs-tools container.

By default, virtctl guestfs sets up kvm for the interactive container, which greatly speeds up the libguest-tools execution because it uses QEMU.

If a cluster does not have any kvm supporting nodes, you must disable kvm by setting the option --kvm=false.

If not set, the libguestfs-tools pod remains pending because it cannot be scheduled on any node.

--pull-policy string

Shows the pull policy for the libguestfs image.

You can also overwrite the image’s pull policy by setting the pull-policy option.

The command also checks if a PVC is in use by another pod, in which case an error message appears. However, once the libguestfs-tools process starts, the setup cannot avoid a new pod using the same PVC. You must verify that there are no active virtctl guestfs pods before starting the VM that accesses the same PVC.

Note

The virtctl guestfs command accepts only a single PVC attached to the interactive pod.

3.2.4. Using Ansible

To use the Ansible collection for OpenShift Virtualization, see Red Hat Ansible Automation Hub (Red Hat Hybrid Cloud Console).

Chapter 4. Installing

4.1. Preparing your cluster for OpenShift Virtualization

Review this section before you install OpenShift Virtualization to ensure that your cluster meets the requirements.

Important
Installation method considerations
You can use any installation method, including user-provisioned, installer-provisioned, or assisted installer, to deploy OpenShift Container Platform. However, the installation method and the cluster topology might affect OpenShift Virtualization functionality, such as snapshots or live migration.
Red Hat OpenShift Data Foundation
If you deploy OpenShift Virtualization with Red Hat OpenShift Data Foundation, you must create a dedicated storage class for Windows virtual machine disks. See Optimizing ODF PersistentVolumes for Windows VMs for details.
IPv6
You cannot run OpenShift Virtualization on a single-stack IPv6 cluster.

FIPS mode

If you install your cluster in FIPS mode, no additional setup is required for OpenShift Virtualization.

4.1.1. Supported platforms

You can use the following platforms with OpenShift Virtualization:

Bare metal instances or servers offered by other cloud providers are not supported.

4.1.1.1. OpenShift Virtualization on AWS bare metal

You can run OpenShift Virtualization on an Amazon Web Services (AWS) bare-metal OpenShift Container Platform cluster.

Note

OpenShift Virtualization is also supported on Red Hat OpenShift Service on AWS (ROSA) Classic clusters, which have the same configuration requirements as AWS bare-metal clusters.

Before you set up your cluster, review the following summary of supported features and limitations:

Installing
  • You can install the cluster by using installer-provisioned infrastructure, ensuring that you specify bare-metal instance types for the worker nodes. For example, you can use the c5n.metal type value for a machine based on x86_64 architecture. You specify bare-metal instance types by editing the install-config.yaml file.

    For more information, see the OpenShift Container Platform documentation about installing on AWS.

Accessing virtual machines (VMs)
  • There is no change to how you access VMs by using the virtctl CLI tool or the OpenShift Container Platform web console.
  • You can expose VMs by using a NodePort or LoadBalancer service.

    • The load balancer approach is preferable because OpenShift Container Platform automatically creates the load balancer in AWS and manages its lifecycle. A security group is also created for the load balancer, and you can use annotations to attach existing security groups. When you remove the service, OpenShift Container Platform removes the load balancer and its associated resources.
Networking
  • You cannot use Single Root I/O Virtualization (SR-IOV) or bridge Container Network Interface (CNI) networks, including virtual LAN (VLAN). If your application requires a flat layer 2 network or control over the IP pool, consider using OVN-Kubernetes secondary overlay networks.
Storage
  • You can use any storage solution that is certified by the storage vendor to work with the underlying platform.

    Important

    AWS bare-metal and ROSA clusters might have different supported storage solutions. Ensure that you confirm support with your storage vendor.

  • Using Amazon Elastic File System (EFS) or Amazon Elastic Block Store (EBS) with OpenShift Virtualization might cause performance and functionality limitations as shown in the following table:

    Table 4.1. EFS and EBS performance and functionality limitations
    FeatureEBS volumeEFS volumeShared storage solutions
     

    gp2

    gp3

    io2

      

    VM live migration

    Not available

    Not available

    Available

    Available

    Available

    Fast VM creation by using cloning

    Available

    Not available

    Available

    VM backup and restore by using snapshots

    Available

    Not available

    Available

    Consider using CSI storage, which supports ReadWriteMany (RWX), cloning, and snapshots to enable live migration, fast VM creation, and VM snapshots capabilities.

Hosted control planes (HCPs)
  • HCPs for OpenShift Virtualization are not currently supported on AWS infrastructure.

4.1.2. Hardware and operating system requirements

Review the following hardware and operating system requirements for OpenShift Virtualization.

4.1.2.1. CPU requirements
  • Supported by Red Hat Enterprise Linux (RHEL) 9.

    See Red Hat Ecosystem Catalog for supported CPUs.

    Note

    If your worker nodes have different CPUs, live migration failures might occur because different CPUs have different capabilities. You can mitigate this issue by ensuring that your worker nodes have CPUs with the appropriate capacity and by configuring node affinity rules for your virtual machines.

    See Configuring a required node affinity rule for details.

  • Support for AMD and Intel 64-bit architectures (x86-64-v2).
  • Support for Intel 64 or AMD64 CPU extensions.
  • Intel VT or AMD-V hardware virtualization extensions enabled.
  • NX (no execute) flag enabled.
4.1.2.2. Operating system requirements
  • Red Hat Enterprise Linux CoreOS (RHCOS) installed on worker nodes.

    See About RHCOS for details.

    Note

    RHEL worker nodes are not supported.

4.1.2.3. Storage requirements
  • Supported by OpenShift Container Platform. See Optimizing storage.
  • You must create a default OpenShift Virtualization or OpenShift Container Platform storage class. The purpose of this is to address the unique storage needs of VM workloads and offer optimized performance, reliability, and user experience. If both OpenShift Virtualization and OpenShift Container Platform default storage classes exist, the OpenShift Virtualization class takes precedence when creating VM disks.
Note

To mark a storage class as the default for virtualization workloads, set the annotation storageclass.kubevirt.io/is-default-virt-class to "true".

  • If the storage provisioner supports snapshots, you must associate a VolumeSnapshotClass object with the default storage class.
4.1.2.3.1. About volume and access modes for virtual machine disks

If you use the storage API with known storage providers, the volume and access modes are selected automatically. However, if you use a storage class that does not have a storage profile, you must configure the volume and access mode.

For best results, use the ReadWriteMany (RWX) access mode and the Block volume mode. This is important for the following reasons:

  • ReadWriteMany (RWX) access mode is required for live migration.
  • The Block volume mode performs significantly better than the Filesystem volume mode. This is because the Filesystem volume mode uses more storage layers, including a file system layer and a disk image file. These layers are not necessary for VM disk storage.

    For example, if you use Red Hat OpenShift Data Foundation, Ceph RBD volumes are preferable to CephFS volumes.

Important

You cannot live migrate virtual machines with the following configurations:

  • Storage volume with ReadWriteOnce (RWO) access mode
  • Passthrough features such as GPUs

Set the evictionStrategy field to None for these virtual machines. The None strategy powers down VMs during node reboots.

4.1.3. Live migration requirements

  • Shared storage with ReadWriteMany (RWX) access mode.
  • Sufficient RAM and network bandwidth.

    Note

    You must ensure that there is enough memory request capacity in the cluster to support node drains that result in live migrations. You can determine the approximate required spare memory by using the following calculation:

    Product of (Maximum number of nodes that can drain in parallel) and (Highest total VM memory request allocations across nodes)

    The default number of migrations that can run in parallel in the cluster is 5.

  • If the virtual machine uses a host model CPU, the nodes must support the virtual machine’s host model CPU.
  • A dedicated Multus network for live migration is highly recommended. A dedicated network minimizes the effects of network saturation on tenant workloads during migration.

4.1.4. Physical resource overhead requirements

OpenShift Virtualization is an add-on to OpenShift Container Platform and imposes additional overhead that you must account for when planning a cluster. Each cluster machine must accommodate the following overhead requirements in addition to the OpenShift Container Platform requirements. Oversubscribing the physical resources in a cluster can affect performance.

Important

The numbers noted in this documentation are based on Red Hat’s test methodology and setup. These numbers can vary based on your own individual setup and environments.

Memory overhead

Calculate the memory overhead values for OpenShift Virtualization by using the equations below.

Cluster memory overhead

Memory overhead per infrastructure node ≈ 150 MiB

Memory overhead per worker node ≈ 360 MiB

Additionally, OpenShift Virtualization environment resources require a total of 2179 MiB of RAM that is spread across all infrastructure nodes.

Virtual machine memory overhead

Memory overhead per virtual machine ≈ (1.002 × requested memory) \
              + 218 MiB \ 1
              + 8 MiB × (number of vCPUs) \ 2
              + 16 MiB × (number of graphics devices) \ 3
              + (additional memory overhead) 4

1
Required for the processes that run in the virt-launcher pod.
2
Number of virtual CPUs requested by the virtual machine.
3
Number of virtual graphics cards requested by the virtual machine.
4
Additional memory overhead:
  • If your environment includes a Single Root I/O Virtualization (SR-IOV) network device or a Graphics Processing Unit (GPU), allocate 1 GiB additional memory overhead for each device.
  • If Secure Encrypted Virtualization (SEV) is enabled, add 256 MiB.
  • If Trusted Platform Module (TPM) is enabled, add 53 MiB.
CPU overhead

Calculate the cluster processor overhead requirements for OpenShift Virtualization by using the equation below. The CPU overhead per virtual machine depends on your individual setup.

Cluster CPU overhead

CPU overhead for infrastructure nodes ≈ 4 cores

OpenShift Virtualization increases the overall utilization of cluster level services such as logging, routing, and monitoring. To account for this workload, ensure that nodes that host infrastructure components have capacity allocated for 4 additional cores (4000 millicores) distributed across those nodes.

CPU overhead for worker nodes ≈ 2 cores + CPU overhead per virtual machine

Each worker node that hosts virtual machines must have capacity for 2 additional cores (2000 millicores) for OpenShift Virtualization management workloads in addition to the CPUs required for virtual machine workloads.

Virtual machine CPU overhead

If dedicated CPUs are requested, there is a 1:1 impact on the cluster CPU overhead requirement. Otherwise, there are no specific rules about how many CPUs a virtual machine requires.

Storage overhead

Use the guidelines below to estimate storage overhead requirements for your OpenShift Virtualization environment.

Cluster storage overhead

Aggregated storage overhead per node ≈ 10 GiB

10 GiB is the estimated on-disk storage impact for each node in the cluster when you install OpenShift Virtualization.

Virtual machine storage overhead

Storage overhead per virtual machine depends on specific requests for resource allocation within the virtual machine. The request could be for ephemeral storage on the node or storage resources hosted elsewhere in the cluster. OpenShift Virtualization does not currently allocate any additional ephemeral storage for the running container itself.

Example

As a cluster administrator, if you plan to host 10 virtual machines in the cluster, each with 1 GiB of RAM and 2 vCPUs, the memory impact across the cluster is 11.68 GiB. The estimated on-disk storage impact for each node in the cluster is 10 GiB and the CPU impact for worker nodes that host virtual machine workloads is a minimum of 2 cores.

4.1.5. Single-node OpenShift differences

You can install OpenShift Virtualization on single-node OpenShift.

However, you should be aware that Single-node OpenShift does not support the following features:

  • High availability
  • Pod disruption
  • Live migration
  • Virtual machines or templates that have an eviction strategy configured

4.1.6. Object maximums

You must consider the following tested object maximums when planning your cluster:

4.1.7. Cluster high-availability options

You can configure one of the following high-availability (HA) options for your cluster:

  • Automatic high availability for installer-provisioned infrastructure (IPI) is available by deploying machine health checks.

    Note

    In OpenShift Container Platform clusters installed using installer-provisioned infrastructure and with a properly configured MachineHealthCheck resource, if a node fails the machine health check and becomes unavailable to the cluster, it is recycled. What happens next with VMs that ran on the failed node depends on a series of conditions. See Run strategies for more detailed information about the potential outcomes and how run strategies affect those outcomes.

  • Automatic high availability for both IPI and non-IPI is available by using the Node Health Check Operator on the OpenShift Container Platform cluster to deploy the NodeHealthCheck controller. The controller identifies unhealthy nodes and uses a remediation provider, such as the Self Node Remediation Operator or Fence Agents Remediation Operator, to remediate the unhealthy nodes. For more information on remediation, fencing, and maintaining nodes, see the Workload Availability for Red Hat OpenShift documentation.
  • High availability for any platform is available by using either a monitoring system or a qualified human to monitor node availability. When a node is lost, shut it down and run oc delete node <lost_node>.

    Note

    Without an external monitoring system or a qualified human monitoring node health, virtual machines lose high availability.

4.2. Installing OpenShift Virtualization

Install OpenShift Virtualization to add virtualization functionality to your OpenShift Container Platform cluster.

Important

If you install OpenShift Virtualization in a restricted environment with no internet connectivity, you must configure Operator Lifecycle Manager for disconnected environments.

If you have limited internet connectivity, you can configure proxy support in OLM to access the OperatorHub.

4.2.1. Installing the OpenShift Virtualization Operator

Install the OpenShift Virtualization Operator by using the OpenShift Container Platform web console or the command line.

4.2.1.1. Installing the OpenShift Virtualization Operator by using the web console

You can deploy the OpenShift Virtualization Operator by using the OpenShift Container Platform web console.

Prerequisites

  • Install OpenShift Container Platform 4.17 on your cluster.
  • Log in to the OpenShift Container Platform web console as a user with cluster-admin permissions.

Procedure

  1. From the Administrator perspective, click OperatorsOperatorHub.
  2. In the Filter by keyword field, type Virtualization.
  3. Select the OpenShift Virtualization Operator tile with the Red Hat source label.
  4. Read the information about the Operator and click Install.
  5. On the Install Operator page:

    1. Select stable from the list of available Update Channel options. This ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
    2. For Installed Namespace, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory openshift-cnv namespace, which is automatically created if it does not exist.

      Warning

      Attempting to install the OpenShift Virtualization Operator in a namespace other than openshift-cnv causes the installation to fail.

    3. For Approval Strategy, it is highly recommended that you select Automatic, which is the default value, so that OpenShift Virtualization automatically updates when a new version is available in the stable update channel.

      While it is possible to select the Manual approval strategy, this is inadvisable because of the high risk that it presents to the supportability and functionality of your cluster. Only select Manual if you fully understand these risks and cannot use Automatic.

      Warning

      Because OpenShift Virtualization is only supported when used with the corresponding OpenShift Container Platform version, missing OpenShift Virtualization updates can cause your cluster to become unsupported.

  6. Click Install to make the Operator available to the openshift-cnv namespace.
  7. When the Operator installs successfully, click Create HyperConverged.
  8. Optional: Configure Infra and Workloads node placement options for OpenShift Virtualization components.
  9. Click Create to launch OpenShift Virtualization.

Verification

  • Navigate to the WorkloadsPods page and monitor the OpenShift Virtualization pods until they are all Running. After all the pods display the Running state, you can use OpenShift Virtualization.
4.2.1.2. Installing the OpenShift Virtualization Operator by using the command line

Subscribe to the OpenShift Virtualization catalog and install the OpenShift Virtualization Operator by applying manifests to your cluster.

4.2.1.2.1. Subscribing to the OpenShift Virtualization catalog by using the CLI

Before you install OpenShift Virtualization, you must subscribe to the OpenShift Virtualization catalog. Subscribing gives the openshift-cnv namespace access to the OpenShift Virtualization Operators.

To subscribe, configure Namespace, OperatorGroup, and Subscription objects by applying a single manifest to your cluster.

Prerequisites

  • Install OpenShift Container Platform 4.17 on your cluster.
  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a YAML file that contains the following manifest:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: openshift-cnv
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: kubevirt-hyperconverged-group
      namespace: openshift-cnv
    spec:
      targetNamespaces:
        - openshift-cnv
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: hco-operatorhub
      namespace: openshift-cnv
    spec:
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      name: kubevirt-hyperconverged
      startingCSV: kubevirt-hyperconverged-operator.v4.17.3
      channel: "stable" 1
    1
    Using the stable channel ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
  2. Create the required Namespace, OperatorGroup, and Subscription objects for OpenShift Virtualization by running the following command:

    $ oc apply -f <file name>.yaml
Note

You can configure certificate rotation parameters in the YAML file.

4.2.1.2.2. Deploying the OpenShift Virtualization Operator by using the CLI

You can deploy the OpenShift Virtualization Operator by using the oc CLI.

Prerequisites

  • Subscribe to the OpenShift Virtualization catalog in the openshift-cnv namespace.
  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a YAML file that contains the following manifest:

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
      namespace: openshift-cnv
    spec:
  2. Deploy the OpenShift Virtualization Operator by running the following command:

    $ oc apply -f <file_name>.yaml

Verification

  • Ensure that OpenShift Virtualization deployed successfully by watching the PHASE of the cluster service version (CSV) in the openshift-cnv namespace. Run the following command:

    $ watch oc get csv -n openshift-cnv

    The following output displays if deployment was successful:

    Example output

    NAME                                      DISPLAY                    VERSION   REPLACES   PHASE
    kubevirt-hyperconverged-operator.v4.17.3   OpenShift Virtualization   4.17.3                Succeeded

4.2.2. Next steps

  • The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first.

4.3. Uninstalling OpenShift Virtualization

You uninstall OpenShift Virtualization by using the web console or the command line interface (CLI) to delete the OpenShift Virtualization workloads, the Operator, and its resources.

4.3.1. Uninstalling OpenShift Virtualization by using the web console

You uninstall OpenShift Virtualization by using the web console to perform the following tasks:

Important

You must first delete all virtual machines, and virtual machine instances.

You cannot uninstall OpenShift Virtualization while its workloads remain on the cluster.

4.3.1.1. Deleting the HyperConverged custom resource

To uninstall OpenShift Virtualization, you first delete the HyperConverged custom resource (CR).

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to the OperatorsInstalled Operators page.
  2. Select the OpenShift Virtualization Operator.
  3. Click the OpenShift Virtualization Deployment tab.
  4. Click the Options menu kebab beside kubevirt-hyperconverged and select Delete HyperConverged.
  5. Click Delete in the confirmation window.
4.3.1.2. Deleting Operators from a cluster using the web console

Cluster administrators can delete installed Operators from a selected namespace by using the web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster web console using an account with cluster-admin permissions.

Procedure

  1. Navigate to the OperatorsInstalled Operators page.
  2. Scroll or enter a keyword into the Filter by name field to find the Operator that you want to remove. Then, click on it.
  3. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

    An Uninstall Operator? dialog box is displayed.

  4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

    Note

    This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

4.3.1.3. Deleting a namespace using the web console

You can delete a namespace by using the OpenShift Container Platform web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to AdministrationNamespaces.
  2. Locate the namespace that you want to delete in the list of namespaces.
  3. On the far right side of the namespace listing, select Delete Namespace from the Options menu kebab .
  4. When the Delete Namespace pane opens, enter the name of the namespace that you want to delete in the field.
  5. Click Delete.
4.3.1.4. Deleting OpenShift Virtualization custom resource definitions

You can delete the OpenShift Virtualization custom resource definitions (CRDs) by using the web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to AdministrationCustomResourceDefinitions.
  2. Select the Label filter and enter operators.coreos.com/kubevirt-hyperconverged.openshift-cnv in the Search field to display the OpenShift Virtualization CRDs.
  3. Click the Options menu kebab beside each CRD and select Delete CustomResourceDefinition.

4.3.2. Uninstalling OpenShift Virtualization by using the CLI

You can uninstall OpenShift Virtualization by using the OpenShift CLI (oc).

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have deleted all virtual machines and virtual machine instances. You cannot uninstall OpenShift Virtualization while its workloads remain on the cluster.

Procedure

  1. Delete the HyperConverged custom resource:

    $ oc delete HyperConverged kubevirt-hyperconverged -n openshift-cnv
  2. Delete the OpenShift Virtualization Operator subscription:

    $ oc delete subscription kubevirt-hyperconverged -n openshift-cnv
  3. Delete the OpenShift Virtualization ClusterServiceVersion resource:

    $ oc delete csv -n openshift-cnv -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv
  4. Delete the OpenShift Virtualization namespace:

    $ oc delete namespace openshift-cnv
  5. List the OpenShift Virtualization custom resource definitions (CRDs) by running the oc delete crd command with the dry-run option:

    $ oc delete crd --dry-run=client -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv

    Example output

    customresourcedefinition.apiextensions.k8s.io "cdis.cdi.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "hostpathprovisioners.hostpathprovisioner.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "hyperconvergeds.hco.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "kubevirts.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "ssps.ssp.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "tektontasks.tektontasks.kubevirt.io" deleted (dry run)

  6. Delete the CRDs by running the oc delete crd command without the dry-run option:

    $ oc delete crd -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv

Chapter 5. Postinstallation configuration

5.1. Postinstallation configuration

The following procedures are typically performed after OpenShift Virtualization is installed. You can configure the components that are relevant for your environment:

5.2. Specifying nodes for OpenShift Virtualization components

The default scheduling for virtual machines (VMs) on bare metal nodes is appropriate. Optionally, you can specify the nodes where you want to deploy OpenShift Virtualization Operators, workloads, and controllers by configuring node placement rules.

Note

You can configure node placement rules for some components after installing OpenShift Virtualization, but virtual machines cannot be present if you want to configure node placement rules for workloads.

5.2.1. About node placement rules for OpenShift Virtualization components

You can use node placement rules for the following tasks:

  • Deploy virtual machines only on nodes intended for virtualization workloads.
  • Deploy Operators only on infrastructure nodes.
  • Maintain separation between workloads.

Depending on the object, you can use one or more of the following rule types:

nodeSelector
Allows pods to be scheduled on nodes that are labeled with the key-value pair or pairs that you specify in this field. The node must have labels that exactly match all listed pairs.
affinity
Enables you to use more expressive syntax to set rules that match nodes with pods. Affinity also allows for more nuance in how the rules are applied. For example, you can specify that a rule is a preference, not a requirement. If a rule is a preference, pods are still scheduled when the rule is not satisfied.
tolerations
Allows pods to be scheduled on nodes that have matching taints. If a taint is applied to a node, that node only accepts pods that tolerate the taint.

5.2.2. Applying node placement rules

You can apply node placement rules by editing a Subscription, HyperConverged, or HostPathProvisioner object using the command line.

Prerequisites

  • The oc CLI tool is installed.
  • You are logged in with cluster administrator permissions.

Procedure

  1. Edit the object in your default editor by running the following command:

    $ oc edit <resource_type> <resource_name> -n {CNVNamespace}
  2. Save the file to apply the changes.

5.2.3. Node placement rule examples

You can specify node placement rules for a OpenShift Virtualization component by editing a Subscription, HyperConverged, or HostPathProvisioner object.

5.2.3.1. Subscription object node placement rule examples

To specify the nodes where OLM deploys the OpenShift Virtualization Operators, edit the Subscription object during OpenShift Virtualization installation.

Currently, you cannot configure node placement rules for the Subscription object by using the web console.

The Subscription object does not support the affinity node pplacement rule.

Example Subscription object with nodeSelector rule

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.17.3
  channel: "stable"
  config:
    nodeSelector:
      example.io/example-infra-key: example-infra-value 1

1
OLM deploys the OpenShift Virtualization Operators on nodes labeled example.io/example-infra-key = example-infra-value.

Example Subscription object with tolerations rule

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source:  redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.17.3
  channel: "stable"
  config:
    tolerations:
    - key: "key"
      operator: "Equal"
      value: "virtualization" 1
      effect: "NoSchedule"

1
OLM deploys OpenShift Virtualization Operators on nodes labeled key = virtualization:NoSchedule taint. Only pods with the matching tolerations are scheduled on these nodes.
5.2.3.2. HyperConverged object node placement rule example

To specify the nodes where OpenShift Virtualization deploys its components, you can edit the nodePlacement object in the HyperConverged custom resource (CR) file that you create during OpenShift Virtualization installation.

Example HyperConverged object with nodeSelector rule

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      nodeSelector:
        example.io/example-infra-key: example-infra-value 1
  workloads:
    nodePlacement:
      nodeSelector:
        example.io/example-workloads-key: example-workloads-value 2

1
Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-infra-value.
2
workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.

Example HyperConverged object with affinity rule

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-infra-key
                operator: In
                values:
                - example-infra-value 1
  workloads:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-workloads-key 2
                operator: In
                values:
                - example-workloads-value
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: example.io/num-cpus
                operator: Gt
                values:
                - 8 3

1
Infrastructure resources are placed on nodes labeled example.io/example-infra-key = example-value.
2
workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.
3
Nodes that have more than eight CPUs are preferred for workloads, but if they are not available, pods are still scheduled.

Example HyperConverged object with tolerations rule

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  workloads:
    nodePlacement:
      tolerations: 1
      - key: "key"
        operator: "Equal"
        value: "virtualization"
        effect: "NoSchedule"

1
Nodes reserved for OpenShift Virtualization components are labeled with the key = virtualization:NoSchedule taint. Only pods with matching tolerations are scheduled on reserved nodes.
5.2.3.3. HostPathProvisioner object node placement rule example

You can edit the HostPathProvisioner object directly or by using the web console.

Warning

You must schedule the hostpath provisioner and the OpenShift Virtualization components on the same nodes. Otherwise, virtualization pods that use the hostpath provisioner cannot run. You cannot run virtual machines.

After you deploy a virtual machine (VM) with the hostpath provisioner (HPP) storage class, you can remove the hostpath provisioner pod from the same node by using the node selector. However, you must first revert that change, at least for that specific node, and wait for the pod to run before trying to delete the VM.

You can configure node placement rules by specifying nodeSelector, affinity, or tolerations for the spec.workload field of the HostPathProvisioner object that you create when you install the hostpath provisioner.

Example HostPathProvisioner object with nodeSelector rule

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  pathConfig:
    path: "</path/to/backing/directory>"
    useNamingPrefix: false
  workload:
    nodeSelector:
      example.io/example-workloads-key: example-workloads-value 1

1
Workloads are placed on nodes labeled example.io/example-workloads-key = example-workloads-value.

5.2.4. Additional resources

5.3. Postinstallation network configuration

By default, OpenShift Virtualization is installed with a single, internal pod network.

After you install OpenShift Virtualization, you can install networking Operators and configure additional networks.

5.3.1. Installing networking Operators

You must install the Kubernetes NMState Operator to configure a Linux bridge network for live migration or external access to virtual machines (VMs). For installation instructions, see Installing the Kubernetes NMState Operator by using the web console.

You can install the SR-IOV Operator to manage SR-IOV network devices and network attachments. For installation instructions, see Installing the SR-IOV Network Operator.

You can add the About MetalLB and the MetalLB Operator to manage the lifecycle for an instance of MetalLB on your cluster. For installation instructions, see Installing the MetalLB Operator from the OperatorHub using the web console.

5.3.2. Configuring a Linux bridge network

After you install the Kubernetes NMState Operator, you can configure a Linux bridge network for live migration or external access to virtual machines (VMs).

5.3.2.1. Creating a Linux bridge NNCP

You can create a NodeNetworkConfigurationPolicy (NNCP) manifest for a Linux bridge network.

Prerequisites

  • You have installed the Kubernetes NMState Operator.

Procedure

  • Create the NodeNetworkConfigurationPolicy manifest. This example includes sample values that you must replace with your own information.

    apiVersion: nmstate.io/v1
    kind: NodeNetworkConfigurationPolicy
    metadata:
      name: br1-eth1-policy 1
    spec:
      desiredState:
        interfaces:
          - name: br1 2
            description: Linux bridge with eth1 as a port 3
            type: linux-bridge 4
            state: up 5
            ipv4:
              enabled: false 6
            bridge:
              options:
                stp:
                  enabled: false 7
              port:
                - name: eth1 8
    1
    Name of the policy.
    2
    Name of the interface.
    3
    Optional: Human-readable description of the interface.
    4
    The type of interface. This example creates a bridge.
    5
    The requested state for the interface after creation.
    6
    Disables IPv4 in this example.
    7
    Disables STP in this example.
    8
    The node NIC to which the bridge is attached.
5.3.2.2. Creating a Linux bridge NAD by using the web console

You can create a network attachment definition (NAD) to provide layer-2 networking to pods and virtual machines by using the OpenShift Container Platform web console.

A Linux bridge network attachment definition is the most efficient method for connecting a virtual machine to a VLAN.

Warning

Configuring IP address management (IPAM) in a network attachment definition for virtual machines is not supported.

Procedure

  1. In the web console, click NetworkingNetworkAttachmentDefinitions.
  2. Click Create Network Attachment Definition.

    Note

    The network attachment definition must be in the same namespace as the pod or virtual machine.

  3. Enter a unique Name and optional Description.
  4. Select CNV Linux bridge from the Network Type list.
  5. Enter the name of the bridge in the Bridge Name field.
  6. Optional: If the resource has VLAN IDs configured, enter the ID numbers in the VLAN Tag Number field.
  7. Optional: Select MAC Spoof Check to enable MAC spoof filtering. This feature provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod.
  8. Click Create.

5.3.3. Configuring a network for live migration

After you have configured a Linux bridge network, you can configure a dedicated network for live migration. A dedicated network minimizes the effects of network saturation on tenant workloads during live migration.

5.3.3.1. Configuring a dedicated secondary network for live migration

To configure a dedicated secondary network for live migration, you must first create a bridge network attachment definition (NAD) by using the CLI. Then, you add the name of the NetworkAttachmentDefinition object to the HyperConverged custom resource (CR).

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster as a user with the cluster-admin role.
  • Each node has at least two Network Interface Cards (NICs).
  • The NICs for live migration are connected to the same VLAN.

Procedure

  1. Create a NetworkAttachmentDefinition manifest according to the following example:

    Example configuration file

    apiVersion: "k8s.cni.cncf.io/v1"
    kind: NetworkAttachmentDefinition
    metadata:
      name: my-secondary-network 1
      namespace: openshift-cnv
    spec:
      config: '{
        "cniVersion": "0.3.1",
        "name": "migration-bridge",
        "type": "macvlan",
        "master": "eth1", 2
        "mode": "bridge",
        "ipam": {
          "type": "whereabouts", 3
          "range": "10.200.5.0/24" 4
        }
      }'

    1
    Specify the name of the NetworkAttachmentDefinition object.
    2
    Specify the name of the NIC to be used for live migration.
    3
    Specify the name of the CNI plugin that provides the network for the NAD.
    4
    Specify an IP address range for the secondary network. This range must not overlap the IP addresses of the main network.
  2. Open the HyperConverged CR in your default editor by running the following command:

    oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
  3. Add the name of the NetworkAttachmentDefinition object to the spec.liveMigrationConfig stanza of the HyperConverged CR:

    Example HyperConverged manifest

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
    spec:
      liveMigrationConfig:
        completionTimeoutPerGiB: 800
        network: <network> 1
        parallelMigrationsPerCluster: 5
        parallelOutboundMigrationsPerNode: 2
        progressTimeout: 150
    # ...

    1
    Specify the name of the Multus NetworkAttachmentDefinition object to be used for live migrations.
  4. Save your changes and exit the editor. The virt-handler pods restart and connect to the secondary network.

Verification

  • When the node that the virtual machine runs on is placed into maintenance mode, the VM automatically migrates to another node in the cluster. You can verify that the migration occurred over the secondary network and not the default pod network by checking the target IP address in the virtual machine instance (VMI) metadata.

    $ oc get vmi <vmi_name> -o jsonpath='{.status.migrationState.targetNodeAddress}'
5.3.3.2. Selecting a dedicated network by using the web console

You can select a dedicated network for live migration by using the OpenShift Container Platform web console.

Prerequisites

  • You configured a Multus network for live migration.
  • You created a network attachment definition for the network.

Procedure

  1. Navigate to Virtualization > Overview in the OpenShift Container Platform web console.
  2. Click the Settings tab and then click Live migration.
  3. Select the network from the Live migration network list.

5.3.4. Configuring an SR-IOV network

After you install the SR-IOV Operator, you can configure an SR-IOV network.

5.3.4.1. Configuring SR-IOV network devices

The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io CustomResourceDefinition to OpenShift Container Platform. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR).

Note

When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes. Reboot only happens in the following cases:

  • With Mellanox NICs (mlx5 driver) a node reboot happens every time the number of virtual functions (VFs) increase on a physical function (PF).
  • With Intel NICs, a reboot only happens if the kernel parameters do not include intel_iommu=on and iommu=pt.

It might take several minutes for a configuration change to apply.

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the SR-IOV Network Operator.
  • You have enough available nodes in your cluster to handle the evicted workload from drained nodes.
  • You have not selected any control plane nodes for SR-IOV network device configuration.

Procedure

  1. Create an SriovNetworkNodePolicy object, and then save the YAML in the <name>-sriov-node-network.yaml file. Replace <name> with the name for this configuration.

    apiVersion: sriovnetwork.openshift.io/v1
    kind: SriovNetworkNodePolicy
    metadata:
      name: <name> 1
      namespace: openshift-sriov-network-operator 2
    spec:
      resourceName: <sriov_resource_name> 3
      nodeSelector:
        feature.node.kubernetes.io/network-sriov.capable: "true" 4
      priority: <priority> 5
      mtu: <mtu> 6
      numVfs: <num> 7
      nicSelector: 8
        vendor: "<vendor_code>" 9
        deviceID: "<device_id>" 10
        pfNames: ["<pf_name>", ...] 11
        rootDevices: ["<pci_bus_id>", "..."] 12
      deviceType: vfio-pci 13
      isRdma: false 14
    1
    Specify a name for the CR object.
    2
    Specify the namespace where the SR-IOV Operator is installed.
    3
    Specify the resource name of the SR-IOV device plugin. You can create multiple SriovNetworkNodePolicy objects for a resource name.
    4
    Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plugin and device plugin are deployed only on selected nodes.
    5
    Optional: Specify an integer value between 0 and 99. A smaller number gets higher priority, so a priority of 10 is higher than a priority of 99. The default value is 99.
    6
    Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.
    7
    Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than 127.
    8
    The nicSelector mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specify rootDevices, you must also specify a value for vendor, deviceID, or pfNames. If you specify both pfNames and rootDevices at the same time, ensure that they point to an identical device.
    9
    Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either 8086 or 15b3.
    10
    Optional: Specify the device hex code of SR-IOV network device. The only allowed values are 158b, 1015, 1017.
    11
    Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device.
    12
    The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format: 0000:02:00.1.
    13
    The vfio-pci driver type is required for virtual functions in OpenShift Virtualization.
    14
    Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set isRdma to false. The default value is false.
    Note

    If isRDMA flag is set to true, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.

  2. Optional: Label the SR-IOV capable cluster nodes with SriovNetworkNodePolicy.Spec.NodeSelector if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".
  3. Create the SriovNetworkNodePolicy object:

    $ oc create -f <name>-sriov-node-network.yaml

    where <name> specifies the name for this configuration.

    After applying the configuration update, all the pods in sriov-network-operator namespace transition to the Running status.

  4. To verify that the SR-IOV network device is configured, enter the following command. Replace <node_name> with the name of a node with the SR-IOV network device that you just configured.

    $ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'

5.3.5. Enabling load balancer service creation by using the web console

You can enable the creation of load balancer services for a virtual machine (VM) by using the OpenShift Container Platform web console.

Prerequisites

  • You have configured a load balancer for the cluster.
  • You are logged in as a user with the cluster-admin role.
  • You created a network attachment definition for the network.

Procedure

  1. Navigate to VirtualizationOverview.
  2. On the Settings tab, click Cluster.
  3. Expand General settings and SSH configuration.
  4. Set SSH over LoadBalancer service to on.

5.4. Postinstallation storage configuration

The following storage configuration tasks are mandatory:

  • You must configure a default storage class for your cluster. Otherwise, the cluster cannot receive automated boot source updates.
  • You must configure storage profiles if your storage provider is not recognized by CDI. A storage profile provides recommended storage settings based on the associated storage class.

Optional: You can configure local storage by using the hostpath provisioner (HPP).

See the storage configuration overview for more options, including configuring the Containerized Data Importer (CDI), data volumes, and automatic boot source updates.

5.4.1. Configuring local storage by using the HPP

When you install the OpenShift Virtualization Operator, the Hostpath Provisioner (HPP) Operator is automatically installed. The HPP Operator creates the HPP provisioner.

The HPP is a local storage provisioner designed for OpenShift Virtualization. To use the HPP, you must create an HPP custom resource (CR).

Important

HPP storage pools must not be in the same partition as the operating system. Otherwise, the storage pools might fill the operating system partition. If the operating system partition is full, performance can be effected or the node can become unstable or unusable.

5.4.1.1. Creating a storage class for the CSI driver with the storagePools stanza

To use the hostpath provisioner (HPP) you must create an associated storage class for the Container Storage Interface (CSI) driver.

When you create a storage class, you set parameters that affect the dynamic provisioning of persistent volumes (PVs) that belong to that storage class. You cannot update a StorageClass object’s parameters after you create it.

Note

Virtual machines use data volumes that are based on local PVs. Local PVs are bound to specific nodes. While a disk image is prepared for consumption by the virtual machine, it is possible that the virtual machine cannot be scheduled to the node where the local storage PV was previously pinned.

To solve this problem, use the Kubernetes pod scheduler to bind the persistent volume claim (PVC) to a PV on the correct node. By using the StorageClass value with volumeBindingMode parameter set to WaitForFirstConsumer, the binding and provisioning of the PV is delayed until a pod is created using the PVC.

Procedure

  1. Create a storageclass_csi.yaml file to define the storage class:

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: hostpath-csi
    provisioner: kubevirt.io.hostpath-provisioner
    reclaimPolicy: Delete 1
    volumeBindingMode: WaitForFirstConsumer 2
    parameters:
      storagePool: my-storage-pool 3
    1
    The two possible reclaimPolicy values are Delete and Retain. If you do not specify a value, the default value is Delete.
    2
    The volumeBindingMode parameter determines when dynamic provisioning and volume binding occur. Specify WaitForFirstConsumer to delay the binding and provisioning of a persistent volume (PV) until after a pod that uses the persistent volume claim (PVC) is created. This ensures that the PV meets the pod’s scheduling requirements.
    3
    Specify the name of the storage pool defined in the HPP CR.
  2. Save the file and exit.
  3. Create the StorageClass object by running the following command:

    $ oc create -f storageclass_csi.yaml

5.5. Configuring higher VM workload density

You can increase the number of virtual machines (VMs) on nodes by overcommitting memory (RAM). Increasing VM workload density can be useful in the following situations:

  • You have many similar workloads.
  • You have underused workloads.
Note

Memory overcommitment can lower workload performance on a highly utilized system.

5.5.1. Using wasp-agent to increase VM workload density

The wasp-agent component facilitates memory overcommitment by assigning swap resources to worker nodes. It also manages pod evictions when nodes are at risk due to high swap I/O traffic or high utilization.

Important

Swap resources can be only assigned to virtual machine workloads (VM pods) of the Burstable Quality of Service (QoS) class. VM pods of the Guaranteed QoS class and pods of any QoS class that do not belong to VMs cannot swap resources.

For descriptions of QoS classes, see Configure Quality of Service for Pods (Kubernetes documentation).

Prerequisites

  • You have installed the OpenShift CLI (oc).
  • You are logged into the cluster with the cluster-admin role.
  • A memory overcommit ratio is defined.
  • The node belongs to a worker pool.
Note

The wasp-agent component deploys an Open Container Initiative (OCI) hook to enable swap usage for containers on the node level. The low-level nature requires the DaemonSet object to be privileged.

Procedure

  1. Configure the kubelet service to permit swap usage:

    1. Create or edit a KubeletConfig file with the parameters shown in the following example:

      Example of a KubeletConfig file

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-config
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: ''  # MCP
            #machine.openshift.io/cluster-api-machine-role: worker # machine
            #node-role.kubernetes.io/worker: '' # node
        kubeletConfig:
          failSwapOn: false

    2. Wait for the worker nodes to sync with the new configuration by running the following command:

      $ oc wait mcp worker --for condition=Updated=True --timeout=-1s
  2. Provision swap by creating a MachineConfig object. For example:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 90-worker-swap
    spec:
      config:
        ignition:
          version: 3.4.0
        systemd:
          units:
            - contents: |
                [Unit]
                Description=Provision and enable swap
                ConditionFirstBoot=no
    
                [Service]
                Type=oneshot
                Environment=SWAP_SIZE_MB=5000
                ExecStart=/bin/sh -c "sudo dd if=/dev/zero of=/var/tmp/swapfile count=${SWAP_SIZE_MB} bs=1M && \
                sudo chmod 600 /var/tmp/swapfile && \
                sudo mkswap /var/tmp/swapfile && \
                sudo swapon /var/tmp/swapfile && \
                free -h && \
                sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""
    
                [Install]
                RequiredBy=kubelet-dependencies.target
              enabled: true
              name: swap-provision.service

    To have enough swap space for the worst-case scenario, make sure to have at least as much swap space provisioned as overcommitted RAM. Calculate the amount of swap space to be provisioned on a node by using the following formula:

    NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

    Example

    NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
                   = 16 GB * (1.5 - 1)
                   = 16 GB * (0.5)
                   =  8 GB

  3. Create a privileged service account by running the following commands:

    $ oc adm new-project wasp
    $ oc create sa -n wasp wasp
    $ oc create clusterrolebinding wasp --clusterrole=cluster-admin --serviceaccount=wasp:wasp
    $ oc adm policy add-scc-to-user -n wasp privileged -z wasp
  4. Wait for the worker nodes to sync with the new configuration by running the following command:

    $ oc wait mcp worker --for condition=Updated=True --timeout=-1s
  5. Determine the pull URL for the wasp agent image by running the following command:

    $ oc get csv -n openshift-cnv -l=operators.coreos.com/kubevirt-hyperconverged.openshift-cnv -ojson | jq '.items[0].spec.relatedImages[] | select(.name|test(".*wasp-agent.*")) | .image'
  6. Deploy wasp-agent by creating a DaemonSet object as shown in the following example:

    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: wasp-agent
      namespace: wasp
      labels:
        app: wasp
        tier: node
    spec:
      selector:
        matchLabels:
          name: wasp
      template:
        metadata:
          annotations:
            description: >-
              Configures swap for workloads
          labels:
            name: wasp
        spec:
          containers:
            - env:
                - name: SWAP_UTILIZATION_THRESHOLD_FACTOR
                  value: "0.8"
                - name: MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND
                  value: "1000"
                - name: MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND
                  value: "1000"
                - name: AVERAGE_WINDOW_SIZE_SECONDS
                  value: "30"
                - name: VERBOSITY
                  value: "1"
                - name: FSROOT
                  value: /host
                - name: NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
              image: >-
                quay.io/openshift-virtualization/wasp-agent:v4.17 1
              imagePullPolicy: Always
              name: wasp-agent
              resources:
                requests:
                  cpu: 100m
                  memory: 50M
              securityContext:
                privileged: true
              volumeMounts:
                - mountPath: /host
                  name: host
                - mountPath: /rootfs
                  name: rootfs
          hostPID: true
          hostUsers: true
          priorityClassName: system-node-critical
          serviceAccountName: wasp
          terminationGracePeriodSeconds: 5
          volumes:
            - hostPath:
                path: /
              name: host
            - hostPath:
                path: /
              name: rootfs
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 10%
          maxSurge: 0
    1
    Replace the image value with the image URL from the previous step.
  7. Deploy alerting rules by creating a PrometheusRule object. For example:

    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      labels:
        tier: node
        wasp.io: ""
      name: wasp-rules
      namespace: wasp
    spec:
      groups:
        - name: alerts.rules
          rules:
            - alert: NodeHighSwapActivity
              annotations:
                description: High swap activity detected at {{ $labels.instance }}. The rate
                  of swap out and swap in exceeds 200 in both operations in the last minute.
                  This could indicate memory pressure and may affect system performance.
                runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/docs/runbooks/NodeHighSwapActivity.md
                summary: High swap activity detected at {{ $labels.instance }}.
              expr: rate(node_vmstat_pswpout[1m]) > 200 and rate(node_vmstat_pswpin[1m]) >
                200
              for: 1m
              labels:
                kubernetes_operator_component: kubevirt
                kubernetes_operator_part_of: kubevirt
                operator_health_impact: warning
                severity: warning
  8. Add the cluster-monitoring label to the wasp namespace by running the following command:

    $ oc label namespace wasp openshift.io/cluster-monitoring="true"
  9. Enable memory overcommitment in OpenShift Virtualization by using the web console or the CLI.

    • Web console

      1. In the OpenShift Container Platform web console, go to VirtualizationOverviewSettingsGeneral settingsMemory density.
      2. Set Enable memory density to on.
    • CLI

      • Configure your OpenShift Virtualization to enable higher memory density and set the overcommit rate:

        $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \
          { \
          "op": "replace", \
          "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \
          "value": 150 \
          } \
        ]'

        Successful output

        hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

Verification

  1. To verify the deployment of wasp-agent, run the following command:

    $ oc rollout status ds wasp-agent -n wasp

    If the deployment is successful, the following message is displayed:

    Example output

    daemon set "wasp-agent" successfully rolled out

  2. To verify that swap is correctly provisioned, complete the following steps:

    1. View a list of worker nodes by running the following command:

      $ oc get nodes -l node-role.kubernetes.io/worker
    2. Select a node from the list and display its memory usage by running the following command:

      $ oc debug node/<selected_node> -- free -m 1
      1
      Replace <selected_node> with the node name.

      If swap is provisioned, an amount greater than zero is displayed in the Swap: row.

      Table 5.1. Example output
       

      total

      used

      free

      shared

      buff/cache

      available

      Mem:

      31846

      23155

      1044

      6014

      14483

      8690

      Swap:

      8191

      2337

      5854

         
  3. Verify the OpenShift Virtualization memory overcommitment configuration by running the following command:

    $ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'

    Example output

    {"memoryOvercommitPercentage":150}

    The returned value must match the value you had previously configured.

5.5.2. Pod eviction conditions used by wasp-agent

The wasp agent manages pod eviction when the system is heavily loaded and nodes are at risk. Eviction is triggered if one of the following conditions is met:

High swap I/O traffic

This condition is met when swap-related I/O traffic is excessively high.

Condition

averageSwapInPerSecond > maxAverageSwapInPagesPerSecond
&&
averageSwapOutPerSecond > maxAverageSwapOutPagesPerSecond

By default, maxAverageSwapInPagesPerSecond and maxAverageSwapOutPagesPerSecond are set to 1000 pages. The default time interval for calculating the average is 30 seconds.

High swap utilization

This condition is met when swap utilization is excessively high, causing the current virtual memory usage to exceed the factored threshold. The NODE_SWAP_SPACE setting in your MachineConfig object can impact this condition.

Condition

nodeWorkingSet + nodeSwapUsage < totalNodeMemory + totalSwapMemory × thresholdFactor

5.5.2.1. Environment variables

You can use the following environment variables to adjust the values used to calculate eviction conditions:

Environment variable

Function

MAX_AVERAGE_SWAP_IN_PAGES_PER_SECOND

Sets the value of maxAverageSwapInPagesPerSecond.

MAX_AVERAGE_SWAP_OUT_PAGES_PER_SECOND

Sets the value of maxAverageSwapOutPagesPerSecond.

SWAP_UTILIZATION_THRESHOLD_FACTOR

Sets the thresholdFactor value used to calculate high swap utilization.

AVERAGE_WINDOW_SIZE_SECONDS

Sets the time interval for calculating the average swap usage.

Chapter 6. Updating

6.1. Updating OpenShift Virtualization

Learn how Operator Lifecycle Manager (OLM) delivers z-stream and minor version updates for OpenShift Virtualization.

6.1.1. OpenShift Virtualization on RHEL 9

OpenShift Virtualization 4.17 is based on Red Hat Enterprise Linux (RHEL) 9. You can update to OpenShift Virtualization 4.17 from a version that was based on RHEL 8 by following the standard OpenShift Virtualization update procedure. No additional steps are required.

As in previous versions, you can perform the update without disrupting running workloads. OpenShift Virtualization 4.17 supports live migration from RHEL 8 nodes to RHEL 9 nodes.

6.1.1.1. RHEL 9 machine type

All VM templates that are included with OpenShift Virtualization now use the RHEL 9 machine type by default: machineType: pc-q35-rhel9.<y>.0, where <y> is a single digit corresponding to the latest minor version of RHEL 9. For example, the value pc-q35-rhel9.2.0 is used for RHEL 9.2.

Updating OpenShift Virtualization does not change the machineType value of any existing VMs. These VMs continue to function as they did before the update. You can optionally change a VM’s machine type so that it can benefit from RHEL 9 improvements.

Important

Before you change a VM’s machineType value, you must shut down the VM.

6.1.2. About updating OpenShift Virtualization

  • Operator Lifecycle Manager (OLM) manages the lifecycle of the OpenShift Virtualization Operator. The Marketplace Operator, which is deployed during OpenShift Container Platform installation, makes external Operators available to your cluster.
  • OLM provides z-stream and minor version updates for OpenShift Virtualization. Minor version updates become available when you update OpenShift Container Platform to the next minor version. You cannot update OpenShift Virtualization to the next minor version without first updating OpenShift Container Platform.
  • OpenShift Virtualization subscriptions use a single update channel that is named stable. The stable channel ensures that your OpenShift Virtualization and OpenShift Container Platform versions are compatible.
  • If your subscription’s approval strategy is set to Automatic, the update process starts as soon as a new version of the Operator is available in the stable channel. It is highly recommended to use the Automatic approval strategy to maintain a supportable environment. Each minor version of OpenShift Virtualization is only supported if you run the corresponding OpenShift Container Platform version. For example, you must run OpenShift Virtualization 4.17 on OpenShift Container Platform 4.17.

    • Though it is possible to select the Manual approval strategy, this is not recommended because it risks the supportability and functionality of your cluster. With the Manual approval strategy, you must manually approve every pending update. If OpenShift Container Platform and OpenShift Virtualization updates are out of sync, your cluster becomes unsupported.
  • The amount of time an update takes to complete depends on your network connection. Most automatic updates complete within fifteen minutes.
  • Updating OpenShift Virtualization does not interrupt network connections.
  • Data volumes and their associated persistent volume claims are preserved during update.
Important

If you have virtual machines running that use hostpath provisioner storage, they cannot be live migrated and might block an OpenShift Container Platform cluster update.

As a workaround, you can reconfigure the virtual machines so that they can be powered off automatically during a cluster update. Set the evictionStrategy field to None and the runStrategy field to Always.

6.1.2.1. About workload updates

When you update OpenShift Virtualization, virtual machine workloads, including libvirt, virt-launcher, and qemu, update automatically if they support live migration.

Note

Each virtual machine has a virt-launcher pod that runs the virtual machine instance (VMI). The virt-launcher pod runs an instance of libvirt, which is used to manage the virtual machine (VM) process.

You can configure how workloads are updated by editing the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource (CR). There are two available workload update methods: LiveMigrate and Evict.

Because the Evict method shuts down VMI pods, only the LiveMigrate update strategy is enabled by default.

When LiveMigrate is the only update strategy enabled:

  • VMIs that support live migration are migrated during the update process. The VM guest moves into a new pod with the updated components enabled.
  • VMIs that do not support live migration are not disrupted or updated.

    • If a VMI has the LiveMigrate eviction strategy but does not support live migration, it is not updated.

If you enable both LiveMigrate and Evict:

  • VMIs that support live migration use the LiveMigrate update strategy.
  • VMIs that do not support live migration use the Evict update strategy. If a VMI is controlled by a VirtualMachine object that has runStrategy: Always set, a new VMI is created in a new pod with updated components.
Migration attempts and timeouts

When updating workloads, live migration fails if a pod is in the Pending state for the following periods:

5 minutes
If the pod is pending because it is Unschedulable.
15 minutes
If the pod is stuck in the pending state for any reason.

When a VMI fails to migrate, the virt-controller tries to migrate it again. It repeats this process until all migratable VMIs are running on new virt-launcher pods. If a VMI is improperly configured, however, these attempts can repeat indefinitely.

Note

Each attempt corresponds to a migration object. Only the five most recent attempts are held in a buffer. This prevents migration objects from accumulating on the system while retaining information for debugging.

6.1.2.2. About Control Plane Only updates

Every even-numbered minor version of OpenShift Container Platform, including 4.10 and 4.12, is an Extended Update Support (EUS) version. However, because Kubernetes design mandates serial minor version updates, you cannot directly update from one EUS version to the next.

After you update from the source EUS version to the next odd-numbered minor version, you must sequentially update OpenShift Virtualization to all z-stream releases of that minor version that are on your update path. When you have upgraded to the latest applicable z-stream version, you can then update OpenShift Container Platform to the target EUS minor version.

When the OpenShift Container Platform update succeeds, the corresponding update for OpenShift Virtualization becomes available. You can now update OpenShift Virtualization to the target EUS version.

6.1.2.2.1. Preparing to update

Before beginning a Control Plane Only update, you must:

  • Pause worker nodes' machine config pools before you start a Control Plane Only update so that the workers are not rebooted twice.
  • Disable automatic workload updates before you begin the update process. This is to prevent OpenShift Virtualization from migrating or evicting your virtual machines (VMs) until you update to your target EUS version.
Note

By default, OpenShift Virtualization automatically updates workloads, such as the virt-launcher pod, when you update the OpenShift Virtualization Operator. You can configure this behavior in the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource.

Learn more about Performing a Control Plane Only update.

6.1.3. Preventing workload updates during a Control Plane Only update

When you update from one Extended Update Support (EUS) version to the next, you must manually disable automatic workload updates to prevent OpenShift Virtualization from migrating or evicting workloads during the update process.

Prerequisites

  • You are running an EUS version of OpenShift Container Platform and want to update to the next EUS version. You have not yet updated to the odd-numbered version in between.
  • You read "Preparing to perform a Control Plane Only update" and learned the caveats and requirements that pertain to your OpenShift Container Platform cluster.
  • You paused the worker nodes' machine config pools as directed by the OpenShift Container Platform documentation.
  • It is recommended that you use the default Automatic approval strategy. If you use the Manual approval strategy, you must approve all pending updates in the web console. For more details, refer to the "Manually approving a pending Operator update" section.

Procedure

  1. Run the following command and record the workloadUpdateMethods configuration:

    $ oc get kv kubevirt-kubevirt-hyperconverged \
      -n openshift-cnv -o jsonpath='{.spec.workloadUpdateStrategy.workloadUpdateMethods}'
  2. Turn off all workload update methods by running the following command:

    $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
      --type json -p '[{"op":"replace","path":"/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]'

    Example output

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

  3. Ensure that the HyperConverged Operator is Upgradeable before you continue. Enter the following command and monitor the output:

    $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"

    Example 6.1. Example output

    [
      {
        "lastTransitionTime": "2022-12-09T16:29:11Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "ReconcileComplete"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "Available"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "False",
        "type": "Progressing"
      },
      {
        "lastTransitionTime": "2022-12-09T16:39:11Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "False",
        "type": "Degraded"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "Upgradeable" 1
      }
    ]
    1
    The OpenShift Virtualization Operator has the Upgradeable status.
  4. Manually update your cluster from the source EUS version to the next minor version of OpenShift Container Platform:

    $ oc adm upgrade

    Verification

    • Check the current version by running the following command:

      $ oc get clusterversion
      Note

      Updating OpenShift Container Platform to the next version is a prerequisite for updating OpenShift Virtualization. For more details, refer to the "Updating clusters" section of the OpenShift Container Platform documentation.

  5. Update OpenShift Virtualization.

    • With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform.
    • If you use the Manual approval strategy, approve the pending updates by using the web console.
  6. Monitor the OpenShift Virtualization update by running the following command:

    $ oc get csv -n openshift-cnv
  7. Update OpenShift Virtualization to every z-stream version that is available for the non-EUS minor version, monitoring each update by running the command shown in the previous step.
  8. Confirm that OpenShift Virtualization successfully updated to the latest z-stream release of the non-EUS version by running the following command:

    $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"

    Example output

    [
      {
        "name": "operator",
        "version": "4.17.3"
      }
    ]

  9. Wait until the HyperConverged Operator has the Upgradeable status before you perform the next update. Enter the following command and monitor the output:

    $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
  10. Update OpenShift Container Platform to the target EUS version.
  11. Confirm that the update succeeded by checking the cluster version:

    $ oc get clusterversion
  12. Update OpenShift Virtualization to the target EUS version.

    • With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform.
    • If you use the Manual approval strategy, approve the pending updates by using the web console.
  13. Monitor the OpenShift Virtualization update by running the following command:

    $ oc get csv -n openshift-cnv

    The update completes when the VERSION field matches the target EUS version and the PHASE field reads Succeeded.

  14. Restore the workloadUpdateMethods configuration that you recorded from step 1 with the following command:

    $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p \
      "[{\"op\":\"add\",\"path\":\"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\":{WorkloadUpdateMethodConfig}}]"

    Example output

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

    Verification

    • Check the status of VM migration by running the following command:

      $ oc get vmim -A

Next steps

  • You can now unpause the worker nodes' machine config pools.

6.1.4. Configuring workload update methods

You can configure workload update methods by editing the HyperConverged custom resource (CR).

Prerequisites

  • To use live migration as an update method, you must first enable live migration in the cluster.

    Note

    If a VirtualMachineInstance CR contains evictionStrategy: LiveMigrate and the virtual machine instance (VMI) does not support live migration, the VMI will not update.

Procedure

  1. To open the HyperConverged CR in your default editor, run the following command:

    $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
  2. Edit the workloadUpdateStrategy stanza of the HyperConverged CR. For example:

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
    spec:
      workloadUpdateStrategy:
        workloadUpdateMethods: 1
        - LiveMigrate 2
        - Evict 3
        batchEvictionSize: 10 4
        batchEvictionInterval: "1m0s" 5
    # ...
    1
    The methods that can be used to perform automated workload updates. The available values are LiveMigrate and Evict. If you enable both options as shown in this example, updates use LiveMigrate for VMIs that support live migration and Evict for any VMIs that do not support live migration. To disable automatic workload updates, you can either remove the workloadUpdateStrategy stanza or set workloadUpdateMethods: [] to leave the array empty.
    2
    The least disruptive update method. VMIs that support live migration are updated by migrating the virtual machine (VM) guest into a new pod with the updated components enabled. If LiveMigrate is the only workload update method listed, VMIs that do not support live migration are not disrupted or updated.
    3
    A disruptive method that shuts down VMI pods during upgrade. Evict is the only update method available if live migration is not enabled in the cluster. If a VMI is controlled by a VirtualMachine object that has runStrategy: Always configured, a new VMI is created in a new pod with updated components.
    4
    The number of VMIs that can be forced to be updated at a time by using the Evict method. This does not apply to the LiveMigrate method.
    5
    The interval to wait before evicting the next batch of workloads. This does not apply to the LiveMigrate method.
    Note

    You can configure live migration limits and timeouts by editing the spec.liveMigrationConfig stanza of the HyperConverged CR.

  3. To apply your changes, save and exit the editor.

6.1.5. Approving pending Operator updates

6.1.5.1. Manually approving a pending Operator update

If an installed Operator has the approval strategy in its subscription set to Manual, when new updates are released in its current update channel, the update must be manually approved before installation can begin.

Prerequisites

  • An Operator previously installed using Operator Lifecycle Manager (OLM).

Procedure

  1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
  2. Operators that have a pending update display a status with Upgrade available. Click the name of the Operator you want to update.
  3. Click the Subscription tab. Any updates requiring approval are displayed next to Upgrade status. For example, it might display 1 requires approval.
  4. Click 1 requires approval, then click Preview Install Plan.
  5. Review the resources that are listed as available for update. When satisfied, click Approve.
  6. Navigate back to the Operators → Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

6.1.6. Monitoring update status

6.1.6.1. Monitoring OpenShift Virtualization upgrade status

To monitor the status of a OpenShift Virtualization Operator upgrade, watch the cluster service version (CSV) PHASE. You can also monitor the CSV conditions in the web console or by running the command provided here.

Note

The PHASE and conditions values are approximations that are based on available information.

Prerequisites

  • Log in to the cluster as a user with the cluster-admin role.
  • Install the OpenShift CLI (oc).

Procedure

  1. Run the following command:

    $ oc get csv -n openshift-cnv
  2. Review the output, checking the PHASE field. For example:

    Example output

    VERSION  REPLACES                                        PHASE
    4.9.0    kubevirt-hyperconverged-operator.v4.8.2         Installing
    4.9.0    kubevirt-hyperconverged-operator.v4.9.0         Replacing

  3. Optional: Monitor the aggregated status of all OpenShift Virtualization component conditions by running the following command:

    $ oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv \
      -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

    A successful upgrade results in the following output:

    Example output

    ReconcileComplete  True  Reconcile completed successfully
    Available          True  Reconcile completed successfully
    Progressing        False Reconcile completed successfully
    Degraded           False Reconcile completed successfully
    Upgradeable        True  Reconcile completed successfully

6.1.6.2. Viewing outdated OpenShift Virtualization workloads

You can view a list of outdated workloads by using the CLI.

Note

If there are outdated virtualization pods in your cluster, the OutdatedVirtualMachineInstanceWorkloads alert fires.

Procedure

  • To view a list of outdated virtual machine instances (VMIs), run the following command:

    $ oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces
Note

Configure workload updates to ensure that VMIs update automatically.

6.1.7. Additional resources

Chapter 7. Virtual machines

7.1. Creating VMs from Red Hat images

7.1.1. Creating virtual machines from Red Hat images overview

Red Hat images are golden images. They are published as container disks in a secure registry. The Containerized Data Importer (CDI) polls and imports the container disks into your cluster and stores them in the openshift-virtualization-os-images project as snapshots or persistent volume claims (PVCs). You can optionally use a custom namespace for golden images.

Red Hat images are automatically updated. You can disable and re-enable automatic updates for these images. See Managing Red Hat boot source updates.

Cluster administrators can enable automatic subscription for Red Hat Enterprise Linux (RHEL) virtual machines in the OpenShift Virtualization web console.

You can create virtual machines (VMs) from operating system images provided by Red Hat by using one of the following methods:

Important

Do not create VMs in the default openshift-* namespaces. Instead, create a new namespace or use an existing namespace without the openshift prefix.

7.1.1.1. About golden images

A golden image is a preconfigured snapshot of a virtual machine (VM) that you can use as a resource to deploy new VMs. For example, you can use golden images to provision the same system environment consistently and deploy systems more quickly and efficiently.

7.1.1.1.1. How do golden images work?

Golden images are created by installing and configuring an operating system and software applications on a reference machine or virtual machine. This includes setting up the system, installing required drivers, applying patches and updates, and configuring specific options and preferences.

After the golden image is created, it is saved as a template or image file that can be replicated and deployed across multiple clusters. The golden image can be updated by its maintainer periodically to incorporate necessary software updates and patches, ensuring that the image remains up to date and secure, and newly created VMs are based on this updated image.

7.1.1.1.2. Red Hat implementation of golden images

Red Hat publishes golden images as container disks in the registry for versions of Red Hat Enterprise Linux (RHEL). Container disks are virtual machine images that are stored as a container image in a container image registry. Any published image will automatically be made available in connected clusters after the installation of OpenShift Virtualization. After the images are available in a cluster, they are ready to use to create VMs.

7.1.1.2. About VM boot sources

Virtual machines (VMs) consist of a VM definition and one or more disks that are backed by data volumes. VM templates enable you to create VMs using predefined specifications.

Every template requires a boot source, which is a fully configured disk image including configured drivers. Each template contains a VM definition with a pointer to the boot source. Each boot source has a predefined name and namespace. For some operating systems, a boot source is automatically provided. If it is not provided, then an administrator must prepare a custom boot source.

Provided boot sources are updated automatically to the latest version of the operating system. For auto-updated boot sources, persistent volume claims (PVCs) and volume snapshots are created with the cluster’s default storage class. If you select a different default storage class after configuration, you must delete the existing boot sources in the cluster namespace that are configured with the previous default storage class.

7.1.1.3. Configuring a custom namespace for golden images

The default namespace for golden images is openshift-virtualization-os-images, but you can configure a custom namespace to restrict user access to the default boot sources.

7.1.1.3.1. Configuring a custom namespace for golden images by using the web console

You can configure a custom namespace for golden images in your cluster by using the OpenShift Container Platform web console.

Procedure

  1. In the web console, select VirtualizationOverview.
  2. Select the Settings tab.
  3. On the Cluster tab, select General settingsBootable volumes project.
  4. Select a namespace to use for golden images.

    1. If you already created a namespace, select it from the Project list.
    2. If you did not create a namespace, scroll to the bottom of the list and click Create project.

      1. Enter a name for your new namespace in the Name field of the Create project dialog.
      2. Click Create.
7.1.1.3.2. Configuring a custom namespace for golden images by using the CLI

You can configure a custom namespace for golden images in your cluster by setting the spec.commonBootImageNamespace field in the HyperConverged custom resource (CR).

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You created a namespace to use for golden images.

Procedure

  1. Open the HyperConverged CR in your default editor by running the following command:

    $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
  2. Configure the custom namespace by updating the value of the spec.commonBootImageNamespace field:

    Example configuration file

    apiVersion: hco.kubevirt.io/v1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
      namespace: openshift-cnv
    spec:
      commonBootImageNamespace: <custom_namespace> 1
    # ...

    1
    The namespace to use for golden images.
  3. Save your changes and exit the editor.

7.1.2. Creating virtual machines from instance types

You can simplify virtual machine (VM) creation by using instance types, whether you use the OpenShift Container Platform web console or the CLI to create VMs.

7.1.2.1. About instance types

An instance type is a reusable object where you can define resources and characteristics to apply to new VMs. You can define custom instance types or use the variety that are included when you install OpenShift Virtualization.

To create a new instance type, you must first create a manifest, either manually or by using the virtctl CLI tool. You then create the instance type object by applying the manifest to your cluster.

OpenShift Virtualization provides two CRDs for configuring instance types:

  • A namespaced object: VirtualMachineInstancetype
  • A cluster-wide object: VirtualMachineClusterInstancetype

These objects use the same VirtualMachineInstancetypeSpec.

7.1.2.1.1. Required attributes

When you configure an instance type, you must define the cpu and memory attributes. Other attributes are optional.

Note

When you create a VM from an instance type, you cannot override any parameters defined in the instance type.

Because instance types require defined CPU and memory attributes, OpenShift Virtualization always rejects additional requests for these resources when creating a VM from an instance type.

You can manually create an instance type manifest. For example:

Example YAML file with required fields

apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineInstancetype
metadata:
  name: example-instancetype
spec:
  cpu:
    guest: 1 1
  memory:
    guest: 128Mi 2

1
Required. Specifies the number of vCPUs to allocate to the guest.
2
Required. Specifies an amount of memory to allocate to the guest.

You can create an instance type manifest by using the virtctl CLI utility. For example:

Example virtctl command with required fields

$ virtctl create instancetype --cpu 2 --memory 256Mi

where:

--cpu <value>
Specifies the number of vCPUs to allocate to the guest. Required.
--memory <value>
Specifies an amount of memory to allocate to the guest. Required.
Tip

You can immediately create the object from the new manifest by running the following command:

$ virtctl create instancetype --cpu 2 --memory 256Mi | oc apply -f -
7.1.2.1.2. Optional attributes

In addition to the required cpu and memory attributes, you can include the following optional attributes in the VirtualMachineInstancetypeSpec:

annotations
List annotations to apply to the VM.
gpus
List vGPUs for passthrough.
hostDevices
List host devices for passthrough.
ioThreadsPolicy
Define an IO threads policy for managing dedicated disk access.
launchSecurity
Configure Secure Encrypted Virtualization (SEV).
nodeSelector
Specify node selectors to control the nodes where this VM is scheduled.
schedulerName
Define a custom scheduler to use for this VM instead of the default scheduler.
7.1.2.2. Pre-defined instance types

OpenShift Virtualization includes a set of pre-defined instance types called common-instancetypes. Some are specialized for specific workloads and others are workload-agnostic.

These instance type resources are named according to their series, version, and size. The size value follows the . delimiter and ranges from nano to 8xlarge.

Table 7.1. common-instancetypes series comparison
Use caseSeriesCharacteristicsvCPU to memory ratioExample resource

Universal

U

  • Burstable CPU performance

1:4

u1.medium
  • 1 vCPUs
  • 4 Gi memory

Overcommitted

O

  • Overcommitted memory
  • Burstable CPU performance

1:4

o1.small
  • 1 vCPU
  • 2Gi memory

Compute-exclusive

CX

  • Hugepages
  • Dedicated CPU
  • Isolated emulator threads
  • vNUMA

1:2

cx1.2xlarge
  • 8 vCPUs
  • 16Gi memory

NVIDIA GPU

GN

  • For VMs that use GPUs provided by the NVIDIA GPU Operator
  • Has predefined GPUs
  • Burstable CPU performance

1:4

gn1.8xlarge
  • 32 vCPUs
  • 128Gi memory

Memory-intensive

M

  • Hugepages
  • Burstable CPU performance

1:8

m1.large
  • 2 vCPUs
  • 16Gi memory

Network-intensive

N

  • Hugepages
  • Dedicated CPU
  • Isolated emulator threads
  • Requires nodes capable of running DPDK workloads

1:2

n1.medium
  • 4 vCPUs
  • 4Gi memory
7.1.2.3. Creating manifests by using the virtctl tool

You can use the virtctl CLI utility to simplify creating manifests for VMs, VM instance types, and VM preferences. For more information, see VM manifest creation commands.

If you have a VirtualMachine manifest, you can create a VM from the command line.

7.1.2.4. Creating a VM from an instance type by using the web console

You can create a virtual machine (VM) from an instance type by using the OpenShift Container Platform web console. You can also use the web console to create a VM by copying an existing snapshot or to clone a VM.

You can create a VM from a list of available bootable volumes. You can add Linux- or Windows-based volumes to the list.

Procedure

  1. In the web console, navigate to VirtualizationCatalog.

    The InstanceTypes tab opens by default.

  2. Select either of the following options:

    • Select a suitable bootable volume from the list. If the list is truncated, click the Show all button to display the entire list.

      Note

      The bootable volume table lists only those volumes in the openshift-virtualization-os-images namespace that have the instancetype.kubevirt.io/default-preference label.

      • Optional: Click the star icon to designate a bootable volume as a favorite. Starred bootable volumes appear first in the volume list.
    • Click Add volume to upload a new volume or to use an existing persistent volume claim (PVC), a volume snapshot, or a containerDisk volume. Click Save.

      Logos of operating systems that are not available in the cluster are shown at the bottom of the list. You can add a volume for the required operating system by clicking the Add volume link.

      In addition, there is a link to the Create a Windows boot source quick start. The same link appears in a popover if you hover the pointer over the question mark icon next to the Select volume to boot from line.

      Immediately after you install the environment or when the environment is disconnected, the list of volumes to boot from is empty. In that case, three operating system logos are displayed: Windows, RHEL, and Linux. You can add a new volume that meets your requirements by clicking the Add volume button.

  3. Click an instance type tile and select the resource size appropriate for your workload.
  4. Optional: Choose the virtual machine details, including the VM’s name, that apply to the volume you are booting from:

    • For a Linux-based volume, follow these steps to configure SSH:

      1. If you have not already added a public SSH key to your project, click the edit icon beside Authorized SSH key in the VirtualMachine details section.
      2. Select one of the following options:

        • Use existing: Select a secret from the secrets list.
        • Add new: Follow these steps:

          1. Browse to the public SSH key file or paste the file in the key field.
          2. Enter the secret name.
          3. Optional: Select Automatically apply this key to any new VirtualMachine you create in this project.
      3. Click Save.
    • For a Windows volume, follow either of these set of steps to configure sysprep options:

      • If you have not already added sysprep options for the Windows volume, follow these steps:

        1. Click the edit icon beside Sysprep in the VirtualMachine details section.
        2. Add the Autoattend.xml answer file.
        3. Add the Unattend.xml answer file.
        4. Click Save.
      • If you want to use existing sysprep options for the Windows volume, follow these steps:

        1. Click Attach existing sysprep.
        2. Enter the name of the existing sysprep Unattend.xml answer file.
        3. Click Save.
  5. Optional: If you are creating a Windows VM, you can mount a Windows driver disk:

    1. Click the Customize VirtualMachine button.
    2. On the VirtualMachine details page, click Storage.
    3. Select the Mount Windows drivers disk checkbox.
  6. Optional: Click View YAML & CLI to view the YAML file. Click CLI to view the CLI commands. You can also download or copy either the YAML file contents or the CLI commands.
  7. Click Create VirtualMachine.

After the VM is created, you can monitor the status on the VirtualMachine details page.

7.1.3. Creating virtual machines from templates

You can create virtual machines (VMs) from Red Hat templates by using the OpenShift Container Platform web console.

7.1.3.1. About VM templates
Boot sources

You can expedite VM creation by using templates that have an available boot source. Templates with a boot source are labeled Available boot source if they do not have a custom label.

Templates without a boot source are labeled Boot source required. See Creating virtual machines from custom images.

Customization
You can customize the disk source and VM parameters before you start the VM.

See storage volume types and storage fields for details about disk source settings.

Note

If you copy a VM template with all its labels and annotations, your version of the template is marked as deprecated when a new version of the Scheduling, Scale, and Performance (SSP) Operator is deployed. You can remove this designation. See Customizing a VM template by using the web console.

Single-node OpenShift
Due to differences in storage behavior, some templates are incompatible with single-node OpenShift. To ensure compatibility, do not set the evictionStrategy field for templates or VMs that use data volumes or storage profiles.
7.1.3.2. Creating a VM from a template

You can create a virtual machine (VM) from a template with an available boot source by using the OpenShift Container Platform web console.

Optional: You can customize template or VM parameters, such as data sources, cloud-init, or SSH keys, before you start the VM.

Procedure

  1. Navigate to VirtualizationCatalog in the web console.
  2. Click Boot source available to filter templates with boot sources.

    The catalog displays the default templates. Click All Items to view all available templates for your filters.

  3. Click a template tile to view its details.
  4. Optional: If you are using a Windows template, you can mount a Windows driver disk by selecting the Mount Windows drivers disk checkbox.
  5. If you do not need to customize the template or VM parameters, click Quick create VirtualMachine to create a VM from the template.

    If you need to customize the template or VM parameters, do the following:

    1. Click Customize VirtualMachine.
    2. Expand Storage or Optional parameters to edit data source settings.
    3. Click Customize VirtualMachine parameters.

      The Customize and create VirtualMachine pane displays the Overview, YAML, Scheduling, Environment, Network interfaces, Disks, Scripts, and Metadata tabs.

    4. Edit the parameters that must be set before the VM boots, such as cloud-init or a static SSH key.
    5. Click Create VirtualMachine.

      The VirtualMachine details page displays the provisioning status.

7.1.3.2.1. Storage volume types
Table 7.2. Storage volume types
TypeDescription

ephemeral

A local copy-on-write (COW) image that uses a network volume as a read-only backing store. The backing volume must be a PersistentVolumeClaim. The ephemeral image is created when the virtual machine starts and stores all writes locally. The ephemeral image is discarded when the virtual machine is stopped, restarted, or deleted. The backing volume (PVC) is not mutated in any way.

persistentVolumeClaim

Attaches an available PV to a virtual machine. Attaching a PV allows for the virtual machine data to persist between sessions.

Importing an existing virtual machine disk into a PVC by using CDI and attaching the PVC to a virtual machine instance is the recommended method for importing existing virtual machines into OpenShift Container Platform. There are some requirements for the disk to be used within a PVC.

dataVolume

Data volumes build on the persistentVolumeClaim disk type by managing the process of preparing the virtual machine disk via an import, clone, or upload operation. VMs that use this volume type are guaranteed not to start until the volume is ready.

Specify type: dataVolume or type: "". If you specify any other value for type, such as persistentVolumeClaim, a warning is displayed, and the virtual machine does not start.

cloudInitNoCloud

Attaches a disk that contains the referenced cloud-init NoCloud data source, providing user data and metadata to the virtual machine. A cloud-init installation is required inside the virtual machine disk.

containerDisk

References an image, such as a virtual machine disk, that is stored in the container image registry. The image is pulled from the registry and attached to the virtual machine as a disk when the virtual machine is launched.

A containerDisk volume is not limited to a single virtual machine and is useful for creating large numbers of virtual machine clones that do not require persistent storage.

Only RAW and QCOW2 formats are supported disk types for the container image registry. QCOW2 is recommended for reduced image size.

Note

A containerDisk volume is ephemeral. It is discarded when the virtual machine is stopped, restarted, or deleted. A containerDisk volume is useful for read-only file systems such as CD-ROMs or for disposable virtual machines.

emptyDisk

Creates an additional sparse QCOW2 disk that is tied to the life-cycle of the virtual machine interface. The data survives guest-initiated reboots in the virtual machine but is discarded when the virtual machine stops or is restarted from the web console. The empty disk is used to store application dependencies and data that otherwise exceeds the limited temporary file system of an ephemeral disk.

The disk capacity size must also be provided.

7.1.3.2.2. Storage fields
FieldDescription

Blank (creates PVC)

Create an empty disk.

Import via URL (creates PVC)

Import content via URL (HTTP or HTTPS endpoint).

Use an existing PVC

Use a PVC that is already available in the cluster.

Clone existing PVC (creates PVC)

Select an existing PVC available in the cluster and clone it.

Import via Registry (creates PVC)

Import content via container registry.

Container (ephemeral)

Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines.

Name

Name of the disk. The name can contain lowercase letters (a-z), numbers (0-9), hyphens (-), and periods (.), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters.

Size

Size of the disk in GiB.

Type

Type of disk. Example: Disk or CD-ROM

Interface

Type of disk device. Supported interfaces are virtIO, SATA, and SCSI.

Storage Class

The storage class that is used to create the disk.

Advanced storage settings

The following advanced storage settings are optional and available for Blank, Import via URL, and Clone existing PVC disks.

If you do not specify these parameters, the system uses the default storage profile values.

ParameterOptionParameter description

Volume Mode

Filesystem

Stores the virtual disk on a file system-based volume.

Block

Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it.

Access Mode

ReadWriteOnce (RWO)

Volume can be mounted as read-write by a single node.

ReadWriteMany (RWX)

Volume can be mounted as read-write by many nodes at one time.

Note

This mode is required for live migration.

7.1.3.2.3. Customizing a VM template by using the web console

You can customize an existing virtual machine (VM) template by modifying the VM or template parameters, such as data sources, cloud-init, or SSH keys, before you start the VM. If you customize a template by copying it and including all of its labels and annotations, the customized template is marked as deprecated when a new version of the Scheduling, Scale, and Performance (SSP) Operator is deployed.

You can remove the deprecated designation from the customized template.

Procedure

  1. Navigate to VirtualizationTemplates in the web console.
  2. From the list of VM templates, click the template marked as deprecated.
  3. Click Edit next to the pencil icon beside Labels.
  4. Remove the following two labels:

    • template.kubevirt.io/type: "base"
    • template.kubevirt.io/version: "version"
  5. Click Save.
  6. Click the pencil icon beside the number of existing Annotations.
  7. Remove the following annotation:

    • template.kubevirt.io/deprecated
  8. Click Save.
7.1.3.2.4. Creating a custom VM template in the web console

You create a virtual machine template by editing a YAML file example in the OpenShift Container Platform web console.

Procedure

  1. In the web console, click VirtualizationTemplates in the side menu.
  2. Optional: Use the Project drop-down menu to change the project associated with the new template. All templates are saved to the openshift project by default.
  3. Click Create Template.
  4. Specify the template parameters by editing the YAML file.
  5. Click Create.

    The template is displayed on the Templates page.

  6. Optional: Click Download to download and save the YAML file.

7.1.4. Creating virtual machines from the command line

You can create virtual machines (VMs) from the command line by editing or creating a VirtualMachine manifest. You can simplify VM configuration by using an instance type in your VM manifest.

Note
7.1.4.1. Creating manifests by using the virtctl tool

You can use the virtctl CLI utility to simplify creating manifests for VMs, VM instance types, and VM preferences. For more information, see VM manifest creation commands.

7.1.4.2. Creating a VM from a VirtualMachine manifest

You can create a virtual machine (VM) from a VirtualMachine manifest.

Procedure

  1. Edit the VirtualMachine manifest for your VM. The following example configures a Red Hat Enterprise Linux (RHEL) VM:

    Note

    This example manifest does not configure VM authentication.

    Example manifest for a RHEL VM

     apiVersion: kubevirt.io/v1
     kind: VirtualMachine
     metadata:
      name: rhel-9-minimal
     spec:
      dataVolumeTemplates:
        - metadata:
            name: rhel-9-minimal-volume
          spec:
            sourceRef:
              kind: DataSource
              name: rhel9 1
              namespace: openshift-virtualization-os-images 2
            storage: {}
      instancetype:
        name: u1.medium 3
      preference:
        name: rhel.9 4
      running: true
      template:
        spec:
          domain:
            devices: {}
          volumes:
            - dataVolume:
                name: rhel-9-minimal-volume
              name: rootdisk

    1
    The rhel9 golden image is used to install RHEL 9 as the guest operating system.
    2
    Golden images are stored in the openshift-virtualization-os-images namespace.
    3
    The u1.medium instance type requests 1 vCPU and 4Gi memory for the VM. These resource values cannot be overridden within the VM.
    4
    The rhel.9 preference specifies additional attributes that support the RHEL 9 guest operating system.
  2. Create a virtual machine by using the manifest file:

    $ oc create -f <vm_manifest_file>.yaml
  3. Optional: Start the virtual machine:

    $ virtctl start <vm_name> -n <namespace>

7.2. Creating VMs from custom images

7.2.1. Creating virtual machines from custom images overview

You can create virtual machines (VMs) from custom operating system images by using one of the following methods:

The Containerized Data Importer (CDI) imports the image into a PVC by using a data volume. You add the PVC to the VM by using the OpenShift Container Platform web console or command line.

Important

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

You must also install VirtIO drivers on Windows VMs.

The QEMU guest agent is included with Red Hat images.

7.2.2. Creating VMs by using container disks

You can create virtual machines (VMs) by using container disks built from operating system images.

You can enable auto updates for your container disks. See Managing automatic boot source updates for details.

Important

If the container disks are large, the I/O traffic might increase and cause worker nodes to be unavailable. You can perform the following tasks to resolve this issue:

You create a VM from a container disk by performing the following steps:

  1. Build an operating system image into a container disk and upload it to your container registry.
  2. If your container registry does not have TLS, configure your environment to disable TLS for your registry.
  3. Create a VM with the container disk as the disk source by using the web console or the command line.
Important

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

7.2.2.1. Building and uploading a container disk

You can build a virtual machine (VM) image into a container disk and upload it to a registry.

The size of a container disk is limited by the maximum layer size of the registry where the container disk is hosted.

Note

For Red Hat Quay, you can change the maximum layer size by editing the YAML configuration file that is created when Red Hat Quay is first deployed.

Prerequisites

  • You must have podman installed.
  • You must have a QCOW2 or RAW image file.

Procedure

  1. Create a Dockerfile to build the VM image into a container image. The VM image must be owned by QEMU, which has a UID of 107, and placed in the /disk/ directory inside the container. Permissions for the /disk/ directory must then be set to 0440.

    The following example uses the Red Hat Universal Base Image (UBI) to handle these configuration changes in the first stage, and uses the minimal scratch image in the second stage to store the result:

    $ cat > Dockerfile << EOF
    FROM registry.access.redhat.com/ubi8/ubi:latest AS builder
    ADD --chown=107:107 <vm_image>.qcow2 /disk/ 1
    RUN chmod 0440 /disk/*
    
    FROM scratch
    COPY --from=builder /disk/* /disk/
    EOF
    1
    Where <vm_image> is the image in either QCOW2 or RAW format. If you use a remote image, replace <vm_image>.qcow2 with the complete URL.
  2. Build and tag the container:

    $ podman build -t <registry>/<container_disk_name>:latest .
  3. Push the container image to the registry:

    $ podman push <registry>/<container_disk_name>:latest
7.2.2.2. Disabling TLS for a container registry

You can disable TLS (transport layer security) for one or more container registries by editing the insecureRegistries field of the HyperConverged custom resource.

Prerequisites

  1. Open the HyperConverged CR in your default editor by running the following command:

    $ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
  2. Add a list of insecure registries to the spec.storageImport.insecureRegistries field.

    Example HyperConverged custom resource

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
      namespace: openshift-cnv
    spec:
      storageImport:
        insecureRegistries: 1
          - "private-registry-example-1:5000"
          - "private-registry-example-2:5000"

    1
    Replace the examples in this list with valid registry hostnames.
7.2.2.3. Creating a VM from a container disk by using the web console

You can create a virtual machine (VM) by importing a container disk from a container registry by using the OpenShift Container Platform web console.

Procedure

  1. Navigate to VirtualizationCatalog in the web console.
  2. Click a template tile without an available boot source.
  3. Click Customize VirtualMachine.
  4. On the Customize template parameters page, expand Storage and select Registry (creates PVC) from the Disk source list.
  5. Enter the container image URL. Example: https://mirror.arizona.edu/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
  6. Set the disk size.
  7. Click Next.
  8. Click Create VirtualMachine.
7.2.2.4. Creating a VM from a container disk by using the command line

You can create a virtual machine (VM) from a container disk by using the command line.

When the virtual machine (VM) is created, the data volume with the container disk is imported into persistent storage.

Prerequisites

  • You must have access credentials for the container registry that contains the container disk.

Procedure

  1. Edit the VirtualMachine manifest and save it as a vm-rhel-datavolume.yaml file:

    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      creationTimestamp: null
      name: vm-rhel-datavolume 1
      labels:
        kubevirt.io/vm: vm-rhel-datavolume
    spec:
      dataVolumeTemplates:
      - metadata:
          creationTimestamp: null
          name: rhel-dv 2
        spec:
          sourceRef:
            kind: DataSource
            name: rhel9
            namespace: openshift-virtualization-os-images
          storage:
            resources:
              requests:
                storage: 10Gi 3
      instancetype:
        name: u1.small 4
      preference:
        inferFromVolume: datavolumedisk1
      runStrategy: Always
      template:
        metadata:
          creationTimestamp: null
          labels:
            kubevirt.io/vm: vm-rhel-datavolume
        spec:
          domain:
            devices: {}
            resources: {}
          terminationGracePeriodSeconds: 180
          volumes:
          - dataVolume:
              name: rhel-dv
            name: datavolumedisk1
    status: {}
    1
    Specify the name of the VM.
    2
    Specify the name of the data volume.
    3
    Specify the size of the storage requested for the data volume.
    4
    Optional: Specify the instance type to use to control resource sizing of the VM.
  2. Create the VM by running the following command:

    $ oc create -f vm-rhel-datavolume.yaml

    The oc create command creates the data volume and the VM. The CDI controller creates an underlying PVC with the correct annotation and the import process begins. When the import is complete, the data volume status changes to Succeeded. You can start the VM.

    Data volume provisioning happens in the background, so there is no need to monitor the process.

Verification

  1. The importer pod downloads the container disk from the specified URL and stores it on the provisioned persistent volume. View the status of the importer pod by running the following command:

    $ oc get pods
  2. Monitor the data volume until its status is Succeeded by running the following command:

    $ oc describe dv rhel-dv 1
    1
    Specify the data volume name that you defined in the VirtualMachine manifest.
  3. Verify that provisioning is complete and that the VM has started by accessing its serial console:

    $ virtctl console vm-rhel-datavolume

7.2.3. Creating VMs by importing images from web pages

You can create virtual machines (VMs) by importing operating system images from web pages.

Important

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

7.2.3.1. Creating a VM from an image on a web page by using the web console

You can create a virtual machine (VM) by importing an image from a web page by using the OpenShift Container Platform web console.

Prerequisites

  • You must have access to the web page that contains the image.

Procedure

  1. Navigate to VirtualizationCatalog in the web console.
  2. Click a template tile without an available boot source.
  3. Click Customize VirtualMachine.
  4. On the Customize template parameters page, expand Storage and select URL (creates PVC) from the Disk source list.
  5. Enter the image URL. Example: https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.9/x86_64/product-software
  6. Enter the container image URL. Example: https://mirror.arizona.edu/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
  7. Set the disk size.
  8. Click Next.
  9. Click Create VirtualMachine.
7.2.3.2. Creating a VM from an image on a web page by using the command line

You can create a virtual machine (VM) from an image on a web page by using the command line.

When the virtual machine (VM) is created, the data volume with the image is imported into persistent storage.

Prerequisites

  • You must have access credentials for the web page that contains the image.

Procedure

  1. Edit the VirtualMachine manifest and save it as a vm-rhel-datavolume.yaml file:

    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      creationTimestamp: null
      name: vm-rhel-datavolume 1
      labels:
        kubevirt.io/vm: vm-rhel-datavolume
    spec:
      dataVolumeTemplates:
      - metadata:
          creationTimestamp: null
          name: rhel-dv 2
        spec:
          sourceRef:
            kind: DataSource
            name: rhel9
            namespace: openshift-virtualization-os-images
          storage:
            resources:
              requests:
                storage: 10Gi 3
      instancetype:
        name: u1.small 4
      preference:
        inferFromVolume: datavolumedisk1
      runStrategy: Always
      template:
        metadata:
          creationTimestamp: null
          labels:
            kubevirt.io/vm: vm-rhel-datavolume
        spec:
          domain:
            devices: {}
            resources: {}
          terminationGracePeriodSeconds: 180
          volumes:
          - dataVolume:
              name: rhel-dv
            name: datavolumedisk1
    status: {}
    1
    Specify the name of the VM.
    2
    Specify the name of the data volume.
    3
    Specify the size of the storage requested for the data volume.
    4
    Optional: Specify the instance type to use to control resource sizing of the VM.
  2. Create the VM by running the following command:

    $ oc create -f vm-rhel-datavolume.yaml

    The oc create command creates the data volume and the VM. The CDI controller creates an underlying PVC with the correct annotation and the import process begins. When the import is complete, the data volume status changes to Succeeded. You can start the VM.

    Data volume provisioning happens in the background, so there is no need to monitor the process.

Verification

  1. The importer pod downloads the image from the specified URL and stores it on the provisioned persistent volume. View the status of the importer pod by running the following command:

    $ oc get pods
  2. Monitor the data volume until its status is Succeeded by running the following command:

    $ oc describe dv rhel-dv 1
    1
    Specify the data volume name that you defined in the VirtualMachine manifest.
  3. Verify that provisioning is complete and that the VM has started by accessing its serial console:

    $ virtctl console vm-rhel-datavolume

7.2.4. Creating VMs by uploading images

You can create virtual machines (VMs) by uploading operating system images from your local machine.

You can create a Windows VM by uploading a Windows image to a PVC. Then you clone the PVC when you create the VM.

Important

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

You must also install VirtIO drivers on Windows VMs.

7.2.4.1. Creating a VM from an uploaded image by using the web console

You can create a virtual machine (VM) from an uploaded operating system image by using the OpenShift Container Platform web console.

Prerequisites

  • You must have an IMG, ISO, or QCOW2 image file.

Procedure

  1. Navigate to VirtualizationCatalog in the web console.
  2. Click a template tile without an available boot source.
  3. Click Customize VirtualMachine.
  4. On the Customize template parameters page, expand Storage and select Upload (Upload a new file to a PVC) from the Disk source list.
  5. Browse to the image on your local machine and set the disk size.
  6. Click Customize VirtualMachine.
  7. Click Create VirtualMachine.
7.2.4.1.1. Generalizing a VM image

You can generalize a Red Hat Enterprise Linux (RHEL) image to remove all system-specific configuration data before you use the image to create a golden image, a preconfigured snapshot of a virtual machine (VM). You can use a golden image to deploy new VMs.

You can generalize a RHEL VM by using the virtctl, guestfs, and virt-sysprep tools.

Prerequisites

  • You have a RHEL virtual machine (VM) to use as a base VM.
  • You have installed the OpenShift CLI (oc).
  • You have installed the virtctl tool.

Procedure

  1. Stop the RHEL VM if it is running, by entering the following command:

    $ virtctl stop <my_vm_name>
  2. Optional: Clone the virtual machine to avoid losing the data from your original VM. You can then generalize the cloned VM.
  3. Retrieve the dataVolume that stores the root filesystem for the VM by running the following command:

    $ oc get vm <my_vm_name> -o jsonpath="{.spec.template.spec.volumes}{'\n'}"

    Example output

    [{"dataVolume":{"name":"<my_vm_volume>"},"name":"rootdisk"},{"cloudInitNoCloud":{...}]

  4. Retrieve the persistent volume claim (PVC) that matches the listed dataVolume by running the followimg command:

    $ oc get pvc

    Example output

    NAME            STATUS   VOLUME  CAPACITY   ACCESS MODES  STORAGECLASS     AGE
    <my_vm_volume> Bound  …

    Note

    If your cluster configuration does not enable you to clone a VM, to avoid losing the data from your original VM, you can clone the VM PVC to a data volume instead. You can then use the cloned PVC to create a golden image.

    If you are creating a golden image by cloning a PVC, continue with the next steps, using the cloned PVC.

  5. Deploy a new interactive container with libguestfs-tools and attach the PVC to it by running the following command:

    $ virtctl guestfs <my-vm-volume> --uid 107

    This command opens a shell for you to run the next command.

  6. Remove all configurations specific to your system by running the following command:

    $ virt-sysprep -a disk.img
  7. In the OpenShift Container Platform console, click VirtualizationCatalog.
  8. Click Add volume.
  9. In the Add volume window:

    1. From the Source type list, select Use existing Volume.
    2. From the Volume project list, select your project.
    3. From the Volume name list, select the correct PVC.
    4. In the Volume name field, enter a name for the new golden image.
    5. From the Preference list, select the RHEL version you are using.
    6. From the Default Instance Type list, select the instance type with the correct CPU and memory requirements for the version of RHEL you selected previously.
    7. Click Save.

The new volume appears in the Select volume to boot from list. This is your new golden image. You can use this volume to create new VMs.

Additional resources for generalizing VMs

7.2.4.2. Creating a Windows VM

You can create a Windows virtual machine (VM) by uploading a Windows image to a persistent volume claim (PVC) and then cloning the PVC when you create a VM by using the OpenShift Container Platform web console.

Prerequisites

Procedure

  1. Upload the Windows image as a new PVC:

    1. Navigate to StoragePersistentVolumeClaims in the web console.
    2. Click Create PersistentVolumeClaimWith Data upload form.
    3. Browse to the Windows image and select it.
    4. Enter the PVC name, select the storage class and size and then click Upload.

      The Windows image is uploaded to a PVC.

  2. Configure a new VM by cloning the uploaded PVC:

    1. Navigate to VirtualizationCatalog.
    2. Select a Windows template tile and click Customize VirtualMachine.
    3. Select Clone (clone PVC) from the Disk source list.
    4. Select the PVC project, the Windows image PVC, and the disk size.
  3. Apply the answer file to the VM:

    1. Click Customize VirtualMachine parameters.
    2. On the Sysprep section of the Scripts tab, click Edit.
    3. Browse to the autounattend.xml answer file and click Save.
  4. Set the run strategy of the VM:

    1. Clear Start this VirtualMachine after creation so that the VM does not start immediately.
    2. Click Create VirtualMachine.
    3. On the YAML tab, replace running:false with runStrategy: RerunOnFailure and click Save.
  5. Click the Options menu kebab and select Start.

    The VM boots from the sysprep disk containing the autounattend.xml answer file.

7.2.4.2.1. Generalizing a Windows VM image

You can generalize a Windows operating system image to remove all system-specific configuration data before you use the image to create a new virtual machine (VM).

Before generalizing the VM, you must ensure the sysprep tool cannot detect an answer file after the unattended Windows installation.

Prerequisites

  • A running Windows VM with the QEMU guest agent installed.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationVirtualMachines.
  2. Select a Windows VM to open the VirtualMachine details page.
  3. Click ConfigurationDisks.
  4. Click the Options menu kebab beside the sysprep disk and select Detach.
  5. Click Detach.
  6. Rename C:\Windows\Panther\unattend.xml to avoid detection by the sysprep tool.
  7. Start the sysprep program by running the following command:

    %WINDIR%\System32\Sysprep\sysprep.exe /generalize /shutdown /oobe /mode:vm
  8. After the sysprep tool completes, the Windows VM shuts down. The disk image of the VM is now available to use as an installation image for Windows VMs.

You can now specialize the VM.

7.2.4.2.2. Specializing a Windows VM image

Specializing a Windows virtual machine (VM) configures the computer-specific information from a generalized Windows image onto the VM.

Prerequisites

  • You must have a generalized Windows disk image.
  • You must create an unattend.xml answer file. See the Microsoft documentation for details.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationCatalog.
  2. Select a Windows template and click Customize VirtualMachine.
  3. Select PVC (clone PVC) from the Disk source list.
  4. Select the PVC project and PVC name of the generalized Windows image.
  5. Click Customize VirtualMachine parameters.
  6. Click the Scripts tab.
  7. In the Sysprep section, click Edit, browse to the unattend.xml answer file, and click Save.
  8. Click Create VirtualMachine.

During the initial boot, Windows uses the unattend.xml answer file to specialize the VM. The VM is now ready to use.

7.2.4.3. Creating a VM from an uploaded image by using the command line

You can upload an operating system image by using the virtctl command line tool. You can use an existing data volume or create a new data volume for the image.

Prerequisites

  • You must have an ISO, IMG, or QCOW2 operating system image file.
  • For best performance, compress the image file by using the virt-sparsify tool or the xz or gzip utilities.
  • You must have virtctl installed.
  • The client machine must be configured to trust the OpenShift Container Platform router’s certificate.

Procedure

  1. Upload the image by running the virtctl image-upload command:

    $ virtctl image-upload dv <datavolume_name> \ 1
      --size=<datavolume_size> \ 2
      --image-path=</path/to/image> \ 3
    1
    The name of the data volume.
    2
    The size of the data volume. For example: --size=500Mi, --size=1G
    3
    The file path of the image.
    Note
    • If you do not want to create a new data volume, omit the --size parameter and include the --no-create flag.
    • When uploading a disk image to a PVC, the PVC size must be larger than the size of the uncompressed virtual disk.
    • To allow insecure server connections when using HTTPS, use the --insecure parameter. When you use the --insecure flag, the authenticity of the upload endpoint is not verified.
  2. Optional. To verify that a data volume was created, view all data volumes by running the following command:

    $ oc get dvs

7.2.5. Installing the QEMU guest agent and VirtIO drivers

The QEMU guest agent is a daemon that runs on the virtual machine (VM) and passes information to the host about the VM, users, file systems, and secondary networks.

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

7.2.5.1. Installing the QEMU guest agent
7.2.5.1.1. Installing the QEMU guest agent on a Linux VM

The qemu-guest-agent is widely available and available by default in Red Hat Enterprise Linux (RHEL) virtual machines (VMs). Install the agent and start the service.

Note

To create snapshots of an online (Running state) VM with the highest integrity, install the QEMU guest agent.

The QEMU guest agent takes a consistent snapshot by attempting to quiesce the VM file system as much as possible, depending on the system workload. This ensures that in-flight I/O is written to the disk before the snapshot is taken. If the guest agent is not present, quiescing is not possible and a best-effort snapshot is taken. The conditions under which the snapshot was taken are reflected in the snapshot indications that are displayed in the web console or CLI.

Procedure

  1. Log in to the VM by using a console or SSH.
  2. Install the QEMU guest agent by running the following command:

    $ yum install -y qemu-guest-agent
  3. Ensure the service is persistent and start it:

    $ systemctl enable --now qemu-guest-agent

Verification

  • Run the following command to verify that AgentConnected is listed in the VM spec:

    $ oc get vm <vm_name>
7.2.5.1.2. Installing the QEMU guest agent on a Windows VM

For Windows virtual machines (VMs), the QEMU guest agent is included in the VirtIO drivers. You can install the drivers during a Windows installation or on an existing Windows VM.

Note

To create snapshots of an online (Running state) VM with the highest integrity, install the QEMU guest agent.

The QEMU guest agent takes a consistent snapshot by attempting to quiesce the VM file system as much as possible, depending on the system workload. This ensures that in-flight I/O is written to the disk before the snapshot is taken. If the guest agent is not present, quiescing is not possible and a best-effort snapshot is taken. The conditions under which the snapshot was taken are reflected in the snapshot indications that are displayed in the web console or CLI.

Procedure

  1. In the Windows guest operating system, use the File Explorer to navigate to the guest-agent directory in the virtio-win CD drive.
  2. Run the qemu-ga-x86_64.msi installer.

Verification

  1. Obtain a list of network services by running the following command:

    $ net start
  2. Verify that the output contains the QEMU Guest Agent.
7.2.5.2. Installing VirtIO drivers on Windows VMs

VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines (VMs) to run in OpenShift Virtualization. The drivers are shipped with the rest of the images and do not require a separate download.

The container-native-virtualization/virtio-win container disk must be attached to the VM as a SATA CD drive to enable driver installation. You can install VirtIO drivers during Windows installation or added to an existing Windows installation.

After the drivers are installed, the container-native-virtualization/virtio-win container disk can be removed from the VM.

Table 7.3. Supported drivers
Driver nameHardware IDDescription

viostor

VEN_1AF4&DEV_1001
VEN_1AF4&DEV_1042

The block driver. Sometimes labeled as an SCSI Controller in the Other devices group.

viorng

VEN_1AF4&DEV_1005
VEN_1AF4&DEV_1044

The entropy source driver. Sometimes labeled as a PCI Device in the Other devices group.

NetKVM

VEN_1AF4&DEV_1000
VEN_1AF4&DEV_1041

The network driver. Sometimes labeled as an Ethernet Controller in the Other devices group. Available only if a VirtIO NIC is configured.

7.2.5.2.1. Attaching VirtIO container disk to Windows VMs during installation

You must attach the VirtIO container disk to the Windows VM to install the necessary Windows drivers. This can be done during creation of the VM.

Procedure

  1. When creating a Windows VM from a template, click Customize VirtualMachine.
  2. Select Mount Windows drivers disk.
  3. Click the Customize VirtualMachine parameters.
  4. Click Create VirtualMachine.

After the VM is created, the virtio-win SATA CD disk will be attached to the VM.

7.2.5.2.2. Attaching VirtIO container disk to an existing Windows VM

You must attach the VirtIO container disk to the Windows VM to install the necessary Windows drivers. This can be done to an existing VM.

Procedure

  1. Navigate to the existing Windows VM, and click ActionsStop.
  2. Go to VM DetailsConfigurationDisks and click Add disk.
  3. Add windows-driver-disk from container source, set the Type to CD-ROM, and then set the Interface to SATA.
  4. Click Save.
  5. Start the VM, and connect to a graphical console.
7.2.5.2.3. Installing VirtIO drivers during Windows installation

You can install the VirtIO drivers while installing Windows on a virtual machine (VM).

Note

This procedure uses a generic approach to the Windows installation and the installation method might differ between versions of Windows. See the documentation for the version of Windows that you are installing.

Prerequisites

  • A storage device containing the virtio drivers must be attached to the VM.

Procedure

  1. In the Windows operating system, use the File Explorer to navigate to the virtio-win CD drive.
  2. Double-click the drive to run the appropriate installer for your VM.

    For a 64-bit vCPU, select the virtio-win-gt-x64 installer. 32-bit vCPUs are no longer supported.

  3. Optional: During the Custom Setup step of the installer, select the device drivers you want to install. The recommended driver set is selected by default.
  4. After the installation is complete, select Finish.
  5. Reboot the VM.

Verification

  1. Open the system disk on the PC. This is typically C:.
  2. Navigate to Program FilesVirtio-Win.

If the Virtio-Win directory is present and contains a sub-directory for each driver, the installation was successful.

7.2.5.2.4. Installing VirtIO drivers from a SATA CD drive on an existing Windows VM

You can install the VirtIO drivers from a SATA CD drive on an existing Windows virtual machine (VM).

Note

This procedure uses a generic approach to adding drivers to Windows. See the installation documentation for your version of Windows for specific installation steps.

Prerequisites

  • A storage device containing the virtio drivers must be attached to the VM as a SATA CD drive.

Procedure

  1. Start the VM and connect to a graphical console.
  2. Log in to a Windows user session.
  3. Open Device Manager and expand Other devices to list any Unknown device.

    1. Open the Device Properties to identify the unknown device.
    2. Right-click the device and select Properties.
    3. Click the Details tab and select Hardware Ids in the Property list.
    4. Compare the Value for the Hardware Ids with the supported VirtIO drivers.
  4. Right-click the device and select Update Driver Software.
  5. Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
  6. Click Next to install the driver.
  7. Repeat this process for all the necessary VirtIO drivers.
  8. After the driver installs, click Close to close the window.
  9. Reboot the VM to complete the driver installation.
7.2.5.2.5. Installing VirtIO drivers from a container disk added as a SATA CD drive

You can install VirtIO drivers from a container disk that you add to a Windows virtual machine (VM) as a SATA CD drive.

Tip

Downloading the container-native-virtualization/virtio-win container disk from the Red Hat Ecosystem Catalog is not mandatory, because the container disk is downloaded from the Red Hat registry if it not already present in the cluster. However, downloading reduces the installation time.

Prerequisites

  • You must have access to the Red Hat registry or to the downloaded container-native-virtualization/virtio-win container disk in a restricted environment.

Procedure

  1. Add the container-native-virtualization/virtio-win container disk as a CD drive by editing the VirtualMachine manifest:

    # ...
    spec:
      domain:
        devices:
          disks:
            - name: virtiocontainerdisk
              bootOrder: 2 1
              cdrom:
                bus: sata
    volumes:
      - containerDisk:
          image: container-native-virtualization/virtio-win
        name: virtiocontainerdisk
    1
    OpenShift Virtualization boots the VM disks in the order defined in the VirtualMachine manifest. You can either define other VM disks that boot before the container-native-virtualization/virtio-win container disk or use the optional bootOrder parameter to ensure the VM boots from the correct disk. If you configure the boot order for a disk, you must configure the boot order for the other disks.
  2. Apply the changes:

    • If the VM is not running, run the following command:

      $ virtctl start <vm> -n <namespace>
    • If the VM is running, reboot the VM or run the following command:

      $ oc apply -f <vm.yaml>
  3. After the VM has started, install the VirtIO drivers from the SATA CD drive.
7.2.5.3. Updating VirtIO drivers
7.2.5.3.1. Updating VirtIO drivers on a Windows VM

Update the virtio drivers on a Windows virtual machine (VM) by using the Windows Update service.

Prerequisites

  • The cluster must be connected to the internet. Disconnected clusters cannot reach the Windows Update service.

Procedure

  1. In the Windows Guest operating system, click the Windows key and select Settings.
  2. Navigate to Windows UpdateAdvanced OptionsOptional Updates.
  3. Install all updates from Red Hat, Inc..
  4. Reboot the VM.

Verification

  1. On the Windows VM, navigate to the Device Manager.
  2. Select a device.
  3. Select the Driver tab.
  4. Click Driver Details and confirm that the virtio driver details displays the correct version.

7.2.6. Cloning VMs

You can clone virtual machines (VMs) or create new VMs from snapshots.

7.2.6.1. Cloning a VM by using the web console

You can clone an existing VM by using the web console.

Procedure

  1. Navigate to VirtualizationVirtualMachines in the web console.
  2. Select a VM to open the VirtualMachine details page.
  3. Click Actions.
  4. Select Clone.
  5. On the Clone VirtualMachine page, enter the name of the new VM.
  6. (Optional) Select the Start cloned VM checkbox to start the cloned VM.
  7. Click Clone.
7.2.6.2. Creating a VM from an existing snapshot by using the web console

You can create a new VM by copying an existing snapshot.

Procedure

  1. Navigate to VirtualizationVirtualMachines in the web console.
  2. Select a VM to open the VirtualMachine details page.
  3. Click the Snapshots tab.
  4. Click the Options menu kebab for the snapshot you want to copy.
  5. Select Create VirtualMachine.
  6. Enter the name of the virtual machine.
  7. (Optional) Select the Start this VirtualMachine after creation checkbox to start the new virtual machine.
  8. Click Create.
7.2.6.3. Additional resources

7.2.7. Creating VMs by cloning PVCs

You can create virtual machines (VMs) by cloning existing persistent volume claims (PVCs) with custom images.

You must install the QEMU guest agent on VMs created from operating system images that are not provided by Red Hat.

You clone a PVC by creating a data volume that references a source PVC.

7.2.7.1. About cloning

When cloning a data volume, the Containerized Data Importer (CDI) chooses one of the following Container Storage Interface (CSI) clone methods:

  • CSI volume cloning
  • Smart cloning

Both CSI volume cloning and smart cloning methods are efficient, but they have certain requirements for use. If the requirements are not met, the CDI uses host-assisted cloning. Host-assisted cloning is the slowest and least efficient method of cloning, but it has fewer requirements than either of the other two cloning methods.

7.2.7.1.1. CSI volume cloning

Container Storage Interface (CSI) cloning uses CSI driver features to more efficiently clone a source data volume.

CSI volume cloning has the following requirements:

  • The CSI driver that backs the storage class of the persistent volume claim (PVC) must support volume cloning.
  • For provisioners not recognized by the CDI, the corresponding storage profile must have the cloneStrategy set to CSI Volume Cloning.
  • The source and target PVCs must have the same storage class and volume mode.
  • If you create the data volume, you must have permission to create the datavolumes/source resource in the source namespace.
  • The source volume must not be in use.
7.2.7.1.2. Smart cloning

When a Container Storage Interface (CSI) plugin with snapshot capabilities is available, the Containerized Data Importer (CDI) creates a persistent volume claim (PVC) from a snapshot, which then allows efficient cloning of additional PVCs.

Smart cloning has the following requirements:

  • A snapshot class associated with the storage class must exist.
  • The source and target PVCs must have the same storage class and volume mode.
  • If you create the data volume, you must have permission to create the datavolumes/source resource in the source namespace.
  • The source volume must not be in use.
7.2.7.1.3. Host-assisted cloning

When the requirements for neither Container Storage Interface (CSI) volume cloning nor smart cloning have been met, host-assisted cloning is used as a fallback method. Host-assisted cloning is less efficient than either of the two other cloning methods.

Host-assisted cloning uses a source pod and a target pod to copy data from the source volume to the target volume. The target persistent volume claim (PVC) is annotated with the fallback reason that explains why host-assisted cloning has been used, and an event is created.

Example PVC target annotation

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/cloneFallbackReason: The volume modes of source and target are incompatible
    cdi.kubevirt.io/clonePhase: Succeeded
    cdi.kubevirt.io/cloneType: copy

Example event

NAMESPACE   LAST SEEN   TYPE      REASON                    OBJECT                              MESSAGE
test-ns     0s          Warning   IncompatibleVolumeModes   persistentvolumeclaim/test-target   The volume modes of source and target are incompatible

7.2.7.2. Creating a VM from a PVC by using the web console

You can create a virtual machine (VM) by importing an image from a web page by using the OpenShift Container Platform web console. You can create a virtual machine (VM) by cloning a persistent volume claim (PVC) by using the OpenShift Container Platform web console.

Prerequisites

  • You must have access to the web page that contains the image.
  • You must have access to the namespace that contains the source PVC.

Procedure

  1. Navigate to VirtualizationCatalog in the web console.
  2. Click a template tile without an available boot source.
  3. Click Customize VirtualMachine.
  4. On the Customize template parameters page, expand Storage and select PVC (clone PVC) from the Disk source list.
  5. Enter the image URL. Example: https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.9/x86_64/product-software
  6. Enter the container image URL. Example: https://mirror.arizona.edu/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
  7. Select the PVC project and the PVC name.
  8. Set the disk size.
  9. Click Next.
  10. Click Create VirtualMachine.
7.2.7.3. Creating a VM from a PVC by using the command line

You can create a virtual machine (VM) by cloning the persistent volume claim (PVC) of an existing VM by using the command line.

You can clone a PVC by using one of the following options:

  • Cloning a PVC to a new data volume.

    This method creates a data volume whose lifecycle is independent of the original VM. Deleting the original VM does not affect the new data volume or its associated PVC.

  • Cloning a PVC by creating a VirtualMachine manifest with a dataVolumeTemplates stanza.

    This method creates a data volume whose lifecycle is dependent on the original VM. Deleting the original VM deletes the cloned data volume and its associated PVC.

7.2.7.3.1. Cloning a PVC to a data volume

You can clone the persistent volume claim (PVC) of an existing virtual machine (VM) disk to a data volume by using the command line.

You create a data volume that references the original source PVC. The lifecycle of the new data volume is independent of the original VM. Deleting the original VM does not affect the new data volume or its associated PVC.

Cloning between different volume modes is supported for host-assisted cloning, such as cloning from a block persistent volume (PV) to a file system PV, as long as the source and target PVs belong to the kubevirt content type.

Note

Smart-cloning is faster and more efficient than host-assisted cloning because it uses snapshots to clone PVCs. Smart-cloning is supported by storage providers that support snapshots, such as Red Hat OpenShift Data Foundation.

Cloning between different volume modes is not supported for smart-cloning.

Prerequisites

  • The VM with the source PVC must be powered down.
  • If you clone a PVC to a different namespace, you must have permissions to create resources in the target namespace.
  • Additional prerequisites for smart-cloning:

    • Your storage provider must support snapshots.
    • The source and target PVCs must have the same storage provider and volume mode.
    • The value of the driver key of the VolumeSnapshotClass object must match the value of the provisioner key of the StorageClass object as shown in the following example:

      Example VolumeSnapshotClass object

      kind: VolumeSnapshotClass
      apiVersion: snapshot.storage.k8s.io/v1
      driver: openshift-storage.rbd.csi.ceph.com
      # ...

      Example StorageClass object

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      # ...
      provisioner: openshift-storage.rbd.csi.ceph.com

Procedure

  1. Create a DataVolume manifest as shown in the following example:

    apiVersion: cdi.kubevirt.io/v1beta1
    kind: DataVolume
    metadata:
      name: <datavolume> 1
    spec:
      source:
        pvc:
          namespace: "<source_namespace>" 2
          name: "<my_vm_disk>" 3
      storage: {}
    1
    Specify the name of the new data volume.
    2
    Specify the namespace of the source PVC.
    3
    Specify the name of the source PVC.
  2. Create the data volume by running the following command:

    $ oc create -f <datavolume>.yaml
    Note

    Data volumes prevent a VM from starting before the PVC is prepared. You can create a VM that references the new data volume while the PVC is being cloned.

7.2.7.3.2. Creating a VM from a cloned PVC by using a data volume template

You can create a virtual machine (VM) that clones the persistent volume claim (PVC) of an existing VM by using a data volume template.

This method creates a data volume whose lifecycle is dependent on the original VM. Deleting the original VM deletes the cloned data volume and its associated PVC.

Prerequisites

  • The VM with the source PVC must be powered down.

Procedure

  1. Create a VirtualMachine manifest as shown in the following example:

    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      labels:
        kubevirt.io/vm: vm-dv-clone
      name: vm-dv-clone 1
    spec:
      running: false
      template:
        metadata:
          labels:
            kubevirt.io/vm: vm-dv-clone
        spec:
          domain:
            devices:
              disks:
              - disk:
                  bus: virtio
                name: root-disk
            resources:
              requests:
                memory: 64M
          volumes:
          - dataVolume:
              name: favorite-clone
            name: root-disk
      dataVolumeTemplates:
      - metadata:
          name: favorite-clone
        spec:
          storage:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 2Gi
          source:
            pvc:
              namespace: <source_namespace> 2
              name: "<source_pvc>" 3
    1
    Specify the name of the VM.
    2
    Specify the namespace of the source PVC.
    3
    Specify the name of the source PVC.
  2. Create the virtual machine with the PVC-cloned data volume:

    $ oc create -f <vm-clone-datavolumetemplate>.yaml

7.3. Connecting to virtual machine consoles

You can connect to the following consoles to access running virtual machines (VMs):

7.3.1. Connecting to the VNC console

You can connect to the VNC console of a virtual machine by using the OpenShift Container Platform web console or the virtctl command line tool.

7.3.1.1. Connecting to the VNC console by using the web console

You can connect to the VNC console of a virtual machine (VM) by using the OpenShift Container Platform web console.

Note

If you connect to a Windows VM with a vGPU assigned as a mediated device, you can switch between the default display and the vGPU display.

Procedure

  1. On the VirtualizationVirtualMachines page, click a VM to open the VirtualMachine details page.
  2. Click the Console tab. The VNC console session starts automatically.
  3. Optional: To switch to the vGPU display of a Windows VM, select Ctl + Alt + 2 from the Send key list.

    • Select Ctl + Alt + 1 from the Send key list to restore the default display.
  4. To end the console session, click outside the console pane and then click Disconnect.
7.3.1.2. Connecting to the VNC console by using virtctl

You can use the virtctl command line tool to connect to the VNC console of a running virtual machine.

Note

If you run the virtctl vnc command on a remote machine over an SSH connection, you must forward the X session to your local machine by running the ssh command with the -X or -Y flags.

Prerequisites

  • You must install the virt-viewer package.

Procedure

  1. Run the following command to start the console session:

    $ virtctl vnc <vm_name>
  2. If the connection fails, run the following command to collect troubleshooting information:

    $ virtctl vnc <vm_name> -v 4
7.3.1.3. Generating a temporary token for the VNC console

To access the VNC of a virtual machine (VM), generate a temporary authentication bearer token for the Kubernetes API.

Note

Kubernetes also supports authentication using client certificates, instead of a bearer token, by modifying the curl command.

Prerequisites

  • A running VM with OpenShift Virtualization 4.14 or later and ssp-operator 4.14 or later

Procedure

  1. Enable the feature gate in the HyperConverged (HCO) custom resource (CR):

    $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv --type json -p '[{"op": "replace", "path": "/spec/featureGates/deployVmConsoleProxy", "value": true}]'
  2. Generate a token by entering the following command:

    $ curl --header "Authorization: Bearer ${TOKEN}" \
         "https://api.<cluster_fqdn>/apis/token.kubevirt.io/v1alpha1/namespaces/<namespace>/virtualmachines/<vm_name>/vnc?duration=<duration>"

    The <duration> parameter can be set in hours and minutes, with a minimum duration of 10 minutes. For example: 5h30m. If this parameter is not set, the token is valid for 10 minutes by default.

    Sample output:

    { "token": "eyJhb..." }
  3. Optional: Use the token provided in the output to create a variable:

    $ export VNC_TOKEN="<token>"

You can now use the token to access the VNC console of a VM.

Verification