Virtualization


OpenShift Container Platform 4.13

OpenShift Virtualization installation, usage, and release notes

Red Hat OpenShift Documentation Team

Abstract

This document provides information about how to use OpenShift Virtualization in OpenShift Container Platform.

Chapter 1. About OpenShift Virtualization

Learn about OpenShift Virtualization’s capabilities and support scope.

1.1. What you can do with OpenShift Virtualization

OpenShift Virtualization is an add-on to OpenShift Container Platform that allows you to run and manage virtual machine workloads alongside container workloads.

OpenShift Virtualization adds new objects into your OpenShift Container Platform cluster by using Kubernetes custom resources to enable virtualization tasks. These tasks include:

  • Creating and managing Linux and Windows virtual machines (VMs)
  • Running pod and VM workloads alongside each other in a cluster
  • Connecting to virtual machines through a variety of consoles and CLI tools
  • Importing and cloning existing virtual machines
  • Managing network interface controllers and storage disks attached to virtual machines
  • Live migrating virtual machines between nodes

An enhanced web console provides a graphical portal to manage these virtualized resources alongside the OpenShift Container Platform cluster containers and infrastructure.

OpenShift Virtualization is designed and tested to work well with Red Hat OpenShift Data Foundation features.

Important

When you deploy OpenShift Virtualization with OpenShift Data Foundation, you must create a dedicated storage class for Windows virtual machine disks. See Optimizing ODF PersistentVolumes for Windows VMs for details.

You can use OpenShift Virtualization with the OVN-Kubernetes, OpenShift SDN, or one of the other certified network plugins listed in Certified OpenShift CNI Plug-ins.

You can check your OpenShift Virtualization cluster for compliance issues by installing the Compliance Operator and running a scan with the ocp4-moderate and ocp4-moderate-node profiles. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies.

1.1.1. OpenShift Virtualization supported cluster version

OpenShift Virtualization 4.13 is supported for use on OpenShift Container Platform 4.13 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.

1.2. About storage volumes for virtual machine disks

If you use the storage API with known storage providers, volume and access modes are selected automatically. However, if you use a storage class that does not have a storage profile, you must select the volume and access mode.

For best results, use accessMode: ReadWriteMany and volumeMode: Block. This is important for the following reasons:

  • The ReadWriteMany (RWX) access mode is required for live migration.
  • The Block volume mode performs significantly better in comparison to the Filesystem volume mode. This is because the Filesystem volume mode uses more storage layers, including a file system layer and a disk image file. These layers are not necessary for VM disk storage.

    For example, if you use Red Hat OpenShift Data Foundation, Ceph RBD volumes are preferable to CephFS volumes.

    Important

    You cannot live migrate virtual machines that use:

    • A storage volume with ReadWriteOnce (RWO) access mode
    • Passthrough features such as GPUs

    Do not set the evictionStrategy field to LiveMigrate for these virtual machines.

1.3. Single-node OpenShift differences

You can install OpenShift Virtualization on single-node OpenShift.

However, you should be aware that Single-node OpenShift does not support the following features:

  • High availability
  • Pod disruption
  • Live migration
  • Virtual machines or templates that have an eviction strategy configured

1.4. Additional resources

Chapter 2. OpenShift Virtualization architecture

Learn about OpenShift Virtualization architecture.

2.1. How OpenShift Virtualization architecture works

After you install OpenShift Virtualization, the Operator Lifecycle Manager (OLM) deploys operator pods for each component of OpenShift Virtualization:

  • Compute: virt-operator
  • Storage: cdi-operator
  • Network: cluster-network-addons-operator
  • Scaling: ssp-operator
  • Templating: tekton-tasks-operator

OLM also deploys the hyperconverged-cluster-operator pod, which is responsible for the deployment, configuration, and life cycle of other components, and several helper pods: hco-webhook, and hyperconverged-cluster-cli-download.

After all operator pods are successfully deployed, you should create the HyperConverged custom resource (CR). The configurations set in the HyperConverged CR serve as the single source of truth and the entrypoint for OpenShift Virtualization, and guide the behavior of the CRs.

The HyperConverged CR creates corresponding CRs for the operators of all other components within its reconciliation loop. Each operator then creates resources such as daemon sets, config maps, and additional components for the OpenShift Virtualization control plane. For example, when the hco-operator creates the KubeVirt CR, the virt-operator reconciles it and create additional resources such as virt-controller, virt-handler, and virt-api.

The OLM deploys the hostpath-provisioner-operator, but it is not functional until you create a hostpath provisioner (HPP) CR.

CNV Deployments

2.2. About the hco-operator

The hco-operator (HCO) provides a single entry point for deploying and managing OpenShift Virtualization and several helper operators with opinionated defaults. It also creates custom resources (CRs) for those operators.

hco-operator components
Table 2.1. hco-operator components
ComponentDescription

deployment/hco-webhook

Validates the HyperConverged custom resource contents.

deployment/hyperconverged-cluster-cli-download

Provides the virtctl tool binaries to the cluster so that you can download them directly from the cluster.

KubeVirt/kubevirt-kubevirt-hyperconverged

Contains all operators, CRs, and objects needed by OpenShift Virtualization.

SSP/ssp-kubevirt-hyperconverged

An SSP CR. This is automatically created by the HCO.

CDI/cdi-kubevirt-hyperconverged

A CDI CR. This is automatically created by the HCO.

NetworkAddonsConfig/cluster

A CR that instructs and is managed by the cluster-network-addons-operator.

2.3. About the cdi-operator

The cdi-operator manages the Containerized Data Importer (CDI), and its related resources, which imports a virtual machine (VM) image into a persistent volume claim (PVC) by using a data volume.

cdi-operator components
Table 2.2. cdi-operator components
ComponentDescription

deployment/cdi-apiserver

Manages the authorization to upload VM disks into PVCs by issuing secure upload tokens.

deployment/cdi-uploadproxy

Directs external disk upload traffic to the appropriate upload server pod so that it can be written to the correct PVC. Requires a valid upload token.

pod/cdi-importer

Helper pod that imports a virtual machine image into a PVC when creating a data volume.

2.4. About the cluster-network-addons-operator

The cluster-network-addons-operator deploys networking components on a cluster and manages the related resources for extended network functionality.

cluster-network-addons-operator components
Table 2.3. cluster-network-addons-operator components
ComponentDescription

deployment/kubemacpool-cert-manager

Manages TLS certificates of Kubemacpool’s webhooks.

deployment/kubemacpool-mac-controller-manager

Provides a MAC address pooling service for virtual machine (VM) network interface cards (NICs).

daemonset/bridge-marker

Marks network bridges available on nodes as node resources.

daemonset/kube-cni-linux-bridge-plugin

Installs CNI plugins on cluster nodes, enabling the attachment of VMs to Linux bridges through network attachment definitions.

2.5. About the hostpath-provisioner-operator

The hostpath-provisioner-operator deploys and manages the multi-node hostpath provisioner (HPP) and related resources.

hpp-operator components
Table 2.4. hostpath-provisioner-operator components
ComponentDescription

deployment/hpp-pool-hpp-csi-pvc-block-<worker_node_name>

Provides a worker for each node where the hostpath provisioner (HPP) is designated to run. The pods mount the specified backing storage on the node.

daemonset/hostpath-provisioner-csi

Implements the Container Storage Interface (CSI) driver interface of the HPP.

daemonset/hostpath-provisioner

Implements the legacy driver interface of the HPP.

2.6. About the ssp-operator

The ssp-operator deploys the common templates, the related default boot sources, and the template validator.

ssp-operator components
Table 2.5. ssp-operator components
ComponentDescription

deployment/virt-template-validator

Checks vm.kubevirt.io/validations annotations on virtual machines created from templates, and rejects them if they are invalid.

2.7. About the tekton-tasks-operator

The tekton-tasks-operator deploys example pipelines showing the usage of OpenShift Pipelines for VMs. It also deploys additional OpenShift Pipeline tasks that allow users to create VMs from templates, copy and modify templates, and create data volumes.

tekton-tasks-operator components
Table 2.6. tekton-tasks-operator components
ComponentDescription

deployment/create-vm-from-template

Creates a VM from a template.

deployment/copy-template

Copies a VM template.

deployment/modify-vm-template

Creates or removes a VM template.

deployment/modify-data-object

Creates or removes data volumes or data sources.

deployment/cleanup-vm

Runs a script or a command on a VM, then stops or deletes the VM afterward.

deployment/disk-virt-customize

Runs a customize script on a target PVC using virt-customize.

deployment/disk-virt-sysprep

Runs a sysprep script on a target PVC by using virt-sysprep.

deployment/wait-for-vmi-status

Waits for a specific VMI status, then fails or succeeds according to that status.

2.8. About the virt-operator

The virt-operator deploys, upgrades, and manages OpenShift Virtualization without disrupting current virtual machine (VM) workloads.

virt-operator components
Table 2.7. virt-operator components
ComponentDescription

deployment/virt-api

HTTP API server that serves as the entry point for all virtualization-related flows.

deployment/virt-controller

Observes the creation of a new VM instance object and creates a corresponding pod. When the pod is scheduled on a node, virt-controller updates the VM with the node name.

daemonset/virt-handler

Monitors any changes to a VM and instructs virt-launcher to perform the required operations. This component is node-specific.

pod/virt-launcher

Contains the VM that was created by the user as implemented by libvirt and qemu.

Chapter 3. Getting started with OpenShift Virtualization

You can explore the features and functionalities of OpenShift Virtualization by installing and configuring a basic environment.

Note

Cluster configuration procedures require cluster-admin privileges.

3.1. Planning and installing OpenShift Virtualization

Plan and install OpenShift Virtualization on an OpenShift Container Platform cluster:

Planning and installation resources

3.2. Creating and managing virtual machines

Create virtual machines (VMs) by using the web console:

Connect to VMs:

Manage VMs:

3.3. Next steps

Chapter 4. Web console overview

The Virtualization section of the OpenShift Container Platform web console contains the following pages for managing and monitoring your OpenShift Virtualization environment.

Table 4.1. Virtualization pages
PageDescription

Overview page

Manage and monitor the OpenShift Virtualization environment.

Catalog page

Create VirtualMachines from a catalog of templates.

VirtualMachines page

Configure and monitor VirtualMachines.

Templates page

Create and manage templates.

DataSources page

Create and manage DataSources for VirtualMachine boot sources.

MigrationPolicies page

Create and manage MigrationPolicies for workloads.

Table 4.2. Key
IconDescription

icon pencil

Edit icon

icon link

Link icon

4.1. Overview page

The Overview page displays resources, metrics, migration progress, and cluster-level settings.

Example 4.1. Overview page

ElementDescription

Download virtctl icon link

Download the virtctl command line tool to manage resources.

Overview tab

Resources, usage, alerts, and status

Top consumers tab

Top consumers of CPU, memory, and storage resources

Migrations tab

Status of live migrations

Settings tab

Cluster-wide settings, including live migration limits and user permissions

4.1.1. Overview tab

The Overview tab displays resources, usage, alerts, and status.

Example 4.2. Overview tab

ElementDescription

Getting started resources card

  • Quick Starts tile: Learn how to create, import, and run VirtualMachines with step-by-step instructions and tasks.
  • Feature highlights tile: Read the latest information about key virtualization features.
  • Related operators tile: Install Operators such as the Kubernetes NMState Operator or the OpenShift Data Foundation Operator.

VirtualMachines tile

Number of VirtualMachines, with a chart showing the last 7 days' trend

vCPU usage tile

vCPU usage, with a chart showing the last 7 days' trend

Memory tile

Memory usage, with a chart showing the last 7 days' trend

Storage tile

Storage usage, with a chart showing the last 7 days' trend

Alerts tile

OpenShift Virtualization alerts, grouped by severity

VirtualMachine statuses tile

Number of VirtualMachines, grouped by status

VirtualMachines per template chart

Number of VirtualMachines created from templates, grouped by template name

4.1.2. Top consumers tab

The Top consumers tab displays the top consumers of CPU, memory, and storage.

Example 4.3. Top consumers tab

ElementDescription

View virtualization dashboard icon link

Link to Observe → Dashboards, which displays the top consumers for OpenShift Virtualization

Time period list

Select a time period to filter the results.

Top consumers list

Select the number of top consumers to filter the results.

CPU chart

VirtualMachines with the highest CPU usage

Memory chart

VirtualMachines with the highest memory usage

Memory swap traffic chart

VirtualMachines with the highest memory swap traffic

vCPU wait chart

VirtualMachines with the highest vCPU wait periods

Storage throughput chart

VirtualMachines with the highest storage throughput usage

Storage IOPS chart

VirtualMachines with the highest storage input/output operations per second usage

4.1.3. Migrations tab

The Migrations tab displays the status of VirtualMachineInstance migrations.

Example 4.4. Migrations tab

ElementDescription

Time period list

Select a time period to filter VirtualMachineInstanceMigrations.

VirtualMachineInstanceMigrations table

List of VirtualMachineInstance migrations

4.1.4. Settings tab

The Settings tab displays cluster-wide settings on the following tabs:

Table 4.3. Tabs on Settings tab
TabDescription

General tab

OpenShift Virtualization version and update status

Live migration tab

Live migration limits and network settings

Templates project tab

Project for Red Hat templates

User permissions tab

Cluster-wide user permissions

4.1.4.1. General tab

The General tab displays the OpenShift Virtualization version and update status.

Example 4.5. General tab

LabelDescription

Service name

OpenShift Virtualization

Provider

Red Hat

Installed version

4.13.11

Update status

Example: Up to date

Channel

Channel selected for updates

4.1.4.2. Live migration tab

You can configure live migration on the Live migration tab.

Example 4.6. Live migration tab

ElementDescription

Max. migrations per cluster field

Select the maximum number of live migrations per cluster.

Max. migrations per node field

Select the maximum number of live migrations per node.

Live migration network list

Select a dedicated secondary network for live migration.

4.1.4.3. Templates project tab

You can select a project for templates on the Templates project tab.

Example 4.7. Templates project tab

ElementDescription

Project list

Select a project in which to store Red Hat templates. The default template project is openshift.

If you want to define multiple template projects, you must clone the templates on the Templates page for each project.

4.1.4.4. User permissions tab

The User permissions tab displays cluster-wide user permissions for tasks.

Example 4.8. User permissions tab

ElementDescription

User permissions table

List of tasks, such as Share templates, and permissions

4.2. Catalog page

You can create a VirtualMachine by selecting a template or boot source on the Catalog page.

Example 4.9. Catalog page

ElementDescription

Template catalog tab

Select a template to create a VirtualMachine from.

4.2.1. Template catalog tab

ElementDescription

Template project list

Select the project in which your templates are located.

By default, Red Hat templates are stored in the openshift project. You can edit the template project on the Overview → Settings → Template project tab.

All items|Default templates

Click Default templates to display only default templates.

Boot source available checkbox

Select the checkbox to display templates with an available boot source.

Operating system checkboxes

Select checkboxes to display templates with selected operating systems.

Workload checkboxes

Select checkboxes to display templates with selected workloads.

Search field

Search templates by keyword.

Template tiles

Click a template tile to view template details and to create a VirtualMachine.

4.3. VirtualMachines page

You can create and manage VirtualMachines on the VirtualMachines page.

Example 4.10. VirtualMachines page

ElementDescription

Create → From template

Create a VirtualMachine on the Catalog page → Template catalog tab.

Create → From YAML

Create a VirtualMachine by editing a YAML configuration file.

Filter field

Filter VirtualMachines by status, template, operating system, or node.

Search field

Search for VirtualMachines by name or by label.

VirtualMachines table

List of VirtualMachines

Click the Options menu kebab beside a VirtualMachine to select Stop, Restart, Pause, Clone, Migrate, Copy SSH command, Edit labels, Edit annotations, or Delete.

Click a VirtualMachine to navigate to the VirtualMachine details page.

4.3.1. VirtualMachine details page

You can configure a VirtualMachine on the VirtualMachine details page.

Example 4.11. VirtualMachine details page

ElementDescription

Actions menu

Click the Actions menu to select Stop, Restart, Pause, Clone, Migrate, Copy SSH command, Edit labels, Edit annotations, or Delete.

Overview tab

Resource usage, alerts, disks, and devices

Details tab

VirtualMachine details and configurations

Metrics tab

Memory, CPU, storage, network, and migration metrics

YAML tab

VirtualMachine YAML configuration file

Configuration tab

Contains the Scheduling, Environment, Network interfaces, Disks, and Scripts tabs

ConfigurationScheduling tab

Scheduling a VirtualMachine to run on specific nodes

ConfigurationEnvironment tab

Config map, secret, and service account management

ConfigurationNetwork interfaces tab

Network interfaces

ConfigurationDisks tab

Disks

ConfigurationScripts tab

Cloud-init settings, SSH key for Linux VirtualMachines, Sysprep answer file for Windows VirtualMachines

Events tab

VirtualMachine event stream

Console tab

Console session management

Snapshots tab

Snapshot management

Diagnostics tab

Status conditions and volume snapshot status

4.3.1.1. Overview tab

The Overview tab displays resource usage, alerts, and configuration information.

Example 4.12. Overview tab

ElementDescription

Details tile

General VirtualMachine information

Utilization tile

CPU, Memory, Storage, and Network transfer charts. By default, Network transfer displays the sum of all networks. To view the breakdown for a specific network, click Breakdown by network.

Hardware devices tile

GPU and host devices

Alerts tile

OpenShift Virtualization alerts, grouped by severity

Snapshots tile

Take snapshot icon link and Snapshots table

Network interfaces tile

Network interfaces table

Disks tile

Disks table

4.3.1.2. Details tab

You can view information about the VirtualMachine and edit labels, annotations, and other metadata and on the Details tab.

Example 4.13. Details tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Name

VirtualMachine name

Namespace

VirtualMachine namespace

Labels

Click the edit icon to edit the labels.

Annotations

Click the edit icon to edit the annotations.

Description

Click the edit icon to enter a description.

Operating system

Operating system name

CPU|Memory

Click the edit icon to edit the CPU|Memory request.

The number of CPUs is calculated by using the following formula: sockets * threads * cores.

Machine type

VirtualMachine machine type

Boot mode

Click the edit icon to edit the boot mode.

Start in pause mode

Click the edit icon to enable this setting.

Template

Name of the template used to create the VirtualMachine

Created at

VirtualMachine creation date

Owner

VirtualMachine owner

Status

VirtualMachine status

Pod

virt-launcher pod name

VirtualMachineInstance

VirtualMachineInstance name

Boot order

Click the edit icon to select a boot source.

IP address

IP address of the VirtualMachine

Hostname

Hostname of the VirtualMachine

Time zone

Time zone of the VirtualMachine

Node

Node on which the VirtualMachine is running

Workload profile

Click the edit icon to edit the workload profile.

SSH using virtctl

Click the copy icon to copy the virtctl ssh command to the clipboard.

SSH service type options

Select SSH over LoadBalancer or SSH over NodePort.

GPU devices

Click the edit icon to add a GPU device.

Host devices

Click the edit icon to add a host device.

Headless mode

Click the edit icon to enable headless mode.

Services section

Displays services if QEMU guest agent is installed.

Active users section

Displays active users if QEMU guest agent is installed.

4.3.1.3. Metrics tab

The Metrics tab displays memory, CPU, storage, network, and migration usage charts.

Example 4.14. Metrics tab

ElementDescription

Time range list

Select a time range to filter the results.

Virtualization dashboard icon link

Link to the Workloads tab of the current project

Utilization section

Memory and CPU charts

Storage section

Storage total read/write and Storage IOPS total read/write charts

Network section

Network in, Network out, Network bandwidth, and Network interface charts. Select All networks or a specific network from the Network interface dropdown.

Migration section

Migration and KV data transfer rate charts

4.3.1.4. YAML tab

You can configure the VirtualMachine by editing the YAML file on the YAML tab.

Example 4.15. YAML tab

ElementDescription

Save button

Save changes to the YAML file.

Reload button

Discard your changes and reload the YAML file.

Cancel button

Exit the YAML tab.

Download button

Download the YAML file to your local machine.

4.3.1.5. Configuration tab

You can configure scheduling, network interfaces, disks, and other options on the Configuration tab.

Example 4.16. Tabs on the Configuration tab

TabDescription

Scheduling tab

Scheduling a VirtualMachine to run on specific nodes

Environment tab

Config maps, secrets, and service accounts

Network interfaces tab

Network interfaces

Disks tab

Disks

Scripts tab

Cloud-init settings, SSH key for Linux VirtualMachines, Sysprep answer file for Windows VirtualMachines

4.3.1.5.1. Scheduling tab

You can configure VirtualMachines to run on specific nodes on the Scheduling tab.

Example 4.17. Scheduling tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Node selector

Click the edit icon to add a label to specify qualifying nodes.

Tolerations

Click the edit icon to add a toleration to specify qualifying nodes.

Affinity rules

Click the edit icon to add an affinity rule.

Descheduler switch

Enable or disable the descheduler. The descheduler evicts a running pod so that the pod can be rescheduled onto a more suitable node.

Dedicated resources

Click the edit icon to select Schedule this workload with dedicated resources (guaranteed policy).

Eviction strategy

Click the edit icon to select LiveMigrate as the VirtualMachineInstance eviction strategy.

4.3.1.5.2. Environment tab

You can manage config maps, secrets, and service accounts on the Environment tab.

Example 4.18. Environment tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Add Config Map, Secret or Service Account icon link

Click the link and select a config map, secret, or service account from the resource list.

4.3.1.5.3. Network interfaces tab

You can manage network interfaces on the Network interfaces tab.

Example 4.19. Network interfaces tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Add network interface button

Add a network interface to the VirtualMachine.

Filter field

Filter by interface type.

Search field

Search for a network interface by name or by label.

Network interface table

List of network interfaces

Click the Options menu kebab beside a network interface to select Edit or Delete.

4.3.1.5.4. Disks tab

You can manage disks on the Disks tab.

Example 4.20. Disks tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Add disk button

Add a disk to the VirtualMachine.

Filter field

Filter by disk type.

Search field

Search for a disk by name.

Mount Windows drivers disk checkbox

Select to mount an ephemeral container disk as a CD-ROM.

Disks table

List of VirtualMachine disks

Click the Options menu kebab beside a disk to select Edit, Detach, or Make persistent.

File systems table

List of VirtualMachine file systems if QEMU guest agent is installed

4.3.1.5.5. Scripts tab

You can configure cloud-init, add an SSH key for a Linux VirtualMachine, and upload a Sysprep answer file for a Windows VirtualMachine on the Scripts tab.

Example 4.21. Scripts tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Cloud-init

Click the edit icon to edit the cloud-init settings.

Authorized SSH Key

Click the edit icon to create a new secret or to attach an existing secret.

Sysprep

Click the edit icon to upload an Autounattend.xml or Unattend.xml answer file to automate Windows VirtualMachine setup.

4.3.1.6. Events tab

The Events tab displays a list of VirtualMachine events.

4.3.1.7. Console tab

You can open a console session to the VirtualMachine on the Console tab.

Example 4.22. Console tab

ElementDescription

Guest login credentials section

Expand Guest login credentials to view the credentials created with cloud-init. Click the copy icon to copy the credentials to the clipboard.

Console list

Select VNC console or Serial console.

You can select Desktop viewer to connect to Windows VirtualMachines by using Remote Desktop Protocol (RDP). You must install an RDP client on a machine on the same network.

Send key list

Select a key-stroke combination to send to the console.

Disconnect button

Disconnect the console connection.

You must manually disconnect the console connection if you open a new console session. Otherwise, the first console session continues to run in the background.

Paste button

You can paste a string from your client’s clipboard into the guest when using the VNC console.

4.3.1.8. Snapshots tab

You can create snapshots and restore VirtualMachines from snapshots on the Snapshots tab.

Example 4.23. Snapshots tab

ElementDescription

Take snapshot button

Create a snapshot.

Filter field

Filter snapshots by status.

Search field

Search for snapshots by name or by label.

Snapshot table

List of snapshots

Click the snapshot name to edit the labels or annotations.

Click the Options menu kebab beside a snapshot to select Restore or Delete.

4.3.1.9. Diagnostics tab

You can view the status conditions and volume snapshot status on the Diagnostics tab.

Example 4.24. Diagnostics tab

ElementDescription

Status conditions table

Display a list of conditions that are reported for all aspects of a VM.

Filter field

Filter status conditions by category and condition.

Search field

Search status conditions by reason.

Manage columns icon

Select columns to display.

Volume snapshot table

List of volumes, their snapshot enablement status, and reason

4.4. Templates page

You can create, edit, and clone VirtualMachine templates on the Templates page.

Note

You cannot edit a Red Hat template. You can clone a Red Hat template and edit it to create a custom template.

Example 4.25. Templates page

ElementDescription

Create Template button

Create a template by editing a YAML configuration file.

Filter field

Filter templates by type, boot source, template provider, or operating system.

Search field

Search for templates by name or by label.

Templates table

List of templates

Click the Options menu kebab beside a template to select Edit, Clone, Edit boot source, Edit boot source reference, Edit labels, Edit annotations, or Delete.

4.4.1. Template details page

You can view template settings and edit custom templates on the Template details page.

Example 4.26. Template details page

ElementDescription

Actions menu

Click the Actions menu to select Edit, Clone, Edit boot source, Edit boot source reference, Edit labels, Edit annotations, or Delete.

Details tab

Template settings and configurations

YAML tab

YAML configuration file

Scheduling tab

Scheduling configurations

Network interfaces tab

Network interface management

Disks tab

Disk management

Scripts tab

Cloud-init, SSH key, and Sysprep management

Parameters tab

Parameters

4.4.1.1. Details tab

You can configure a custom template on the Details tab.

Example 4.27. Details tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Name

Template name

Namespace

Template namespace

Labels

Click the edit icon to edit the labels.

Annotations

Click the edit icon to edit the annotations.

Display name

Click the edit icon to edit the display name.

Description

Click the edit icon to enter a description.

Operating system

Operating system name

CPU|Memory

Click the edit icon to edit the CPU|Memory request.

The number of CPUs is calculated by using the following formula: sockets * threads * cores.

Machine type

Template machine type

Boot mode

Click the edit icon to edit the boot mode.

Base template

Name of the base template used to create this template

Created at

Template creation date

Owner

Template owner

Boot order

Template boot order

Boot source

Boot source availability

Provider

Template provider

Support

Template support level

GPU devices

Click the edit icon to add a GPU device.

Host devices

Click the edit icon to add a host device.

4.4.1.2. YAML tab

You can configure a custom template by editing the YAML file on the YAML tab.

Example 4.28. YAML tab

ElementDescription

Save button

Save changes to the YAML file.

Reload button

Discard your changes and reload the YAML file.

Cancel button

Exit the YAML tab.

Download button

Download the YAML file to your local machine.

4.4.1.3. Scheduling tab

You can configure scheduling on the Scheduling tab.

Example 4.29. Scheduling tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Node selector

Click the edit icon to add a label to specify qualifying nodes.

Tolerations

Click the edit icon to add a toleration to specify qualifying nodes.

Affinity rules

Click the edit icon to add an affinity rule.

Descheduler switch

Enable or disable the descheduler. The descheduler evicts a running pod so that the pod can be rescheduled onto a more suitable node.

Dedicated resources

Click the edit icon to select Schedule this workload with dedicated resources (guaranteed policy).

Eviction strategy

Click the edit icon to select LiveMigrate as the VirtualMachineInstance eviction strategy.

4.4.1.4. Network interfaces tab

You can manage network interfaces on the Network interfaces tab.

Example 4.30. Network interfaces tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Add network interface button

Add a network interface to the template.

Filter field

Filter by interface type.

Search field

Search for a network interface by name or by label.

Network interface table

List of network interfaces

Click the Options menu kebab beside a network interface to select Edit or Delete.

4.4.1.5. Disks tab

You can manage disks on the Disks tab.

Example 4.31. Disks tab

SettingDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Add disk button

Add a disk to the template.

Filter field

Filter by disk type.

Search field

Search for a disk by name.

Disks table

List of template disks

Click the Options menu kebab beside a disk to select Edit or Detach.

4.4.1.6. Scripts tab

You can manage the cloud-init settings, SSH keys, and Sysprep answer files on the Scripts tab.

Example 4.32. Scripts tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

Cloud-init

Click the edit icon to edit the cloud-init settings.

Authorized SSH Key

Click the edit icon to create a new secret or to attach an existing secret.

Sysprep

Click the edit icon to upload an Autounattend.xml or Unattend.xml answer file to automate Windows VirtualMachine setup.

4.4.1.7. Parameters tab

You can edit selected template settings on the Parameters tab.

Example 4.33. Parameters tab

ElementDescription

YAML switch

Set to ON to view your live changes in the YAML configuration file.

VM name

Select Generated (expression) for a generated value, Value to set a default value, or None from the Default value type list.

DataSource name

Select Generated (expression) for a generated value, Value to set a default value, or None from the Default value type list.

DataSource namespace

Select Generated (expression) for a generated value, Value to set a default value, or None from the Default value type list.

Cloud user password

Select Generated (expression) for a generated value, Value to set a default value, or None from the Default value type list.

4.5. DataSources page

You can create and configure DataSources for VirtualMachine boot sources on the DataSources page.

When you create a DataSource, a DataImportCron resource defines a cron job to poll and import the disk image unless you disable automatic boot source updates.

Example 4.34. DataSources page

ElementDescription

Create DataSource → With form

Create a DataSource by entering the registry URL, disk size, number of revisions, and cron expression in a form.

Create DataSources → With YAML

Create a DataSource by editing a YAML configuration file.

Filter field

Filter DataSources by attributes such as DataImportCron available.

Search field

Search for a DataSource by name or by label.

DataSources table

List of DataSources

Click the Options menu kebab beside a DataSource to select Edit labels, Edit annotations, or Delete.

Click a DataSource to view the DataSource details page.

4.5.1. DataSource details page

You can configure a DataSource on the DataSource details page.

Example 4.35. DataSource details page

ElementDescription

Details tab

Configure a DataSource by editing a form.

YAML tab

Configure a DataSource by editing a YAML configuration file.

Actions menu

Select Edit labels, Edit annotations, Delete, or Manage source.

Name

DataSource name

Namespace

DataSource namespace

DataImportCron

DataSource DataImportCron

Labels

Click the edit icon to edit the labels.

Annotations

Click the edit icon to edit the annotations.

Conditions

Displays the status conditions of the DataSource.

Created at

DataSource creation date

Owner

DataSource owner

4.6. MigrationPolicies page

You can manage MigrationPolicies for your workloads on the MigrationPolicies page.

Example 4.36. MigrationPolicies page

ElementDescription

Create MigrationPolicy → With form

Create a MigrationPolicy by entering configurations and labels in a form.

Create MigrationPolicy → With YAML

Create a MigrationPolicy by editing a YAML configuration file.

Name | Label search field

Search for a MigrationPolicy by name or by label.

MigrationPolicies table

List of MigrationPolicies

Click the Options menu kebab beside a MigrationPolicy to select Edit or Delete.

Click a MigrationPolicy to view the MigrationPolicy details page.

4.6.1. MigrationPolicy details page

You can configure a MigrationPolicy on the MigrationPolicy details page.

Example 4.37. MigrationPolicy details page

ElementDescription

Details tab

Configure a MigrationPolicy by editing a form.

YAML tab

Configure a MigrationPolicy by editing a YAML configuration file.

Actions menu

Select Edit or Delete.

Name

MigrationPolicy name

Description

MigrationPolicy description

Configurations

Click the edit icon to update the MigrationPolicy configurations.

Bandwidth per migration

Bandwidth request per migration. For unlimited bandwidth, set the value to 0.

Auto converge

Auto converge policy

Post-copy

Post-copy policy

Completion timeout

Completion timeout value in seconds

Project labels

Click Edit to edit the project labels.

VirtualMachine labels

Click Edit to edit the VirtualMachine labels.

Chapter 5. OpenShift Virtualization release notes

5.1. Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

5.2. About Red Hat OpenShift Virtualization

Red Hat OpenShift Virtualization enables you to bring traditional virtual machines (VMs) into OpenShift Container Platform where they run alongside containers, and are managed as native Kubernetes objects.

OpenShift Virtualization is represented by the OpenShift Virtualization icon.

You can use OpenShift Virtualization with either the OVN-Kubernetes or the OpenShiftSDN default Container Network Interface (CNI) network provider.

Learn more about what you can do with OpenShift Virtualization.

Learn more about OpenShift Virtualization architecture and deployments.

Prepare your cluster for OpenShift Virtualization.

5.2.1. OpenShift Virtualization supported cluster version

OpenShift Virtualization 4.13 is supported for use on OpenShift Container Platform 4.13 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.

Important

Updating to OpenShift Virtualization 4.13 from OpenShift Virtualization 4.12.2 is not supported.

5.2.2. Supported guest operating systems

To view the supported guest operating systems for OpenShift Virtualization, see Certified Guest Operating Systems in Red Hat OpenStack Platform, Red Hat Virtualization, OpenShift Virtualization and Red Hat Enterprise Linux with KVM.

5.3. New and changed features

  • OpenShift Virtualization is FIPS ready. However, OpenShift Container Platform 4.13 is based on Red Hat Enterprise Linux (RHEL) 9.2. RHEL 9.2 has not yet been submitted for FIPS validation. Red Hat expects, though cannot commit to a specific timeframe, to obtain FIPS validation for RHEL 9.0 and RHEL 9.2 modules, and later even minor releases of RHEL 9.x. Updates will be available in Compliance Activities and Government Standards.
  • OpenShift Virtualization is certified in Microsoft’s Windows Server Virtualization Validation Program (SVVP) to run Windows Server workloads.

    The SVVP Certification applies to:

    • Red Hat Enterprise Linux CoreOS workers. In the Microsoft SVVP Catalog, they are named Red Hat OpenShift Container Platform 4 on RHEL CoreOS 9.
    • Intel and AMD CPUs.
  • OpenShift Virtualization is now based on Red Hat Enterprise Linux (RHEL) 9.

    • There is a new RHEL 9 machine type for VMs: machineType: pc-q35-rhel9.2.0.

      All VM templates that are included with OpenShift Virtualization now use this machine type by default.

    • For more information, see OpenShift Virtualization on RHEL 9.
  • You can now obtain the VirtualMachine, ConfigMap, and Secret manifests from the export server after you export a VM or snapshot. For more information, see accessing exported VM manifests.
  • The "Logging, events, and monitoring" documentation is now called Support. The monitoring tools documentation has been moved to Monitoring.
  • You can view and filter aggregated OpenShift Virtualization logs in the web console by using the LokiStack.

5.3.1. Quick starts

  • Quick start tours are available for several OpenShift Virtualization features. To view the tours, click the Help icon ? in the menu bar on the header of the OpenShift Virtualization console and then select Quick Starts. You can filter the available tours by entering the virtualization keyword in the Filter field.

5.3.2. Networking

5.3.3. Storage

  • OpenShift Virtualization storage resources now migrate automatically to the beta API versions. Alpha API versions are no longer supported.

5.3.4. Web console

  • On the VirtualMachine details page, the Scheduling, Environment, Network interfaces, Disks, and Scripts tabs are displayed on the new Configuration tab.
  • You can now paste a string from your client’s clipboard into the guest when using the VNC console.
  • The VirtualMachine detailsDetails tab now provides a new SSH service type SSH over LoadBalancer to expose the SSH service over a load balancer.
  • The option to make a hot-plug volume a persistent volume is added to the Disks tab.
  • You can now enable headless mode for high performance VMs in the web console.

5.4. Deprecated and removed features

5.4.1. Deprecated features

Deprecated features are included and supported in the current release. However, they will be removed in a future release and are not recommended for new deployments.

  • Support for virtctl command line tool installation for Red Hat Enterprise Linux (RHEL) 7 and RHEL 9 by an RPM is deprecated and is planned to be removed in a future release.

5.4.2. Removed features

Removed features are not supported in the current release.

  • Red Hat Enterprise Linux 6 is no longer supported on OpenShift Virtualization.
  • Support for the legacy HPP custom resource, and the associated storage class, has been removed for all new deployments. In OpenShift Virtualization 4.13, the HPP Operator uses the Kubernetes Container Storage Interface (CSI) driver to configure local storage. A legacy HPP custom resource is supported only if it had been installed on a previous version of OpenShift Virtualization.

5.5. Technology Preview features

Some features in this release are currently in Technology Preview. These experimental features are not intended for production use. Note the following scope of support on the Red Hat Customer Portal for these features:

Technology Preview Features Support Scope

  • You can now use Prometheus to monitor the following metrics:

    • kubevirt_vmi_cpu_system_usage_seconds returns the physical system CPU time consumed by the hypervisor.
    • kubevirt_vmi_cpu_user_usage_seconds returns the physical user CPU time consumed by the hypervisor.
    • kubevirt_vmi_cpu_usage_seconds returns the total CPU time used in seconds by calculating the sum of the vCPU and the hypervisor usage.
  • You can now run a checkup to verify if your OpenShift Container Platform cluster node can run a virtual machine with a Data Plane Development Kit (DPDK) workload with zero packet loss.
  • You can now use Microsoft Windows 11 as a guest operating system. However, OpenShift Virtualization 4.13 does not support USB disks, which are required for a critical function of BitLocker recovery. To protect recovery keys, use other methods described in the BitLocker recovery guide.

5.6. Bug fix

  • The virtual machine snapshot restore operation no longer hangs indefinitely due to some persistent volume claim (PVC) annotations created by the Containerized Data Importer (CDI). (BZ#2070366)

5.7. Known issues

  • With the release of the RHSA-2023:3722 advisory, the TLS Extended Master Secret (EMS) extension (RFC 7627) is mandatory for TLS 1.2 connections on FIPS-enabled RHEL 9 systems. This is in accordance with FIPS-140-3 requirements. TLS 1.3 is not affected.

    Legacy OpenSSL clients that do not support EMS or TLS 1.3 now cannot connect to FIPS servers running on RHEL 9. Similarly, RHEL 9 clients in FIPS mode cannot connect to servers that only support TLS 1.2 without EMS. This in practice means that these clients cannot connect to servers on RHEL 6, RHEL 7 and non-RHEL legacy operating systems. This is because the legacy 1.0.x versions of OpenSSL do not support EMS or TLS 1.3. For more information, see TLS Extension "Extended Master Secret" enforced with Red Hat Enterprise Linux 9.2.

    As a workaround, upgrade legacy OpenSSL clients to a version that supports TLS 1.3 and configure OpenShift Virtualization to use TLS 1.3, with the Modern TLS security profile type, for FIPS mode.

  • If you enabled the DisableMDEVConfiguration feature gate by editing the HyperConverged custom resource in OpenShift Virtualization 4.12.4, you must re-enable the feature gate after you upgrade to versions 4.13.0 or 4.13.1 by creating a JSON Patch annotation (BZ#2184439):

    $ oc annotate --overwrite -n openshift-cnv hyperconverged kubevirt-hyperconverged \
        kubevirt.kubevirt.io/jsonpatch='[{"op": "add","path": "/spec/configuration/developerConfiguration/featureGates/-", \
        "value": "DisableMDEVConfiguration"}]'
  • OpenShift Virtualization versions 4.12.2 and earlier are not compatible with OpenShift Container Platform 4.13. Updating OpenShift Container Platform to 4.13 is blocked by design in OpenShift Virtualization 4.12.1 and 4.12.2, but this restriction could not be added to OpenShift Virtualization 4.12.0. If you have OpenShift Virtualization 4.12.0, ensure that you do not update OpenShift Container Platform to 4.13.

    Important

    Your cluster becomes unsupported if you run incompatible versions of OpenShift Container Platform and OpenShift Virtualization.

  • Enabling descheduler evictions on a virtual machine is a Technical Preview feature and might cause failed migrations and unstable scheduling.
  • You cannot run OpenShift Virtualization on a single-stack IPv6 cluster. (BZ#2193267)
  • When you use two pods with different SELinux contexts, VMs with the ocs-storagecluster-cephfs storage class fail to migrate and the VM status changes to Paused. This is because both pods try to access the shared ReadWriteMany CephFS volume at the same time. (BZ#2092271)

    • As a workaround, use the ocs-storagecluster-ceph-rbd storage class to live migrate VMs on a cluster that uses Red Hat Ceph Storage.
  • If you clone more than 100 VMs using the csi-clone cloning strategy, then the Ceph CSI might not purge the clones. Manually deleting the clones might also fail. (BZ#2055595)

    • As a workaround, you can restart the ceph-mgr to purge the VM clones.
  • If you stop a node on a cluster and then use the Node Health Check Operator to bring the node back up, connectivity to Multus might be lost. (OCPBUGS-8398)
  • The TopoLVM provisioner name string has changed in OpenShift Virtualization 4.12. As a result, the automatic import of operating system images might fail with the following error message (BZ#2158521):

    DataVolume.storage spec is missing accessMode and volumeMode, cannot get access mode from StorageProfile.
    • As a workaround:

      1. Update the claimPropertySets array of the storage profile:

        $ oc patch storageprofile <storage_profile> --type=merge -p '{"spec": {"claimPropertySets": [{"accessModes": ["ReadWriteOnce"], "volumeMode": "Block"}, \
            {"accessModes": ["ReadWriteOnce"], "volumeMode": "Filesystem"}]}}'
      2. Delete the affected data volumes in the openshift-virtualization-os-images namespace. They are recreated with the access mode and volume mode from the updated storage profile.
  • When restoring a VM snapshot for storage whose binding mode is WaitForFirstConsumer, the restored PVCs remain in the Pending state and the restore operation does not progress.

    • As a workaround, start the restored VM, stop it, and then start it again. The VM will be scheduled, the PVCs will be in the Bound state, and the restore operation will complete. (BZ#2149654)
  • VMs created from common templates on a Single Node OpenShift (SNO) cluster display a VMCannotBeEvicted alert because the template’s default eviction strategy is LiveMigrate. You can ignore this alert or remove the alert by updating the VM’s eviction strategy. (BZ#2092412)
  • Uninstalling OpenShift Virtualization does not remove the feature.node.kubevirt.io node labels created by OpenShift Virtualization. You must remove the labels manually. (CNV-22036)
  • Windows 11 virtual machines do not boot on clusters running in FIPS mode. Windows 11 requires a TPM (trusted platform module) device by default. However, the swtpm (software TPM emulator) package is incompatible with FIPS. (BZ#2089301)
  • If your OpenShift Container Platform cluster uses OVN-Kubernetes as the default Container Network Interface (CNI) provider, you cannot attach a Linux bridge or bonding device to a host’s default interface because of a change in the host network topology of OVN-Kubernetes. (BZ#1885605)

    • As a workaround, you can use a secondary network interface connected to your host, or switch to the OpenShift SDN default CNI provider.
  • In some instances, multiple virtual machines can mount the same PVC in read-write mode, which might result in data corruption. (BZ#1992753)

    • As a workaround, avoid using a single PVC in read-write mode with multiple VMs.
  • The Pod Disruption Budget (PDB) prevents pod disruptions for migratable virtual machine images. If the PDB detects pod disruption, then openshift-monitoring sends a PodDisruptionBudgetAtLimit alert every 60 minutes for virtual machine images that use the LiveMigrate eviction strategy. (BZ#2026733)

  • OpenShift Virtualization links a service account token in use by a pod to that specific pod. OpenShift Virtualization implements a service account volume by creating a disk image that contains a token. If you migrate a VM, then the service account volume becomes invalid. (BZ#2037611)

    • As a workaround, use user accounts rather than service accounts because user account tokens are not bound to a specific pod.
  • In a heterogeneous cluster with different compute nodes, virtual machines that have HyperV Reenlightenment enabled cannot be scheduled on nodes that do not support timestamp-counter scaling (TSC) or have the appropriate TSC frequency. (BZ#2151169)
  • VMs that use logical volume management (LVM) with block storage devices require additional configuration to avoid conflicts with Red Hat Enterprise Linux CoreOS (RHCOS) hosts.

    • As a workaround, you can create a VM, provision an LVM, and restart the VM. This creates an empty system.lvmdevices file. (OCPBUGS-5223)

Chapter 6. Installing

6.1. Preparing your cluster for OpenShift Virtualization

Review this section before you install OpenShift Virtualization to ensure that your cluster meets the requirements.

Important

You can use any installation method, including user-provisioned, installer-provisioned, or assisted installer, to deploy OpenShift Container Platform. However, the installation method and the cluster topology might affect OpenShift Virtualization functionality, such as snapshots or live migration.

IPv6

You cannot run OpenShift Virtualization on a single-stack IPv6 cluster. (BZ#2193267)

6.1.1. Hardware and operating system requirements

Review the following hardware and operating system requirements for OpenShift Virtualization.

Supported platforms

Important

Installing OpenShift Virtualization on AWS bare metal instances or on IBM Cloud Bare Metal Servers is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

  • Bare metal instances or servers offered by other cloud providers are not supported.

CPU requirements

  • Supported by Red Hat Enterprise Linux (RHEL) 9
  • Support for AMD and Intel 64-bit architectures (x86-64-v2)
  • Support for Intel 64 or AMD64 CPU extensions
  • Intel VT or AMD-V hardware virtualization extensions enabled
  • NX (no execute) flag enabled

Storage requirements

  • Supported by OpenShift Container Platform
Warning

If you deploy OpenShift Virtualization with Red Hat OpenShift Data Foundation, you must create a dedicated storage class for Windows virtual machine disks. See Optimizing ODF PersistentVolumes for Windows VMs for details.

Operating system requirements

  • Red Hat Enterprise Linux CoreOS (RHCOS) installed on worker nodes

    Note

    RHEL worker nodes are not supported.

  • If your cluster uses worker nodes with different CPUs, live migration failures can occur because different CPUs have different capabilities. To avoid such failures, use CPUs with appropriate capacity for each node and set node affinity on your virtual machines to ensure successful migration. See Configuring a required node affinity rule for more information.

Additional resources

6.1.2. Physical resource overhead requirements

OpenShift Virtualization is an add-on to OpenShift Container Platform and imposes additional overhead that you must account for when planning a cluster. Each cluster machine must accommodate the following overhead requirements in addition to the OpenShift Container Platform requirements. Oversubscribing the physical resources in a cluster can affect performance.

Important

The numbers noted in this documentation are based on Red Hat’s test methodology and setup. These numbers can vary based on your own individual setup and environments.

6.1.2.1. Memory overhead

Calculate the memory overhead values for OpenShift Virtualization by using the equations below.

Cluster memory overhead

Memory overhead per infrastructure node ≈ 150 MiB

Memory overhead per worker node ≈ 360 MiB

Additionally, OpenShift Virtualization environment resources require a total of 2179 MiB of RAM that is spread across all infrastructure nodes.

Virtual machine memory overhead

Memory overhead per virtual machine ≈ (1.002 × requested memory) \
              + 218 MiB \ 1
              + 8 MiB × (number of vCPUs) \ 2
              + 16 MiB × (number of graphics devices) \ 3
              + (additional memory overhead) 4

1
Required for the processes that run in the virt-launcher pod.
2
Number of virtual CPUs requested by the virtual machine.
3
Number of virtual graphics cards requested by the virtual machine.
4
Additional memory overhead:
  • If your environment includes a Single Root I/O Virtualization (SR-IOV) network device or a Graphics Processing Unit (GPU), allocate 1 GiB additional memory overhead for each device.
6.1.2.2. CPU overhead

Calculate the cluster processor overhead requirements for OpenShift Virtualization by using the equation below. The CPU overhead per virtual machine depends on your individual setup.

Cluster CPU overhead

CPU overhead for infrastructure nodes ≈ 4 cores

OpenShift Virtualization increases the overall utilization of cluster level services such as logging, routing, and monitoring. To account for this workload, ensure that nodes that host infrastructure components have capacity allocated for 4 additional cores (4000 millicores) distributed across those nodes.

CPU overhead for worker nodes ≈ 2 cores + CPU overhead per virtual machine

Each worker node that hosts virtual machines must have capacity for 2 additional cores (2000 millicores) for OpenShift Virtualization management workloads in addition to the CPUs required for virtual machine workloads.

Virtual machine CPU overhead

If dedicated CPUs are requested, there is a 1:1 impact on the cluster CPU overhead requirement. Otherwise, there are no specific rules about how many CPUs a virtual machine requires.

6.1.2.3. Storage overhead

Use the guidelines below to estimate storage overhead requirements for your OpenShift Virtualization environment.

Cluster storage overhead

Aggregated storage overhead per node ≈ 10 GiB

10 GiB is the estimated on-disk storage impact for each node in the cluster when you install OpenShift Virtualization.

Virtual machine storage overhead

Storage overhead per virtual machine depends on specific requests for resource allocation within the virtual machine. The request could be for ephemeral storage on the node or storage resources hosted elsewhere in the cluster. OpenShift Virtualization does not currently allocate any additional ephemeral storage for the running container itself.

6.1.2.4. Example

As a cluster administrator, if you plan to host 10 virtual machines in the cluster, each with 1 GiB of RAM and 2 vCPUs, the memory impact across the cluster is 11.68 GiB. The estimated on-disk storage impact for each node in the cluster is 10 GiB and the CPU impact for worker nodes that host virtual machine workloads is a minimum of 2 cores.

6.1.3. About storage volumes for virtual machine disks

If you use the storage API with known storage providers, volume and access modes are selected automatically. However, if you use a storage class that does not have a storage profile, you must select the volume and access mode.

For best results, use accessMode: ReadWriteMany and volumeMode: Block. This is important for the following reasons:

  • The ReadWriteMany (RWX) access mode is required for live migration.
  • The Block volume mode performs significantly better in comparison to the Filesystem volume mode. This is because the Filesystem volume mode uses more storage layers, including a file system layer and a disk image file. These layers are not necessary for VM disk storage.

    For example, if you use Red Hat OpenShift Data Foundation, Ceph RBD volumes are preferable to CephFS volumes.

    Important

    You cannot live migrate virtual machines that use:

    • A storage volume with ReadWriteOnce (RWO) access mode
    • Passthrough features such as GPUs

    Do not set the evictionStrategy field to LiveMigrate for these virtual machines.

6.1.4. Object maximums

You must consider the following tested object maximums when planning your cluster:

6.1.5. Restricted network environments

If you install OpenShift Virtualization in a restricted environment with no internet connectivity, you must configure Operator Lifecycle Manager for restricted networks.

If you have limited internet connectivity, you can configure proxy support in Operator Lifecycle Manager to access the Red Hat-provided OperatorHub.

6.1.6. Live migration

Live migration has the following requirements:

  • Shared storage with ReadWriteMany (RWX) access mode.
  • Sufficient RAM and network bandwidth.
  • If the virtual machine uses a host model CPU, the nodes must support the virtual machine’s host model CPU.
Note

You must ensure that there is enough memory request capacity in the cluster to support node drains that result in live migrations. You can determine the approximate required spare memory by using the following calculation:

Product of (Maximum number of nodes that can drain in parallel) and (Highest total VM memory request allocations across nodes)

The default number of migrations that can run in parallel in the cluster is 5.

6.1.7. Cluster high-availability options

You can configure one of the following high-availability (HA) options for your cluster:

  • Automatic high availability for installer-provisioned infrastructure (IPI) is available by deploying machine health checks.

    Note

    In OpenShift Container Platform clusters installed using installer-provisioned infrastructure and with MachineHealthCheck properly configured, if a node fails the MachineHealthCheck and becomes unavailable to the cluster, it is recycled. What happens next with VMs that ran on the failed node depends on a series of conditions. See About RunStrategies for virtual machines for more detailed information about the potential outcomes and how RunStrategies affect those outcomes.

  • Automatic high availability for both IPI and non-IPI is available by using the Node Health Check Operator on the OpenShift Container Platform cluster to deploy the NodeHealthCheck controller. The controller identifies unhealthy nodes and uses a remediation provider, such as the Self Node Remediation Operator or Fence Agents Remediation Operator, to remediate the unhealthy nodes. For more information on remediation, fencing, and maintaining nodes, see the Workload Availability for Red Hat OpenShift documentation.
  • High availability for any platform is available by using either a monitoring system or a qualified human to monitor node availability. When a node is lost, shut it down and run oc delete node <lost_node>.

    Note

    Without an external monitoring system or a qualified human monitoring node health, virtual machines lose high availability.

6.2. Specifying nodes for OpenShift Virtualization components

Specify the nodes where you want to deploy OpenShift Virtualization Operators, workloads, and controllers by configuring node placement rules.

Note

You can configure node placement for some components after installing OpenShift Virtualization, but there must not be virtual machines present if you want to configure node placement for workloads.

6.2.1. About node placement for virtualization components

You might want to customize where OpenShift Virtualization deploys its components to ensure that:

  • Virtual machines only deploy on nodes that are intended for virtualization workloads.
  • Operators only deploy on infrastructure nodes.
  • Certain nodes are unaffected by OpenShift Virtualization. For example, you have workloads unrelated to virtualization running on your cluster, and you want those workloads to be isolated from OpenShift Virtualization.
6.2.1.1. How to apply node placement rules to virtualization components

You can specify node placement rules for a component by editing the corresponding object directly or by using the web console.

  • For the OpenShift Virtualization Operators that Operator Lifecycle Manager (OLM) deploys, edit the OLM Subscription object directly. Currently, you cannot configure node placement rules for the Subscription object by using the web console.
  • For components that the OpenShift Virtualization Operators deploy, edit the HyperConverged object directly or configure it by using the web console during OpenShift Virtualization installation.
  • For the hostpath provisioner, edit the HostPathProvisioner object directly or configure it by using the web console.

    Warning

    You must schedule the hostpath provisioner and the virtualization components on the same nodes. Otherwise, virtualization pods that use the hostpath provisioner cannot run.

Depending on the object, you can use one or more of the following rule types:

nodeSelector
Allows pods to be scheduled on nodes that are labeled with the key-value pair or pairs that you specify in this field. The node must have labels that exactly match all listed pairs.
affinity
Enables you to use more expressive syntax to set rules that match nodes with pods. Affinity also allows for more nuance in how the rules are applied. For example, you can specify that a rule is a preference, rather than a hard requirement, so that pods are still scheduled if the rule is not satisfied.
tolerations
Allows pods to be scheduled on nodes that have matching taints. If a taint is applied to a node, that node only accepts pods that tolerate the taint.
6.2.1.2. Node placement in the OLM Subscription object

To specify the nodes where OLM deploys the OpenShift Virtualization Operators, edit the Subscription object during OpenShift Virtualization installation. You can include node placement rules in the spec.config field, as shown in the following example:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.13.11
  channel: "stable"
  config: 1
1
The config field supports nodeSelector and tolerations, but it does not support affinity.
6.2.1.3. Node placement in the HyperConverged object

To specify the nodes where OpenShift Virtualization deploys its components, you can include the nodePlacement object in the HyperConverged Cluster custom resource (CR) file that you create during OpenShift Virtualization installation. You can include nodePlacement under the spec.infra and spec.workloads fields, as shown in the following example:

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement: 1
    ...
  workloads:
    nodePlacement:
    ...
1
The nodePlacement fields support nodeSelector, affinity, and tolerations fields.
6.2.1.4. Node placement in the HostPathProvisioner object

You can configure node placement rules in the spec.workload field of the HostPathProvisioner object that you create when you install the hostpath provisioner.

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  pathConfig:
    path: "</path/to/backing/directory>"
    useNamingPrefix: false
  workload: 1
1
The workload field supports nodeSelector, affinity, and tolerations fields.
6.2.1.5. Additional resources

6.2.2. Example manifests

The following example YAML files use nodePlacement, affinity, and tolerations objects to customize node placement for OpenShift Virtualization components.

6.2.2.1. Operator Lifecycle Manager Subscription object
6.2.2.1.1. Example: Node placement with nodeSelector in the OLM Subscription object

In this example, nodeSelector is configured so that OLM places the OpenShift Virtualization Operators on nodes that are labeled with example.io/example-infra-key = example-infra-value.

apiVersion: operators.coreos.com/v1beta1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.13.11
  channel: "stable"
  config:
    nodeSelector:
      example.io/example-infra-key: example-infra-value
6.2.2.1.2. Example: Node placement with tolerations in the OLM Subscription object

In this example, nodes that are reserved for OLM to deploy OpenShift Virtualization Operators are labeled with the key=virtualization:NoSchedule taint. Only pods with the matching tolerations are scheduled to these nodes.

apiVersion: operators.coreos.com/v1beta1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source:  redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.13.11
  channel: "stable"
  config:
    tolerations:
    - key: "key"
      operator: "Equal"
      value: "virtualization"
      effect: "NoSchedule"
6.2.2.2. HyperConverged object
6.2.2.2.1. Example: Node placement with nodeSelector in the HyperConverged Cluster CR

In this example, nodeSelector is configured so that infrastructure resources are placed on nodes that are labeled with example.io/example-infra-key = example-infra-value and workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value.

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      nodeSelector:
        example.io/example-infra-key: example-infra-value
  workloads:
    nodePlacement:
      nodeSelector:
        example.io/example-workloads-key: example-workloads-value
6.2.2.2.2. Example: Node placement with affinity in the HyperConverged Cluster CR

In this example, affinity is configured so that infrastructure resources are placed on nodes that are labeled with example.io/example-infra-key = example-value and workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value. Nodes that have more than eight CPUs are preferred for workloads, but if they are not available, pods are still scheduled.

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  infra:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-infra-key
                operator: In
                values:
                - example-infra-value
  workloads:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: example.io/example-workloads-key
                operator: In
                values:
                - example-workloads-value
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: example.io/num-cpus
                operator: Gt
                values:
                - 8
6.2.2.2.3. Example: Node placement with tolerations in the HyperConverged Cluster CR

In this example, nodes that are reserved for OpenShift Virtualization components are labeled with the key=virtualization:NoSchedule taint. Only pods with the matching tolerations are scheduled to these nodes.

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  workloads:
    nodePlacement:
      tolerations:
      - key: "key"
        operator: "Equal"
        value: "virtualization"
        effect: "NoSchedule"
6.2.2.3. HostPathProvisioner object
6.2.2.3.1. Example: Node placement with nodeSelector in the HostPathProvisioner object

In this example, nodeSelector is configured so that workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value.

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  pathConfig:
    path: "</path/to/backing/directory>"
    useNamingPrefix: false
  workload:
    nodeSelector:
      example.io/example-workloads-key: example-workloads-value

6.3. Installing OpenShift Virtualization using the web console

Install OpenShift Virtualization to add virtualization functionality to your OpenShift Container Platform cluster.

You can use the OpenShift Container Platform 4.13 web console to subscribe to and deploy the OpenShift Virtualization Operators.

6.3.1. Installing the OpenShift Virtualization Operator

You can install the OpenShift Virtualization Operator from the OpenShift Container Platform web console.

Prerequisites

  • Install OpenShift Container Platform 4.13 on your cluster.
  • Log in to the OpenShift Container Platform web console as a user with cluster-admin permissions.

Procedure

  1. From the Administrator perspective, click OperatorsOperatorHub.
  2. In the Filter by keyword field, type Virtualization.
  3. Select the OpenShift Virtualization Operator tile with the Red Hat source label.
  4. Read the information about the Operator and click Install.
  5. On the Install Operator page:

    1. Select stable from the list of available Update Channel options. This ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
    2. For Installed Namespace, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory openshift-cnv namespace, which is automatically created if it does not exist.

      Warning

      Attempting to install the OpenShift Virtualization Operator in a namespace other than openshift-cnv causes the installation to fail.

    3. For Approval Strategy, it is highly recommended that you select Automatic, which is the default value, so that OpenShift Virtualization automatically updates when a new version is available in the stable update channel.

      While it is possible to select the Manual approval strategy, this is inadvisable because of the high risk that it presents to the supportability and functionality of your cluster. Only select Manual if you fully understand these risks and cannot use Automatic.

      Warning

      Because OpenShift Virtualization is only supported when used with the corresponding OpenShift Container Platform version, missing OpenShift Virtualization updates can cause your cluster to become unsupported.

  6. Click Install to make the Operator available to the openshift-cnv namespace.
  7. When the Operator installs successfully, click Create HyperConverged.
  8. Optional: Configure Infra and Workloads node placement options for OpenShift Virtualization components.
  9. Click Create to launch OpenShift Virtualization.

Verification

  • Navigate to the WorkloadsPods page and monitor the OpenShift Virtualization pods until they are all Running. After all the pods display the Running state, you can use OpenShift Virtualization.

6.3.2. Next steps

You might want to additionally configure the following components:

  • The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first.

6.4. Installing OpenShift Virtualization using the CLI

Install OpenShift Virtualization to add virtualization functionality to your OpenShift Container Platform cluster. You can subscribe to and deploy the OpenShift Virtualization Operators by using the command line to apply manifests to your cluster.

Note

To specify the nodes where you want OpenShift Virtualization to install its components, configure node placement rules.

6.4.1. Prerequisites

  • Install OpenShift Container Platform 4.13 on your cluster.
  • Install the OpenShift CLI (oc).
  • Log in as a user with cluster-admin privileges.

6.4.2. Subscribing to the OpenShift Virtualization catalog by using the CLI

Before you install OpenShift Virtualization, you must subscribe to the OpenShift Virtualization catalog. Subscribing gives the openshift-cnv namespace access to the OpenShift Virtualization Operators.

To subscribe, configure Namespace, OperatorGroup, and Subscription objects by applying a single manifest to your cluster.

Procedure

  1. Create a YAML file that contains the following manifest:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: openshift-cnv
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: kubevirt-hyperconverged-group
      namespace: openshift-cnv
    spec:
      targetNamespaces:
        - openshift-cnv
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: hco-operatorhub
      namespace: openshift-cnv
    spec:
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      name: kubevirt-hyperconverged
      startingCSV: kubevirt-hyperconverged-operator.v4.13.11
      channel: "stable" 1
    1
    Using the stable channel ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
  2. Create the required Namespace, OperatorGroup, and Subscription objects for OpenShift Virtualization by running the following command:

    $ oc apply -f <file name>.yaml
Note

You can configure certificate rotation parameters in the YAML file.

6.4.3. Deploying the OpenShift Virtualization Operator by using the CLI

You can deploy the OpenShift Virtualization Operator by using the oc CLI.

Prerequisites

  • An active subscription to the OpenShift Virtualization catalog in the openshift-cnv namespace.

Procedure

  1. Create a YAML file that contains the following manifest:

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
      namespace: openshift-cnv
    spec:
  2. Deploy the OpenShift Virtualization Operator by running the following command:

    $ oc apply -f <file_name>.yaml

Verification

  • Ensure that OpenShift Virtualization deployed successfully by watching the PHASE of the cluster service version (CSV) in the openshift-cnv namespace. Run the following command:

    $ watch oc get csv -n openshift-cnv

    The following output displays if deployment was successful:

    Example output

    NAME                                      DISPLAY                    VERSION   REPLACES   PHASE
    kubevirt-hyperconverged-operator.v4.13.11   OpenShift Virtualization   4.13.11                Succeeded

6.4.4. Next steps

You might want to additionally configure the following components:

  • The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first.

6.5. Uninstalling OpenShift Virtualization

You uninstall OpenShift Virtualization by using the web console or the command line interface (CLI) to delete the OpenShift Virtualization workloads, the Operator, and its resources.

6.5.1. Uninstalling OpenShift Virtualization by using the web console

You uninstall OpenShift Virtualization by using the web console to perform the following tasks:

Important

You must first delete all virtual machines, and virtual machine instances.

You cannot uninstall OpenShift Virtualization while its workloads remain on the cluster.

6.5.1.1. Deleting the HyperConverged custom resource

To uninstall OpenShift Virtualization, you first delete the HyperConverged custom resource (CR).

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to the OperatorsInstalled Operators page.
  2. Select the OpenShift Virtualization Operator.
  3. Click the OpenShift Virtualization Deployment tab.
  4. Click the Options menu kebab beside kubevirt-hyperconverged and select Delete HyperConverged.
  5. Click Delete in the confirmation window.
6.5.1.2. Deleting Operators from a cluster using the web console

Cluster administrators can delete installed Operators from a selected namespace by using the web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster web console using an account with cluster-admin permissions.

Procedure

  1. Navigate to the OperatorsInstalled Operators page.
  2. Scroll or enter a keyword into the Filter by name field to find the Operator that you want to remove. Then, click on it.
  3. On the right side of the Operator Details page, select Uninstall Operator from the Actions list.

    An Uninstall Operator? dialog box is displayed.

  4. Select Uninstall to remove the Operator, Operator deployments, and pods. Following this action, the Operator stops running and no longer receives updates.

    Note

    This action does not remove resources managed by the Operator, including custom resource definitions (CRDs) and custom resources (CRs). Dashboards and navigation items enabled by the web console and off-cluster resources that continue to run might need manual clean up. To remove these after uninstalling the Operator, you might need to manually delete the Operator CRDs.

6.5.1.3. Deleting a namespace using the web console

You can delete a namespace by using the OpenShift Container Platform web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to AdministrationNamespaces.
  2. Locate the namespace that you want to delete in the list of namespaces.
  3. On the far right side of the namespace listing, select Delete Namespace from the Options menu kebab .
  4. When the Delete Namespace pane opens, enter the name of the namespace that you want to delete in the field.
  5. Click Delete.
6.5.1.4. Deleting OpenShift Virtualization custom resource definitions

You can delete the OpenShift Virtualization custom resource definitions (CRDs) by using the web console.

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.

Procedure

  1. Navigate to AdministrationCustomResourceDefinitions.
  2. Select the Label filter and enter operators.coreos.com/kubevirt-hyperconverged.openshift-cnv in the Search field to display the OpenShift Virtualization CRDs.
  3. Click the Options menu kebab beside each CRD and select Delete CustomResourceDefinition.

6.5.2. Uninstalling OpenShift Virtualization by using the CLI

You can uninstall OpenShift Virtualization by using the OpenShift CLI (oc).

Prerequisites

  • You have access to an OpenShift Container Platform cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have deleted all virtual machines and virtual machine instances. You cannot uninstall OpenShift Virtualization while its workloads remain on the cluster.

Procedure

  1. Delete the HyperConverged custom resource:

    $ oc delete HyperConverged kubevirt-hyperconverged -n openshift-cnv
  2. Delete the OpenShift Virtualization Operator subscription:

    $ oc delete subscription kubevirt-hyperconverged -n openshift-cnv
  3. Delete the OpenShift Virtualization ClusterServiceVersion resource:

    $ oc delete csv -n openshift-cnv -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv
  4. Delete the OpenShift Virtualization namespace:

    $ oc delete namespace openshift-cnv
  5. List the OpenShift Virtualization custom resource definitions (CRDs) by running the oc delete crd command with the dry-run option:

    $ oc delete crd --dry-run=client -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv

    Example output

    customresourcedefinition.apiextensions.k8s.io "cdis.cdi.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "hostpathprovisioners.hostpathprovisioner.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "hyperconvergeds.hco.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "kubevirts.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "ssps.ssp.kubevirt.io" deleted (dry run)
    customresourcedefinition.apiextensions.k8s.io "tektontasks.tektontasks.kubevirt.io" deleted (dry run)

  6. Delete the CRDs by running the oc delete crd command without the dry-run option:

    $ oc delete crd -l operators.coreos.com/kubevirt-hyperconverged.openshift-cnv

Chapter 7. Updating OpenShift Virtualization

Learn how Operator Lifecycle Manager (OLM) delivers z-stream and minor version updates for OpenShift Virtualization.

Important

Updating to OpenShift Virtualization 4.13 from OpenShift Virtualization 4.12.2 is not supported.

7.1. OpenShift Virtualization on RHEL 9

OpenShift Virtualization 4.13 is based on Red Hat Enterprise Linux (RHEL) 9. You can update to OpenShift Virtualization 4.13 from a version that was based on RHEL 8 by following the standard OpenShift Virtualization update procedure. No additional steps are required.

As in previous versions, you can perform the update without disrupting running workloads. OpenShift Virtualization 4.13 supports live migration from RHEL 8 nodes to RHEL 9 nodes.

7.1.1. New RHEL 9 machine type

This update also introduces a new RHEL 9 machine type for VMs: machineType: pc-q35-rhel9.2.0. All VM templates that are included with OpenShift Virtualization now use this machine type by default.

Updating OpenShift Virtualization does not change the machineType value of any existing VMs. These VMs continue to function as they did before the update.

While it is not required, you might want to change a VM’s machine type to pc-q35-rhel9.2.0 so that it can benefit from RHEL 9 improvements.

Important

Before you change a VM’s machineType value, you must shut down the VM.

7.2. About updating OpenShift Virtualization

  • Operator Lifecycle Manager (OLM) manages the lifecycle of the OpenShift Virtualization Operator. The Marketplace Operator, which is deployed during OpenShift Container Platform installation, makes external Operators available to your cluster.
  • OLM provides z-stream and minor version updates for OpenShift Virtualization. Minor version updates become available when you update OpenShift Container Platform to the next minor version. You cannot update OpenShift Virtualization to the next minor version without first updating OpenShift Container Platform.
  • OpenShift Virtualization subscriptions use a single update channel that is named stable. The stable channel ensures that your OpenShift Virtualization and OpenShift Container Platform versions are compatible.
  • If your subscription’s approval strategy is set to Automatic, the update process starts as soon as a new version of the Operator is available in the stable channel. It is highly recommended to use the Automatic approval strategy to maintain a supportable environment. Each minor version of OpenShift Virtualization is only supported if you run the corresponding OpenShift Container Platform version. For example, you must run OpenShift Virtualization 4.13 on OpenShift Container Platform 4.13.

    • Though it is possible to select the Manual approval strategy, this is not recommended because it risks the supportability and functionality of your cluster. With the Manual approval strategy, you must manually approve every pending update. If OpenShift Container Platform and OpenShift Virtualization updates are out of sync, your cluster becomes unsupported.
  • The amount of time an update takes to complete depends on your network connection. Most automatic updates complete within fifteen minutes.
  • Updating OpenShift Virtualization does not interrupt network connections.
  • Data volumes and their associated persistent volume claims are preserved during update.
Important

If you have virtual machines running that use hostpath provisioner storage, they cannot be live migrated and might block an OpenShift Container Platform cluster update.

As a workaround, you can reconfigure the virtual machines so that they can be powered off automatically during a cluster update. Remove the evictionStrategy: LiveMigrate field and set the runStrategy field to Always.

7.2.1. About workload updates

When you update OpenShift Virtualization, virtual machine workloads, including libvirt, virt-launcher, and qemu, update automatically if they support live migration.

Note

Each virtual machine has a virt-launcher pod that runs the virtual machine instance (VMI). The virt-launcher pod runs an instance of libvirt, which is used to manage the virtual machine (VM) process.

You can configure how workloads are updated by editing the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource (CR). There are two available workload update methods: LiveMigrate and Evict.

Because the Evict method shuts down VMI pods, only the LiveMigrate update strategy is enabled by default.

When LiveMigrate is the only update strategy enabled:

  • VMIs that support live migration are migrated during the update process. The VM guest moves into a new pod with the updated components enabled.
  • VMIs that do not support live migration are not disrupted or updated.

    • If a VMI has the LiveMigrate eviction strategy but does not support live migration, it is not updated.

If you enable both LiveMigrate and Evict:

  • VMIs that support live migration use the LiveMigrate update strategy.
  • VMIs that do not support live migration use the Evict update strategy. If a VMI is controlled by a VirtualMachine object that has a runStrategy value of always, a new VMI is created in a new pod with updated components.
Migration attempts and timeouts

When updating workloads, live migration fails if a pod is in the Pending state for the following periods:

5 minutes
If the pod is pending because it is Unschedulable.
15 minutes
If the pod is stuck in the pending state for any reason.

When a VMI fails to migrate, the virt-controller tries to migrate it again. It repeats this process until all migratable VMIs are running on new virt-launcher pods. If a VMI is improperly configured, however, these attempts can repeat indefinitely.

Note

Each attempt corresponds to a migration object. Only the five most recent attempts are held in a buffer. This prevents migration objects from accumulating on the system while retaining information for debugging.

7.2.2. About EUS-to-EUS updates

Every even-numbered minor version of OpenShift Container Platform, including 4.10 and 4.12, is an Extended Update Support (EUS) version. However, because Kubernetes design mandates serial minor version updates, you cannot directly update from one EUS version to the next.

After you update from the source EUS version to the next odd-numbered minor version, you must sequentially update OpenShift Virtualization to all z-stream releases of that minor version that are on your update path. When you have upgraded to the latest applicable z-stream version, you can then update OpenShift Container Platform to the target EUS minor version.

When the OpenShift Container Platform update succeeds, the corresponding update for OpenShift Virtualization becomes available. You can now update OpenShift Virtualization to the target EUS version.

7.2.2.1. Preparing to update

Before beginning an EUS-to-EUS update, you must:

  • Pause worker nodes' machine config pools before you start an EUS-to-EUS update so that the workers are not rebooted twice.
  • Disable automatic workload updates before you begin the update process. This is to prevent OpenShift Virtualization from migrating or evicting your virtual machines (VMs) until you update to your target EUS version.
Note

By default, OpenShift Virtualization automatically updates workloads, such as the virt-launcher pod, when you update the OpenShift Virtualization Operator. You can configure this behavior in the spec.workloadUpdateStrategy stanza of the HyperConverged custom resource.

Learn more about preparing to perform an EUS-to-EUS update.

7.3. Preventing workload updates during an EUS-to-EUS update

When you update from one Extended Update Support (EUS) version to the next, you must manually disable automatic workload updates to prevent OpenShift Virtualization from migrating or evicting workloads during the update process.

Prerequisites

  • You are running an EUS version of OpenShift Container Platform and want to update to the next EUS version. You have not yet updated to the odd-numbered version in between.
  • You read "Preparing to perform an EUS-to-EUS update" and learned the caveats and requirements that pertain to your OpenShift Container Platform cluster.
  • You paused the worker nodes' machine config pools as directed by the OpenShift Container Platform documentation.
  • It is recommended that you use the default Automatic approval strategy. If you use the Manual approval strategy, you must approve all pending updates in the web console. For more details, refer to the "Manually approving a pending Operator update" section.

Procedure

  1. Back up the current workloadUpdateMethods configuration by running the following command:

    $ WORKLOAD_UPDATE_METHODS=$(oc get kv kubevirt-kubevirt-hyperconverged -n openshift-cnv -o jsonpath='{.spec.workloadUpdateStrategy.workloadUpdateMethods}')
  2. Turn off all workload update methods by running the following command:

    $ oc patch hco kubevirt-hyperconverged -n openshift-cnv --type json -p '[{"op":"replace","path":"/spec/workloadUpdateStrategy/workloadUpdateMethods", "value":[]}]'

    Example output

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

  3. Ensure that the HyperConverged Operator is Upgradeable before you continue. Enter the following command and monitor the output:

    $ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"

    Example 7.1. Example output

    [
      {
        "lastTransitionTime": "2022-12-09T16:29:11Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "ReconcileComplete"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "Available"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "False",
        "type": "Progressing"
      },
      {
        "lastTransitionTime": "2022-12-09T16:39:11Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "False",
        "type": "Degraded"
      },
      {
        "lastTransitionTime": "2022-12-09T20:30:10Z",
        "message": "Reconcile completed successfully",
        "observedGeneration": 3,
        "reason": "ReconcileCompleted",
        "status": "True",
        "type": "Upgradeable" 1
      }
    ]
    1
    The OpenShift Virtualization Operator has the Upgradeable status.
  4. Manually update your cluster from the source EUS version to the next minor version of OpenShift Container Platform:

    $ oc adm upgrade

    Verification

    • Check the current version by running the following command:

      $ oc get clusterversion
      Note

      Updating OpenShift Container Platform to the next version is a prerequisite for updating OpenShift Virtualization. For more details, refer to the "Updating clusters" section of the OpenShift Container Platform documentation.

  5. Update OpenShift Virtualization.

    • With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform.
    • If you use the Manual approval strategy, approve the pending updates by using the web console.
  6. Monitor the OpenShift Virtualization update by running the following command:

    $ oc get csv -n openshift-cnv
  7. Update OpenShift Virtualization to every z-stream version that is available for the non-EUS minor version, monitoring each update by running the command shown in the previous step.
  8. Confirm that OpenShift Virtualization successfully updated to the latest z-stream release of the non-EUS version by running the following command:

    $ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"

    Example output

    [
      {
        "name": "operator",
        "version": "4.13.11"
      }
    ]

  9. Wait until the HyperConverged Operator has the Upgradeable status before you perform the next update. Enter the following command and monitor the output:

    $ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
  10. Update OpenShift Container Platform to the target EUS version.
  11. Confirm that the update succeeded by checking the cluster version:

    $ oc get clusterversion
  12. Update OpenShift Virtualization to the target EUS version.

    • With the default Automatic approval strategy, OpenShift Virtualization automatically updates to the corresponding version after you update OpenShift Container Platform.
    • If you use the Manual approval strategy, approve the pending updates by using the web console.
  13. Monitor the OpenShift Virtualization update by running the following command:

    $ oc get csv -n openshift-cnv

    The update completes when the VERSION field matches the target EUS version and the PHASE field reads Succeeded.

  14. Restore the workload update methods configuration that you backed up:

    $ oc patch hco kubevirt-hyperconverged -n openshift-cnv --type json -p "[{\"op\":\"add\",\"path\":\"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\":$WORKLOAD_UPDATE_METHODS}]"

    Example output

    hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

    Verification

    • Check the status of VM migration by running the following command:

      $ oc get vmim -A

Next steps

  • You can now unpause the worker nodes' machine config pools.

7.4. Configuring workload update methods

You can configure workload update methods by editing the HyperConverged custom resource (CR).

Prerequisites

  • To use live migration as an update method, you must first enable live migration in the cluster.

    Note

    If a VirtualMachineInstance CR contains evictionStrategy: LiveMigrate and the virtual machine instance (VMI) does not support live migration, the VMI will not update.

Procedure

  1. To open the HyperConverged CR in your default editor, run the following command:

    $ oc edit hco -n openshift-cnv kubevirt-hyperconverged
  2. Edit the workloadUpdateStrategy stanza of the HyperConverged CR. For example:

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
    spec:
      workloadUpdateStrategy:
        workloadUpdateMethods: 1
        - LiveMigrate 2
        - Evict 3
        batchEvictionSize: 10 4
        batchEvictionInterval: "1m0s" 5
    # ...
    1
    The methods that can be used to perform automated workload updates. The available values are LiveMigrate and Evict. If you enable both options as shown in this example, updates use LiveMigrate for VMIs that support live migration and Evict for any VMIs that do not support live migration. To disable automatic workload updates, you can either remove the workloadUpdateStrategy stanza or set workloadUpdateMethods: [] to leave the array empty.
    2
    The least disruptive update method. VMIs that support live migration are updated by migrating the virtual machine (VM) guest into a new pod with the updated components enabled. If LiveMigrate is the only workload update method listed, VMIs that do not support live migration are not disrupted or updated.
    3
    A disruptive method that shuts down VMI pods during upgrade. Evict is the only update method available if live migration is not enabled in the cluster. If a VMI is controlled by a VirtualMachine object that has runStrategy: always configured, a new VMI is created in a new pod with updated components.
    4
    The number of VMIs that can be forced to be updated at a time by using the Evict method. This does not apply to the LiveMigrate method.
    5
    The interval to wait before evicting the next batch of workloads. This does not apply to the LiveMigrate method.
    Note

    You can configure live migration limits and timeouts by editing the spec.liveMigrationConfig stanza of the HyperConverged CR.

  3. To apply your changes, save and exit the editor.

7.5. Approving pending Operator updates

7.5.1. Manually approving a pending Operator update

If an installed Operator has the approval strategy in its subscription set to Manual, when new updates are released in its current update channel, the update must be manually approved before installation can begin.

Prerequisites

  • An Operator previously installed using Operator Lifecycle Manager (OLM).

Procedure

  1. In the Administrator perspective of the OpenShift Container Platform web console, navigate to Operators → Installed Operators.
  2. Operators that have a pending update display a status with Upgrade available. Click the name of the Operator you want to update.
  3. Click the Subscription tab. Any updates requiring approval are displayed next to Upgrade status. For example, it might display 1 requires approval.
  4. Click 1 requires approval, then click Preview Install Plan.
  5. Review the resources that are listed as available for update. When satisfied, click Approve.
  6. Navigate back to the Operators → Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date.

7.6. Monitoring update status

7.6.1. Monitoring OpenShift Virtualization upgrade status

To monitor the status of a OpenShift Virtualization Operator upgrade, watch the cluster service version (CSV) PHASE. You can also monitor the CSV conditions in the web console or by running the command provided here.

Note

The PHASE and conditions values are approximations that are based on available information.

Prerequisites

  • Log in to the cluster as a user with the cluster-admin role.
  • Install the OpenShift CLI (oc).

Procedure

  1. Run the following command:

    $ oc get csv -n openshift-cnv
  2. Review the output, checking the PHASE field. For example:

    Example output

    VERSION  REPLACES                                        PHASE
    4.9.0    kubevirt-hyperconverged-operator.v4.8.2         Installing
    4.9.0    kubevirt-hyperconverged-operator.v4.9.0         Replacing

  3. Optional: Monitor the aggregated status of all OpenShift Virtualization component conditions by running the following command:

    $ oc get hco -n openshift-cnv kubevirt-hyperconverged \
    -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

    A successful upgrade results in the following output:

    Example output

    ReconcileComplete  True  Reconcile completed successfully
    Available          True  Reconcile completed successfully
    Progressing        False Reconcile completed successfully
    Degraded           False Reconcile completed successfully
    Upgradeable        True  Reconcile completed successfully

7.6.2. Viewing outdated OpenShift Virtualization workloads

You can view a list of outdated workloads by using the CLI.

Note

If there are outdated virtualization pods in your cluster, the OutdatedVirtualMachineInstanceWorkloads alert fires.

Procedure

  • To view a list of outdated virtual machine instances (VMIs), run the following command:

    $ oc get vmi -l kubevirt.io/outdatedLauncherImage --all-namespaces
Note

Configure workload updates to ensure that VMIs update automatically.

7.7. Additional resources

Chapter 8. Security policies

Learn about OpenShift Virtualization security and authorization.

Key points

  • OpenShift Virtualization adheres to the restricted Kubernetes pod security standards profile, which aims to enforce the current best practices for pod security.
  • Virtual machine (VM) workloads run as unprivileged pods.
  • Security context constraints (SCCs) are defined for the kubevirt-controller service account.

8.1. About workload security

By default, virtual machine (VM) workloads do not run with root privileges in OpenShift Virtualization, and there are no supported OpenShift Virtualization features that require root privileges.

For each VM, a virt-launcher pod runs an instance of libvirt in session mode to manage the VM process. In session mode, the libvirt daemon runs as a non-root user account and only permits connections from clients that are running under the same user identifier (UID). Therefore, VMs run as unprivileged pods, adhering to the security principle of least privilege.

8.2. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account

Security context constraints (SCCs) control permissions for pods. These permissions include actions that a pod, a collection of containers, can perform and what resources it can access. You can use SCCs to define a set of conditions that a pod must run with to be accepted into the system.

The virt-controller is a cluster controller that creates the virt-launcher pods for virtual machines in the cluster. These pods are granted permissions by the kubevirt-controller service account.

The kubevirt-controller service account is granted additional SCCs and Linux capabilities so that it can create virt-launcher pods with the appropriate permissions. These extended permissions allow virtual machines to use OpenShift Virtualization features that are beyond the scope of typical pods.

The kubevirt-controller service account is granted the following SCCs:

  • scc.AllowHostDirVolumePlugin = true
    This allows virtual machines to use the hostpath volume plugin.
  • scc.AllowPrivilegedContainer = false
    This ensures the virt-launcher pod is not run as a privileged container.
  • scc.AllowedCapabilities = []corev1.Capability{"SYS_NICE", "NET_BIND_SERVICE"}

    • SYS_NICE allows setting the CPU affinity.
    • NET_BIND_SERVICE allows DHCP and Slirp operations.

8.2.1. Viewing the SCC and RBAC definitions for the kubevirt-controller

You can view the SecurityContextConstraints definition for the kubevirt-controller by using the oc tool:

$ oc get scc kubevirt-controller -o yaml

You can view the RBAC definition for the kubevirt-controller clusterrole by using the oc tool:

$ oc get clusterrole kubevirt-controller -o yaml

8.3. Authorization

OpenShift Virtualization uses role-based access control (RBAC) for authorization. For example, an administrator can create an RBAC role that provides the permissions required to launch a virtual machine. The administrator can then restrict access to that feature by binding the role to specific users.

8.3.1. Default cluster roles for OpenShift Virtualization

By using cluster role aggregation, OpenShift Virtualization extends the default OpenShift Container Platform cluster roles to include permissions for accessing virtualization objects.

Table 8.1. OpenShift Virtualization cluster roles
Default cluster roleOpenShift Virtualization cluster roleOpenShift Virtualization cluster role description

view

kubevirt.io:view

A user that can view all OpenShift Virtualization resources in the cluster but cannot create, delete, modify, or access them. For example, the user can see that a virtual machine (VM) is running but cannot shut it down or gain access to its console.

edit

kubevirt.io:edit

A user that can modify all OpenShift Virtualization resources in the cluster. For example, the user can create VMs, access VM consoles, and delete VMs.

admin

kubevirt.io:admin

A user that has full permissions to all OpenShift Virtualization resources, including the ability to delete collections of resources. The user can also view and modify the OpenShift Virtualization runtime configuration, which is located in the HyperConverged custom resource in the openshift-cnv namespace.

8.4. Additional resources

Chapter 9. Using the virtctl and libguestfs CLI tools

You can manage OpenShift Virtualization resources by using the virtctl command line tool.

You can also deploy a libguestfs-tools container by using virtctl. Libguestfs is a set of tools for accessing and modifying virtual machine (VM) disk images.

9.1. Installing virtctl

To install virtctl on Linux, Windows, and MacOS operating systems, you download and install the the virtctl binary file.

To install virtctl on Red Hat Enterprise Linux (RHEL), you enable the OpenShift Virtualization repository and then install the kubevirt-virtctl package.

9.1.1. Installing the virtctl binary on RHEL 9, Linux, Windows, or macOS

You can download the virtctl binary for your operating system from the OpenShift Container Platform web console and then install it.

Procedure

  1. Navigate to the Virtualization → Overview page in the web console.
  2. Click the Download virtctl link to download the virtctl binary for your operating system.
  3. Install virtctl:

    • For RHEL 9 and other Linux operating systems:

      1. Decompress the archive file:

        $ tar -xvf <virtctl-version-distribution.arch>.tar.gz
      2. Run the following command to make the virtctl binary executable:

        $ chmod +x <path/virtctl-file-name>
      3. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        $ echo $PATH
      4. Set the KUBECONFIG environment variable:

        $ export KUBECONFIG=/home/<user>/clusters/current/auth/kubeconfig
    • For Windows:

      1. Decompress the archive file.
      2. Navigate the extracted folder hierarchy and double-click the virtctl executable file to install the client.
      3. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        C:\> path
    • For macOS:

      1. Decompress the archive file.
      2. Move the virtctl binary to a directory in your PATH environment variable.

        You can check your path by running the following command:

        echo $PATH

9.1.2. Installing the virtctl RPM on RHEL 8

You can install the virtctl RPM package on Red Hat Enterprise Linux (RHEL) 8 by enabling the OpenShift Virtualization repository and installing the kubevirt-virtctl package.

Prerequisites

  • Each host in your cluster must be registered with Red Hat Subscription Manager (RHSM) and have an active OpenShift Container Platform subscription.

Procedure

  1. Enable the OpenShift Virtualization repository for your operating system by using the subscription-manager CLI tool to run the following command:

    # subscription-manager repos --enable cnv-4.13-for-rhel-8-x86_64-rpms
  2. Install the kubevirt-virtctl package by running the following command:

    # yum install kubevirt-virtctl

9.2. Virtctl commands

The virtctl client is a command-line utility for managing OpenShift Virtualization resources.

Note

The virtual machine (VM) commands also apply to virtual machine instances unless otherwise specified.

9.2.1. Virtctl information commands

You use virtctl information commands to view information about the virtctl client.

Table 9.1. Information commands
CommandDescription

virtctl version

View the virtctl client and server versions.

virtctl help

View a list of virtctl commands.

virtctl <command> -h|--help

View a list of options for a specific command.

virtctl options

View a list of global command options for any virtctl command.

9.2.2. VM information commands

You can use virtctl to view information about VMs and VMIs.

Table 9.2. VM information commands
CommandDescription

virtctl fslist <vm_name>

View the file systems available on a guest machine.

virtctl guestosinfo <vm_name>

View information about the operating systems on a guest machine.

virtctl userlist <vm_name>

View the logged-in users on a guest machine.

9.2.3. VM management commands

You use virtctl virtual machine (VM) management commands to manage and migrate VMs and VMIs.

Table 9.3. VM management commands
CommandDescription

virtctl create -name <vm_name>

Create a VirtualMachine manifest.

virtctl start <vm_name>

Start a VM.

virtctl start --paused <vm_name>

Start a VM in a paused state. This option enables you to interrupt the boot process from the VNC console.

virtctl stop <vm_name>

Stop a VM.

virtctl stop <vm_name> --grace-period 0 --force

Force stop a VM. This option might cause data inconsistency or data loss.

virtctl pause vm <vm_name>

Pause a VM. The machine state is kept in memory.

virtctl unpause vm <vm_name>

Unpause a VM.

virtctl migrate <vm_name>

Migrate a VM.

virtctl restart <vm_name>

Restart a VM.

9.2.4. VM connection commands

You use virtctl connection commands to expose ports and connect to VMs and VMIs.

Table 9.4. VM connection commands
CommandDescription

virtctl console <vm_name>

Connect to the serial console of a VM.

virtctl expose <vm_name>

Create a service that forwards a designated port of a VM and expose the service on the specified port of the node.

virtctl scp -i <ssh_key> <file_name> <user_name>@<vm_name>

Copy a file from your machine to a VM. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl scp -i <ssh_key> <user_name@<vm_name>:<file_name> .

Copy a file from a VM to your machine. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl ssh -i <ssh_key> <user_name>@<vm_name>

Open an SSH connection with a VM. This command uses the private key of an SSH key pair. The VM must be configured with the public key.

virtctl vnc --kubeconfig=$KUBECONFIG <vm_name>

Connect to the VNC console of a VM.

Accessing the graphical console of a VM through VNC requires a remote viewer on your local machine.

virtctl vnc --kubeconfig=$KUBECONFIG --proxy-only=true <vm_name>

Display the port number and connect manually to a VM by using any viewer through the VNC connection.

virtctl vnc --kubeconfig=$KUBECONFIG --port=<port-number> <vm_name>

Specify a port number to run the proxy on the specified port, if that port is available.

If a port number is not specified, the proxy runs on a random port.

9.2.5. VM export commands

You use virtctl vmexport commands to create, download, or delete a volume exported from a VM, VM snapshot, or persistent volume claim (PVC).

Table 9.5. VM export commands
CommandDescription

virtctl vmexport create <vmexport_name> --vm|snapshot|pvc=<object_name>

Create a VirtualMachineExport custom resource (CR) to export a volume from a VM, VM snapshot, or PVC.

  • --vm: Exports the PVCs of a VM.
  • --snapshot: Exports the PVCs contained in a VirtualMachineSnapshot CR.
  • --pvc: Exports a PVC.
  • Optional: --ttl=1h specifies the time to live. The default duration is 2 hours.

virtctl vmexport delete <vmexport_name>

Delete a VirtualMachineExport CR manually.

virtctl vmexport download <vmexport_name> --output=<output_file> --volume=<volume_name>

Download the volume defined in a VirtualMachineExport CR.

  • --output specifies the file format. Example: disk.img.gz.
  • --volume specifies the volume to download. This flag is optional if only one volume is available.

Optional:

  • --keep-vme retains the VirtualMachineExport CR after download. The default behavior is to delete the VirtualMachineExport CR after download.
  • --insecure enables an insecure HTTP connection.

virtctl vmexport download <vmexport_name> --<vm|snapshot|pvc>=<object_name> --output=<output_file> --volume=<volume_name>

Create a VirtualMachineExport CR and then download the volume defined in the CR.

9.2.6. VM memory dump commands

You can use the virtctl memory-dump command to output a VM memory dump on a PVC. You can specify an existing PVC or use the --create-claim flag to create a new PVC.

Prerequisites

  • The PVC volume mode must be FileSystem.
  • The PVC must be large enough to contain the memory dump.

    The formula for calculating the PVC size is (VMMemorySize + 100Mi) * FileSystemOverhead, where 100Mi is the memory dump overhead.

  • You must enable the hot plug feature gate in the HyperConverged custom resource by running the following command:

    $ oc patch hco kubevirt-hyperconverged -n openshift-cnv \
      --type json -p '[{"op": "add", "path": "/spec/featureGates", \
      "value": "HotplugVolumes"}]'

Downloading the memory dump

You must use the virtctl vmexport download command to download the memory dump:

$ virtctl vmexport download <vmexport_name> --vm\|pvc=<object_name> \
  --volume=<volume_name> --output=<output_file>
Table 9.6. VM memory dump commands
CommandDescription

virtctl memory-dump get <vm_name> --claim-name=<pvc_name>

Save the memory dump of a VM on a PVC. The memory dump status is displayed in the status section of the VirtualMachine resource.

Optional:

  • --create-claim creates a new PVC with the appropriate size. This flag has the following options:

    • --storage-class=<storage_class>: Specify a storage class for the PVC.
    • --access-mode=<access_mode>: Specify ReadWriteOnce or ReadWriteMany.

virtctl memory-dump get <vm_name>

Rerun the virtctl memory-dump command with the same PVC.

This command overwrites the previous memory dump.

virtctl memory-dump remove <vm_name>

Remove a memory dump.

You must remove a memory dump manually if you want to change the target PVC.

This command removes the association between the VM and the PVC, so that the memory dump is not displayed in the status section of the VirtualMachine resource. The PVC is not affected.

9.2.7. Hot plug and hot unplug commands

You use virtctl to add or remove resources from running VMs and VMIs.

Table 9.7. Hot plug and hot unplug commands
CommandDescription

virtctl addvolume <vm_name> --volume-name=<datavolume_or_PVC> [--persist] [--serial=<label>]

Hot plug a data volume or persistent volume claim (PVC).

Optional:

  • --persist mounts the virtual disk permanently on a VM. This flag does not apply to VMIs.
  • --serial=<label> adds a label to the VM. If you do not specify a label, the default label is the data volume or PVC name.

virtctl removevolume <vm_name> --volume-name=<virtual_disk>

Hot unplug a virtual disk.

9.2.8. Image upload commands

You use the virtctl image-upload commands to upload a VM image to a data volume.

Table 9.8. Image upload commands
CommandDescription

virtctl image-upload dv <datavolume_name> --image-path=</path/to/image> --no-create

Upload a VM image to a data volume that already exists.

virtctl image-upload dv <datavolume_name> --size=<datavolume_size> --image-path=</path/to/image>

Upload a VM image to a new data volume of a specified requested size.

9.3. Using libguestfs

9.3.1. Deploying a libguestfs-tools container by using virtctl

You can use the virtctl guestfs command to deploy an interactive container with libguestfs-tools and a persistent volume claim (PVC) attached to it.

Procedure

  • To deploy a container with libguestfs-tools, mount the PVC, and attach a shell to it, run the following command:

    $ virtctl guestfs -n <namespace> <pvc_name> 1
    1
    The PVC name is a required argument. If you do not include it, an error message appears.

9.3.2. Libguestfs and virtctl guestfs commands

Libguestfs tools help you access and modify virtual machine (VM) disk images. You can use libguestfs tools to view and edit files in a guest, clone and build virtual machines, and format and resize disks.

You can also use the virtctl guestfs command and its sub-commands to modify, inspect, and debug VM disks on a PVC. To see a complete list of possible sub-commands, enter virt- on the command line and press the Tab key. For example:

CommandDescription

virt-edit -a /dev/vda /etc/motd

Edit a file interactively in your terminal.

virt-customize -a /dev/vda --ssh-inject root:string:<public key example>

Inject an ssh key into the guest and create a login.

virt-df -a /dev/vda -h

See how much disk space is used by a VM.

virt-customize -a /dev/vda --run-command 'rpm -qa > /rpm-list'

See the full list of all RPMs installed on a guest by creating an output file containing the full list.

virt-cat -a /dev/vda /rpm-list

Display the output file list of all RPMs created using the virt-customize -a /dev/vda --run-command 'rpm -qa > /rpm-list' command in your terminal.

virt-sysprep -a /dev/vda

Seal a virtual machine disk image to be used as a template.

By default, virtctl guestfs creates a session with everything needed to manage a VM disk. However, the command also supports several flag options if you want to customize the behavior:

Flag OptionDescription

--h or --help

Provides help for guestfs.

-n <namespace> option with a <pvc_name> argument

To use a PVC from a specific namespace.

If you do not use the -n <namespace> option, your current project is used. To change projects, use oc project <namespace>.

If you do not include a <pvc_name> argument, an error message appears.

--image string

Lists the libguestfs-tools container image.

You can configure the container to use a custom image by using the --image option.

--kvm

Indicates that kvm is used by the libguestfs-tools container.

By default, virtctl guestfs sets up kvm for the interactive container, which greatly speeds up the libguest-tools execution because it uses QEMU.

If a cluster does not have any kvm supporting nodes, you must disable kvm by setting the option --kvm=false.

If not set, the libguestfs-tools pod remains pending because it cannot be scheduled on any node.

--pull-policy string

Shows the pull policy for the libguestfs image.

You can also overwrite the image’s pull policy by setting the pull-policy option.

The command also checks if a PVC is in use by another pod, in which case an error message appears. However, once the libguestfs-tools process starts, the setup cannot avoid a new pod using the same PVC. You must verify that there are no active virtctl guestfs pods before starting the VM that accesses the same PVC.

Note

The virtctl guestfs command accepts only a single PVC attached to the interactive pod.

Chapter 10. Virtual machines

10.1. Creating virtual machines

Use one of these procedures to create a virtual machine:

  • Quick Start guided tour
  • Quick create from the Catalog
  • Pasting a pre-configured YAML file with the virtual machine wizard
  • Using the CLI
Warning

Do not create virtual machines in openshift-* namespaces. Instead, create a new namespace or use an existing namespace without the openshift prefix.

When you create virtual machines from the web console, select a virtual machine template that is configured with a boot source. Virtual machine templates with a boot source are labeled as Available boot source or they display a customized label text. Using templates with an available boot source expedites the process of creating virtual machines.

Templates without a boot source are labeled as Boot source required. You can use these templates if you complete the steps for adding a boot source to the virtual machine.

Important

Due to differences in storage behavior, some virtual machine templates are incompatible with single-node OpenShift. To ensure compatibility, do not set the evictionStrategy field for any templates or virtual machines that use data volumes or storage profiles.

10.1.1. Using a Quick Start to create a virtual machine

The web console provides Quick Starts with instructional guided tours for creating virtual machines. You can access the Quick Starts catalog by selecting the Help menu in the Administrator perspective to view the Quick Starts catalog. When you click on a Quick Start tile and begin the tour, the system guides you through the process.

Tasks in a Quick Start begin with selecting a Red Hat template. Then, you can add a boot source and import the operating system image. Finally, you can save the custom template and use it to create a virtual machine.

Prerequisites

  • Access to the website where you can download the URL link for the operating system image.

Procedure

  1. In the web console, select Quick Starts from the Help menu.
  2. Click on a tile in the Quick Starts catalog. For example: Creating a Red Hat Linux Enterprise Linux virtual machine.
  3. Follow the instructions in the guided tour and complete the tasks for importing an operating system image and creating a virtual machine. The VirtualizationVirtualMachines page displays the virtual machine.

10.1.2. Quick creating a virtual machine

You can quickly create a virtual machine (VM) by using a template with an available boot source.

Procedure

  1. Click VirtualizationCatalog in the side menu.
  2. Click Boot source available to filter templates with boot sources.

    Note

    By default, the template list will show only Default Templates. Click All Items when filtering to see all available templates for your chosen filters.

  3. Click a template to view its details.
  4. Click Quick Create VirtualMachine to create a VM from the template.

    The virtual machine Details page is displayed with the provisioning status.

Verification

  1. Click Events to view a stream of events as the VM is provisioned.
  2. Click Console to verify that the VM booted successfully.

10.1.3. Creating a virtual machine from a customized template

Some templates require additional parameters, for example, a PVC with a boot source. You can customize select parameters of a template to create a virtual machine (VM).

Procedure

  1. In the web console, select a template:

    1. Click VirtualizationCatalog in the side menu.
    2. Optional: Filter the templates by project, keyword, operating system, or workload profile.
    3. Click the template that you want to customize.
  2. Click Customize VirtualMachine.
  3. Specify parameters for your VM, including its Name and Disk source. You can optionally specify a data source to clone.

Verification

  1. Click Events to view a stream of events as the VM is provisioned.
  2. Click Console to verify that the VM booted successfully.

Refer to the virtual machine fields section when creating a VM from the web console.

10.1.3.1. Networking fields
NameDescription

Name

Name for the network interface controller.

Model

Indicates the model of the network interface controller. Supported values are e1000e and virtio.

Network

List of available network attachment definitions.

Type

List of available binding methods. Select the binding method suitable for the network interface:

  • Default pod network: masquerade
  • Linux bridge network: bridge
  • SR-IOV network: SR-IOV

MAC Address

MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically.

10.1.3.2. Storage fields
NameSelectionDescription

Source

Blank (creates PVC)

Create an empty disk.

Import via URL (creates PVC)

Import content via URL (HTTP or HTTPS endpoint).

Use an existing PVC

Use a PVC that is already available in the cluster.

Clone existing PVC (creates PVC)

Select an existing PVC available in the cluster and clone it.

Import via Registry (creates PVC)

Import content via container registry.

Container (ephemeral)

Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines.

Name

 

Name of the disk. The name can contain lowercase letters (a-z), numbers (0-9), hyphens (-), and periods (.), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters.

Size

 

Size of the disk in GiB.

Type

 

Type of disk. Example: Disk or CD-ROM

Interface

 

Type of disk device. Supported interfaces are virtIO, SATA, and SCSI.

Storage Class

 

The storage class that is used to create the disk.

Advanced storage settings

The following advanced storage settings are optional and available for Blank, Import via URL, and Clone existing PVC disks. Before OpenShift Virtualization 4.11, if you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. In OpenShift Virtualization 4.11 and later, the system uses the default values from the storage profile.

Note

Use storage profiles to ensure consistent advanced storage settings when provisioning storage for OpenShift Virtualization.

To manually specify Volume Mode and Access Mode, you must clear the Apply optimized StorageProfile settings checkbox, which is selected by default.

NameMode descriptionParameterParameter description

Volume Mode

Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem.

Filesystem

Stores the virtual disk on a file system-based volume.

Block

Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it.

Access Mode

Access mode of the persistent volume.

ReadWriteOnce (RWO)

Volume can be mounted as read-write by a single node.

ReadWriteMany (RWX)

Volume can be mounted as read-write by many nodes at one time.

Note

This is required for some features, such as live migration of virtual machines between nodes.

ReadOnlyMany (ROX)

Volume can be mounted as read only by many nodes.

10.1.3.3. Cloud-init fields
NameDescription

Authorized SSH Keys

The user’s public key that is copied to ~/.ssh/authorized_keys on the virtual machine.

Custom script

Replaces other options with a field in which you paste a custom cloud-init script.

To configure storage class defaults, use storage profiles. For more information, see Customizing the storage profile.

10.1.3.4. Pasting in a pre-configured YAML file to create a virtual machine

Create a virtual machine by writing or pasting a YAML configuration file. A valid example virtual machine configuration is provided by default whenever you open the YAML edit screen.

If your YAML configuration is invalid when you click Create, an error message indicates the parameter in which the error occurs. Only one error is shown at a time.

Note

Navigating away from the YAML screen while editing cancels any changes to the configuration you have made.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Click Create and select With YAML.
  3. Write or paste your virtual machine configuration in the editable window.

    1. Alternatively, use the example virtual machine provided by default in the YAML screen.
  4. Optional: Click Download to download the YAML configuration file in its present state.
  5. Click Create to create the virtual machine.

The virtual machine is listed on the VirtualMachines page.

10.1.4. Using the CLI to create a virtual machine

You can create a virtual machine from a virtualMachine manifest.

Procedure

  1. Edit the VirtualMachine manifest for your VM. For example, the following manifest configures a Red Hat Enterprise Linux (RHEL) VM:

    Example 10.1. Example manifest for a RHEL VM

    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      labels:
        app: <vm_name> 1
      name: <vm_name>
    spec:
      dataVolumeTemplates:
      - apiVersion: cdi.kubevirt.io/v1beta1
        kind: DataVolume
        metadata:
          name: <vm_name>
        spec:
          sourceRef:
            kind: DataSource
            name: rhel9
            namespace: openshift-virtualization-os-images
          storage:
            resources:
              requests:
                storage: 30Gi
      running: false
      template:
        metadata:
          labels:
            kubevirt.io/domain: <vm_name>
        spec:
          domain:
            cpu:
              cores: 1
              sockets: 2
              threads: 1
            devices:
              disks:
              - disk:
                  bus: virtio
                name: rootdisk
              - disk:
                  bus: virtio
                name: cloudinitdisk
              interfaces:
              - masquerade: {}
                name: default
              rng: {}
            features:
              smm:
                enabled: true
            firmware:
              bootloader:
                efi: {}
            resources:
              requests:
                memory: 8Gi
          evictionStrategy: LiveMigrate
          networks:
          - name: default
            pod: {}
          volumes:
          - dataVolume:
              name: <vm_name>
            name: rootdisk
          - cloudInitNoCloud:
              userData: |-
                #cloud-config
                user: cloud-user
                password: '<password>' 2
                chpasswd: { expire: False }
            name: cloudinitdisk
    1
    Specify the name of the virtual machine.
    2
    Specify the password for cloud-user.
  2. Create a virtual machine by using the manifest file:

    $ oc create -f <vm_manifest_file>.yaml
  3. Optional: Start the virtual machine:

    $ virtctl start <vm_name>

10.1.5. Virtual machine storage volume types

Storage volume typeDescription

ephemeral

A local copy-on-write (COW) image that uses a network volume as a read-only backing store. The backing volume must be a PersistentVolumeClaim. The ephemeral image is created when the virtual machine starts and stores all writes locally. The ephemeral image is discarded when the virtual machine is stopped, restarted, or deleted. The backing volume (PVC) is not mutated in any way.

persistentVolumeClaim

Attaches an available PV to a virtual machine. Attaching a PV allows for the virtual machine data to persist between sessions.

Importing an existing virtual machine disk into a PVC by using CDI and attaching the PVC to a virtual machine instance is the recommended method for importing existing virtual machines into OpenShift Container Platform. There are some requirements for the disk to be used within a PVC.

dataVolume

Data volumes build on the persistentVolumeClaim disk type by managing the process of preparing the virtual machine disk via an import, clone, or upload operation. VMs that use this volume type are guaranteed not to start until the volume is ready.

Specify type: dataVolume or type: "". If you specify any other value for type, such as persistentVolumeClaim, a warning is displayed, and the virtual machine does not start.

cloudInitNoCloud

Attaches a disk that contains the referenced cloud-init NoCloud data source, providing user data and metadata to the virtual machine. A cloud-init installation is required inside the virtual machine disk.

containerDisk

References an image, such as a virtual machine disk, that is stored in the container image registry. The image is pulled from the registry and attached to the virtual machine as a disk when the virtual machine is launched.

A containerDisk volume is not limited to a single virtual machine and is useful for creating large numbers of virtual machine clones that do not require persistent storage.

Only RAW and QCOW2 formats are supported disk types for the container image registry. QCOW2 is recommended for reduced image size.

Note

A containerDisk volume is ephemeral. It is discarded when the virtual machine is stopped, restarted, or deleted. A containerDisk volume is useful for read-only file systems such as CD-ROMs or for disposable virtual machines.

emptyDisk

Creates an additional sparse QCOW2 disk that is tied to the life-cycle of the virtual machine interface. The data survives guest-initiated reboots in the virtual machine but is discarded when the virtual machine stops or is restarted from the web console. The empty disk is used to store application dependencies and data that otherwise exceeds the limited temporary file system of an ephemeral disk.

The disk capacity size must also be provided.

10.1.6. About RunStrategies for virtual machines

A RunStrategy for virtual machines determines a virtual machine instance’s (VMI) behavior, depending on a series of conditions. The spec.runStrategy setting exists in the virtual machine configuration process as an alternative to the spec.running setting. The spec.runStrategy setting allows greater flexibility for how VMIs are created and managed, in contrast to the spec.running setting with only true or false responses. However, the two settings are mutually exclusive. Only either spec.running or spec.runStrategy can be used. An error occurs if both are used.

There are four defined RunStrategies.

Always
A VMI is always present when a virtual machine is created. A new VMI is created if the original stops for any reason, which is the same behavior as spec.running: true.
RerunOnFailure
A VMI is re-created if the previous instance fails due to an error. The instance is not re-created if the virtual machine stops successfully, such as when it shuts down.
Manual
The start, stop, and restart virtctl client commands can be used to control the VMI’s state and existence.
Halted
No VMI is present when a virtual machine is created, which is the same behavior as spec.running: false.

Different combinations of the start, stop and restart virtctl commands affect which RunStrategy is used.

The following table follows a VM’s transition from different states. The first column shows the VM’s initial RunStrategy. Each additional column shows a virtctl command and the new RunStrategy after that command is run.

Initial RunStrategystartstoprestart

Always

-

Halted

Always

RerunOnFailure

-

Halted

RerunOnFailure

Manual

Manual

Manual

Manual

Halted

Always

-

-

Note

In OpenShift Virtualization clusters installed using installer-provisioned infrastructure, when a node fails the MachineHealthCheck and becomes unavailable to the cluster, VMs with a RunStrategy of Always or RerunOnFailure are rescheduled on a new node.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  RunStrategy: Always 1
  template:
# ...
1
The VMI’s current RunStrategy setting.

10.1.7. Additional resources

10.2. Editing virtual machines

You can update a virtual machine configuration using either the YAML editor in the web console or the OpenShift CLI on the command line. You can also update a subset of the parameters in the Virtual Machine Details screen.

10.2.1. Editing a virtual machine in the web console

You can edit a virtual machine by using the OpenShift Container Platform web console or the command line interface.

Procedure

  1. Navigate to VirtualizationVirtualMachines in the web console.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. Click any field that has the pencil icon, which indicates that the field is editable. For example, click the current Boot mode setting, such as BIOS or UEFI, to open the Boot mode window and select an option from the list.
  4. Click Save.
Note

If the virtual machine is running, changes to Boot Order or Flavor will not take effect until you restart the virtual machine.

You can view pending changes by clicking View Pending Changes on the right side of the relevant field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

10.2.2. Editing a virtual machine YAML configuration using the web console

You can edit the YAML configuration of a virtual machine in the web console. Some parameters cannot be modified. If you click Save with an invalid configuration, an error message indicates the parameter that cannot be changed.

Note

Navigating away from the YAML screen while editing cancels any changes to the configuration you have made.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine.
  3. Click the YAML tab to display the editable configuration.
  4. Optional: You can click Download to download the YAML file locally in its current state.
  5. Edit the file and click Save.

A confirmation message shows that the modification has been successful and includes the updated version number for the object.

10.2.3. Editing a virtual machine YAML configuration using the CLI

Use this procedure to edit a virtual machine YAML configuration using the CLI.

Prerequisites

  • You configured a virtual machine with a YAML object configuration file.
  • You installed the oc CLI.

Procedure

  1. Run the following command to update the virtual machine configuration:

    $ oc edit <object_type> <object_ID>
  2. Open the object configuration.
  3. Edit the YAML.
  4. If you edit a running virtual machine, you need to do one of the following:

    • Restart the virtual machine.
    • Run the following command for the new configuration to take effect:

      $ oc apply <object_type> <object_ID>

10.2.4. Adding a virtual disk to a virtual machine

Use this procedure to add a virtual disk to a virtual machine.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. On the Configuration → Disks tab, click Add disk.
  4. Specify the Source, Name, Size, Type, Interface, and Storage Class.

    1. Optional: You can enable preallocation if you use a blank disk source and require maximum write performance when creating data volumes. To do so, select the Enable preallocation checkbox.
    2. Optional: You can clear Apply optimized StorageProfile settings to change the Volume Mode and Access Mode for the virtual disk. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map.
  5. Click Add.
Note

If the virtual machine is running, the new disk is in the pending restart state and will not be attached until you restart the virtual machine.

The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

To configure storage class defaults, use storage profiles. For more information, see Customizing the storage profile.

10.2.4.1. Storage fields
NameSelectionDescription

Source

Blank (creates PVC)

Create an empty disk.

Import via URL (creates PVC)

Import content via URL (HTTP or HTTPS endpoint).

Use an existing PVC

Use a PVC that is already available in the cluster.

Clone existing PVC (creates PVC)

Select an existing PVC available in the cluster and clone it.

Import via Registry (creates PVC)

Import content via container registry.

Container (ephemeral)

Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines.

Name

 

Name of the disk. The name can contain lowercase letters (a-z), numbers (0-9), hyphens (-), and periods (.), up to a maximum of 253 characters. The first and last characters must be alphanumeric. The name must not contain uppercase letters, spaces, or special characters.

Size

 

Size of the disk in GiB.

Type

 

Type of disk. Example: Disk or CD-ROM

Interface

 

Type of disk device. Supported interfaces are virtIO, SATA, and SCSI.

Storage Class

 

The storage class that is used to create the disk.

Advanced storage settings

The following advanced storage settings are optional and available for Blank, Import via URL, and Clone existing PVC disks. Before OpenShift Virtualization 4.11, if you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults config map. In OpenShift Virtualization 4.11 and later, the system uses the default values from the storage profile.

Note

Use storage profiles to ensure consistent advanced storage settings when provisioning storage for OpenShift Virtualization.

To manually specify Volume Mode and Access Mode, you must clear the Apply optimized StorageProfile settings checkbox, which is selected by default.

NameMode descriptionParameterParameter description

Volume Mode

Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem.

Filesystem

Stores the virtual disk on a file system-based volume.

Block

Stores the virtual disk directly on the block volume. Only use Block if the underlying storage supports it.

Access Mode

Access mode of the persistent volume.

ReadWriteOnce (RWO)

Volume can be mounted as read-write by a single node.

ReadWriteMany (RWX)

Volume can be mounted as read-write by many nodes at one time.

Note

This is required for some features, such as live migration of virtual machines between nodes.

ReadOnlyMany (ROX)

Volume can be mounted as read only by many nodes.

10.2.5. Adding a secret, config map, or service account to a virtual machine

You add a secret, config map, or service account to a virtual machine by using the OpenShift Container Platform web console.

These resources are added to the virtual machine as disks. You then mount the secret, config map, or service account as you would mount any other disk.

If the virtual machine is running, changes do not take effect until you restart the virtual machine. The newly added resources are marked as pending changes at the top of the page.

Prerequisites

  • The secret, config map, or service account that you want to add must exist in the same namespace as the target virtual machine.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. Click ConfigurationEnvironment.
  4. Click Add Config Map, Secret or Service Account.
  5. Click Select a resource and select a resource from the list. A six character serial number is automatically generated for the selected resource.
  6. Optional: Click Reload to revert the environment to its last saved state.
  7. Click Save.

Verification

  1. On the VirtualMachine details page, click ConfigurationDisks and verify that the resource is displayed in the list of disks.
  2. Restart the virtual machine by clicking ActionsRestart.

You can now mount the secret, config map, or service account as you would mount any other disk.

Additional resources for config maps, secrets, and service accounts

10.2.6. Adding a network interface to a virtual machine

Use this procedure to add a network interface to a virtual machine.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. On the Configuration → Network interfaces tab, click Add Network Interface.
  4. In the Add Network Interface window, specify the Name, Model, Network, Type, and MAC Address of the network interface.
  5. Click Add.
Note

If the virtual machine is running, the new network interface is in the pending restart state and changes will not take effect until you restart the virtual machine.

The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

10.2.6.1. Networking fields
NameDescription

Name

Name for the network interface controller.

Model

Indicates the model of the network interface controller. Supported values are e1000e and virtio.

Network

List of available network attachment definitions.

Type

List of available binding methods. Select the binding method suitable for the network interface:

  • Default pod network: masquerade
  • Linux bridge network: bridge
  • SR-IOV network: SR-IOV

MAC Address

MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically.

10.3. Editing boot order

You can update the values for a boot order list by using the web console or the CLI.

With Boot Order in the Virtual Machine Overview page, you can:

  • Select a disk or network interface controller (NIC) and add it to the boot order list.
  • Edit the order of the disks or NICs in the boot order list.
  • Remove a disk or NIC from the boot order list, and return it back to the inventory of bootable sources.

10.3.1. Adding items to a boot order list in the web console

Add items to a boot order list by using the web console.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. Click the Details tab.
  4. Click the pencil icon that is located on the right side of Boot Order. If a YAML configuration does not exist, or if this is the first time that you are creating a boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file.
  5. Click Add Source and select a bootable disk or network interface controller (NIC) for the virtual machine.
  6. Add any additional disks or NICs to the boot order list.
  7. Click Save.
Note

If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine.

You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

10.3.2. Editing a boot order list in the web console

Edit the boot order list in the web console.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. Click the Details tab.
  4. Click the pencil icon that is located on the right side of Boot Order.
  5. Choose the appropriate method to move the item in the boot order list:

    • If you do not use a screen reader, hover over the arrow icon next to the item that you want to move, drag the item up or down, and drop it in a location of your choice.
    • If you use a screen reader, press the Up Arrow key or Down Arrow key to move the item in the boot order list. Then, press the Tab key to drop the item in a location of your choice.
  6. Click Save.
Note

If the virtual machine is running, changes to the boot order list will not take effect until you restart the virtual machine.

You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

10.3.3. Editing a boot order list in the YAML configuration file

Edit the boot order list in a YAML configuration file by using the CLI.

Procedure

  1. Open the YAML configuration file for the virtual machine by running the following command:

    $ oc edit vm example
  2. Edit the YAML file and modify the values for the boot order associated with a disk or network interface controller (NIC). For example:

    disks:
      - bootOrder: 1 1
        disk:
          bus: virtio
        name: containerdisk
      - disk:
          bus: virtio
        name: cloudinitdisk
      - cdrom:
          bus: virtio
        name: cd-drive-1
    interfaces:
      - boot Order: 2 2
        macAddress: '02:96:c4:00:00'
        masquerade: {}
        name: default
    1
    The boot order value specified for the disk.
    2
    The boot order value specified for the network interface controller.
  3. Save the YAML file.
  4. Click reload the content to apply the updated boot order values from the YAML file to the boot order list in the web console.

10.3.4. Removing items from a boot order list in the web console

Remove items from a boot order list by using the web console.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Select a virtual machine to open the VirtualMachine details page.
  3. Click the Details tab.
  4. Click the pencil icon that is located on the right side of Boot Order.
  5. Click the Remove icon delete next to the item. The item is removed from the boot order list and saved in the list of available boot sources. If you remove all items from the boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file.
Note

If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine.

You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.

10.4. Deleting virtual machines

You can delete a virtual machine from the web console or by using the oc command line interface.

10.4.1. Deleting a virtual machine using the web console

Deleting a virtual machine permanently removes it from the cluster.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationVirtualMachines from the side menu.
  2. Click the Options menu kebab beside a virtual machine and select Delete.

    Alternatively, click the virtual machine name to open the VirtualMachine details page and click ActionsDelete.

  3. Optional: Select With grace period or clear Delete disks.
  4. Click Delete to permanently delete the virtual machine.

10.4.2. Deleting a virtual machine by using the CLI

You can delete a virtual machine by using the oc command line interface (CLI). The oc client enables you to perform actions on multiple virtual machines.

Prerequisites

  • Identify the name of the virtual machine that you want to delete.

Procedure

  • Delete the virtual machine by running the following command:

    $ oc delete vm <vm_name>
    Note

    This command only deletes a VM in the current project. Specify the -n <project_name> option if the VM you want to delete is in a different project or namespace.

10.5. Exporting virtual machines

You can export a virtual machine (VM) and its associated disks in order to import a VM into another cluster or to analyze the volume for forensic purposes.

You create a VirtualMachineExport custom resource (CR) by using the command line interface.

Alternatively, you can use the virtctl vmexport command to create a VirtualMachineExport CR and to download exported volumes.

10.5.1. Creating a VirtualMachineExport custom resource

You can create a VirtualMachineExport custom resource (CR) to export the following objects:

  • Virtual machine (VM): Exports the persistent volume claims (PVCs) of a specified VM.
  • VM snapshot: Exports PVCs contained in a VirtualMachineSnapshot CR.
  • PVC: Exports a PVC. If the PVC is used by another pod, such as the virt-launcher pod, the export remains in a Pending state until the PVC is no longer in use.

The VirtualMachineExport CR creates internal and external links for the exported volumes. Internal links are valid within the cluster. External links can be accessed by using an Ingress or Route.

The export server supports the following file formats:

  • raw: Raw disk image file.
  • gzip: Compressed disk image file.
  • dir: PVC directory and files.
  • tar.gz: Compressed PVC file.

Prerequisites

  • The VM must be shut down for a VM export.

Procedure

  1. Create a VirtualMachineExport manifest to export a volume from a VirtualMachine, VirtualMachineSnapshot, or PersistentVolumeClaim CR according to the following example and save it as example-export.yaml:

    VirtualMachineExport example

    apiVersion: export.kubevirt.io/v1alpha1
    kind: VirtualMachineExport
    metadata:
      name: example-export
    spec:
      source:
        apiGroup: "kubevirt.io" 1
        kind: VirtualMachine 2
        name: example-vm
      ttlDuration: 1h 3

    1
    Specify the appropriate API group:
    • "kubevirt.io" for VirtualMachine.
    • "snapshot.kubevirt.io" for VirtualMachineSnapshot.
    • "" for PersistentVolumeClaim.
    2
    Specify VirtualMachine, VirtualMachineSnapshot, or PersistentVolumeClaim.
    3
    Optional. The default duration is 2 hours.
  2. Create the VirtualMachineExport CR:

    $ oc create -f example-export.yaml
  3. Get the VirtualMachineExport CR:

    $ oc get vmexport example-export -o yaml

    The internal and external links for the exported volumes are displayed in the status stanza:

    Output example

    apiVersion: export.kubevirt.io/v1alpha1
    kind: VirtualMachineExport
    metadata:
      name: example-export
      namespace: example
    spec:
      source:
        apiGroup: ""
        kind: PersistentVolumeClaim
        name: example-pvc
      tokenSecretRef: example-token
    status:
      conditions:
      - lastProbeTime: null
        lastTransitionTime: "2022-06-21T14:10:09Z"
        reason: podReady
        status: "True"
        type: Ready
      - lastProbeTime: null
        lastTransitionTime: "2022-06-21T14:09:02Z"
        reason: pvcBound
        status: "True"
        type: PVCReady
      links:
        external: 1
          cert: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----
          volumes:
          - formats:
            - format: raw
              url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img
            - format: gzip
              url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img.gz
            name: example-disk
        internal:  2
          cert: |-
            -----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----
          volumes:
          - formats:
            - format: raw
              url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img
            - format: gzip
              url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img.gz
            name: example-disk
      phase: Ready
      serviceName: virt-export-example-export

    1
    External links are accessible from outside the cluster by using an Ingress or Route.
    2
    Internal links are only valid inside the cluster.

10.5.2. Accessing exported virtual machine manifests

After you export a virtual machine (VM) or snapshot, you can get the VirtualMachine manifest and related information from the export server.

Prerequisites

  • You exported a virtual machine or VM snapshot by creating a VirtualMachineExport custom resource (CR).

    Note

    VirtualMachineExport objects that have the spec.source.kind: PersistentVolumeClaim parameter do not generate virtual machine manifests.

Procedure

  1. To access the manifests, you must first copy the certificates from the source cluster to the target cluster.

    1. Log in to the source cluster.
    2. Save the certificates to the cacert.crt file by running the following command:

      $ oc get vmexport <export_name> -o jsonpath={.status.links.external.cert} > cacert.crt 1
      1
      Replace <export_name> with the metadata.name value from the VirtualMachineExport object.
    3. Copy the cacert.crt file to the target cluster.
  2. Decode the token in the source cluster and save it to the token_decode file by running the following command:

    $ oc get secret export-token-<export_name> -o jsonpath={.data.token} | base64 --decode > token_decode 1
    1
    Replace <export_name> with the metadata.name value from the VirtualMachineExport object.
  3. Copy the token_decode file to the target cluster.
  4. Get the VirtualMachineExport custom resource by running the following command:

    $ oc get vmexport <export_name> -o yaml
  5. Review the status.links stanza, which is divided into external and internal sections. Note the manifests.url fields within each section:

    Example output

    apiVersion: export.kubevirt.io/v1alpha1
    kind: VirtualMachineExport
    metadata:
      name: example-export
    spec:
      source:
        apiGroup: "kubevirt.io"
        kind: VirtualMachine
        name: example-vm
      tokenSecretRef: example-token
    status:
    #...
      links:
        external:
    #...
          manifests:
          - type: all
            url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/all 1
          - type: auth-header-secret
            url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret 2
        internal:
    #...
          manifests:
          - type: all
            url: https://virt-export-export-pvc.default.svc/internal/manifests/all 3
          - type: auth-header-secret
            url: https://virt-export-export-pvc.default.svc/internal/manifests/secret
      phase: Ready
      serviceName: virt-export-example-export

    1
    Contains the VirtualMachine manifest, DataVolume manifest, if present, and a ConfigMap manifest that contains the public certificate for the external URL’s ingress or route.
    2
    Contains a secret containing a header that is compatible with Containerized Data Importer (CDI). The header contains a text version of the export token.
    3
    Contains the VirtualMachine manifest, DataVolume manifest, if present, and a ConfigMap manifest that contains the certificate for the internal URL’s export server.
  6. Log in to the target cluster.
  7. Get the Secret manifest by running the following command:

    $ curl --cacert cacert.crt <secret_manifest_url> -H \ 1
    "x-kubevirt-export-token:token_decode" -H \ 2
    "Accept:application/yaml"
    1
    Replace <secret_manifest_url> with an auth-header-secret URL from the VirtualMachineExport YAML output.
    2
    Reference the token_decode file that you created earlier.

    For example:

    $ curl --cacert cacert.crt https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret -H "x-kubevirt-export-token:token_decode" -H "Accept:application/yaml"
  8. Get the manifests of type: all, such as the ConfigMap and VirtualMachine manifests, by running the following command:

    $ curl --cacert cacert.crt <all_manifest_url> -H \ 1
    "x-kubevirt-export-token:token_decode" -H \ 2
    "Accept:application/yaml"
    1
    Replace <all_manifest_url> with a URL from the VirtualMachineExport YAML output.
    2
    Reference the token_decode file that you created earlier.

    For example:

    $ curl --cacert cacert.crt https://vmexport-proxy.test.net/api/export.kubevirt.io/v1alpha1/namespaces/example/virtualmachineexports/example-export/external/manifests/all -H "x-kubevirt-export-token:token_decode" -H "Accept:application/yaml"

Next steps

  • You can now create the ConfigMap and VirtualMachine objects on the target cluster by using the exported manifests.

10.5.3. Additional resources

10.6. Managing virtual machine instances

If you have standalone virtual machine instances (VMIs) that were created independently outside of the OpenShift Virtualization environment, you can manage them by using the web console or by using oc or virtctl commands from the command-line interface (CLI).

The virtctl command provides more virtualization options than the oc command. For example, you can use virtctl to pause a VM or expose a port.

10.6.1. About virtual machine instances

A virtual machine instance (VMI) is a representation of a running virtual machine (VM). When a VMI is owned by a VM or by another object, you manage it through its owner in the web console or by using the oc command-line interface (CLI).

A standalone VMI is created and started independently with a script, through automation, or by using other methods in the CLI. In your environment, you might have standalone VMIs that were developed and started outside of the OpenShift Virtualization environment. You can continue to manage those standalone VMIs by using the CLI. You can also use the web console for specific tasks associated with standalone VMIs:

  • List standalone VMIs and their details.
  • Edit labels and annotations for a standalone VMI.
  • Delete a standalone VMI.

When you delete a VM, the associated VMI is automatically deleted. You delete a standalone VMI directly because it is not owned by VMs or other objects.

Note

Before you uninstall OpenShift Virtualization, list and view the standalone VMIs by using the CLI or the web console. Then, delete any outstanding VMIs.

10.6.2. Listing all virtual machine instances using the CLI

You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc command-line interface (CLI).

Procedure

  • List all VMIs by running the following command:

    $ oc get vmis -A

10.6.3. Listing standalone virtual machine instances using the web console

Using the web console, you can list and view standalone virtual machine instances (VMIs) in your cluster that are not owned by virtual machines (VMs).

Note

VMIs that are owned by VMs or other objects are not displayed in the web console. The web console displays only standalone VMIs. If you want to list all VMIs in your cluster, you must use the CLI.

Procedure

  • Click VirtualizationVirtualMachines from the side menu.

    You can identify a standalone VMI by a dark colored badge next to its name.

10.6.4. Editing a standalone virtual machine instance using the web console

You can edit the annotations and labels of a standalone virtual machine instance (VMI) using the web console. Other fields are not editable.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationVirtualMachines from the side menu.
  2. Select a standalone VMI to open the VirtualMachineInstance details page.
  3. On the Details tab, click the pencil icon beside Annotations or Labels.
  4. Make the relevant changes and click Save.

10.6.5. Deleting a standalone virtual machine instance using the CLI

You can delete a standalone virtual machine instance (VMI) by using the oc command-line interface (CLI).

Prerequisites

  • Identify the name of the VMI that you want to delete.

Procedure

  • Delete the VMI by running the following command:

    $ oc delete vmi <vmi_name>

10.6.6. Deleting a standalone virtual machine instance using the web console

Delete a standalone virtual machine instance (VMI) from the web console.

Procedure

  1. In the OpenShift Container Platform web console, click VirtualizationVirtualMachines from the side menu.
  2. Click ActionsDelete VirtualMachineInstance.
  3. In the confirmation pop-up window, click Delete to permanently delete the standalone VMI.

10.7. Controlling virtual machine states

You can stop, start, restart, and unpause virtual machines from the web console.

You can use virtctl to manage virtual machine states and perform other actions from the CLI. For example, you can use virtctl to force stop a VM or expose a port.

10.7.1. Starting a virtual machine

You can start a virtual machine from the web console.

Procedure

  1. Click VirtualizationVirtualMachines from the side menu.
  2. Find the row that contains the virtual machine that you want to start.
  3. Navigate to the appropriate menu for your use case:

    • To stay on this page, where you can perform actions on multiple virtual machines:

      1. Click the Options menu kebab located at the far right end of the row.
    • To view comprehensive information about the selected virtual machine before you start it:

      1. Access the VirtualMachine details page by clicking the name of the virtual machine.
      2. Click Actions.
  4. Select Restart.
  5. In the confirmation window, click Start to start the virtual machine.
Note

When you start virtual machine that is provisioned from a URL source for the first time, the virtual machine has a status of Importing while OpenShift Virtualization imports the container from the URL endpoint. Depending on the si