OpenShift Virtualization
OpenShift Virtualization installation, usage, and release notes
Abstract
Chapter 1. About OpenShift Virtualization
Learn about OpenShift Virtualization’s capabilities and support scope.
1.1. What you can do with OpenShift Virtualization
OpenShift Virtualization is an add-on to OpenShift Container Platform that allows you to run and manage virtual machine workloads alongside container workloads.
OpenShift Virtualization adds new objects into your OpenShift Container Platform cluster via Kubernetes custom resources to enable virtualization tasks. These tasks include:
- Creating and managing Linux and Windows virtual machines
- Connecting to virtual machines through a variety of consoles and CLI tools
- Importing and cloning existing virtual machines
- Managing network interface controllers and storage disks attached to virtual machines
- Live migrating virtual machines between nodes
An enhanced web console provides a graphical portal to manage these virtualized resources alongside the OpenShift Container Platform cluster containers and infrastructure.
OpenShift Virtualization is tested with OpenShift Container Storage (OCS) and designed to use with OCS features for the best experience.
You can use OpenShift Virtualization with either the OVN-Kubernetes or the OpenShiftSDN default Container Network Interface (CNI) network provider.
1.1.1. OpenShift Virtualization supported cluster version
OpenShift Virtualization 2.6 is supported for use on OpenShift Container Platform 4.7 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.
Chapter 2. OpenShift Virtualization release notes
2.1. About Red Hat OpenShift Virtualization
Red Hat OpenShift Virtualization enables you to bring traditional virtual machines (VMs) into OpenShift Container Platform where they run alongside containers, and are managed as native Kubernetes objects.
OpenShift Virtualization is represented by the logo.
You can use OpenShift Virtualization with either the OVN-Kubernetes or the OpenShiftSDN default Container Network Interface (CNI) network provider.
Learn more about what you can do with OpenShift Virtualization.
2.1.1. OpenShift Virtualization supported cluster version
OpenShift Virtualization 2.6 is supported for use on OpenShift Container Platform 4.7 clusters. To use the latest z-stream release of OpenShift Virtualization, you must first upgrade to the latest version of OpenShift Container Platform.
2.1.2. Supported guest operating systems
OpenShift Virtualization guests can use the following operating systems:
- Red Hat Enterprise Linux 6, 7, and 8.
- Microsoft Windows Server 2012 R2, 2016, and 2019.
- Microsoft Windows 10.
Other operating system templates shipped with OpenShift Virtualization are not supported.
2.2. Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
2.3. New and changed features
OpenShift Virtualization is certified in Microsoft’s Windows Server Virtualization Validation Program (SVVP) to run Windows Server workloads.
The SVVP Certification applies to:
- Red Hat Enterprise Linux CoreOS workers. In the Microsoft SVVP Catalog, they are named Red Hat OpenShift Container Platform 4 on RHEL CoreOS.
- Intel and AMD CPUs.
-
Virtual machines that have the
LiveMigratable
condition set toTrue
and thespec.evictionStrategy
field set toLiveMigrate
are now migrated when a node is drained in preparation for maintenance. You can set a node to maintenance mode in the CLI by running theoc adm drain
command. You can now boot a virtual machine (VM) in Extensible Firmware Interface (EFI) mode.
NoteOpenShift Virtualization only supports a VM with Secure Boot when using EFI mode. If Secure Boot is not enabled, the VM crashes repeatedly. However, the VM might not support Secure Boot. Before you boot a VM, verify that it supports Secure Boot by checking the VM settings.
2.3.1. Installation
- Cluster administrators can now configure node placement rules for OpenShift Virtualization components, including the hostpath provisioner.
2.3.2. Storage
-
When you add a virtual machine disk to a persistent volume claim (PVC) that uses the
Filesystem
volume mode, the Containerized Data Importer (CDI) now reserves 5.5% of the PVC space for file system overhead. If the default value is not ideal for your use case, you can change the percentage that CDI reserves for this purpose. -
When preparing local storage for a virtual machine disk image, a volume might be allocated from a different node than is required by the virtual machine. To prevent scheduling problems, the Containerized Data Importer now integrates with the Kubernetes
WaitForFirstConsumer
binding mode to ensure that volumes are allocated from the correct node. - The Containerized Data Importer can now preallocate disk space to import and upload data and create blank data volumes at a faster speed.
You can now control which network the Containerized Data Importer (CDI) uses when importing virtual machine disk images by adding the following annotation to the
metadata.annotations
field of aDataVolume
object:k8s.v1.cni.cncf.io/networks: <network_name>
- If you use this annotation, the transfer pod will use both the default network from the cluster and the secondary Multus network.
2.3.3. Web console
- The virtual machine wizard has been redesigned to simplify the process of creating virtual machines by using preconfigured virtual machine templates.
- When you create a virtual machine template with the interactive wizard, selecting the Operating System will automatically select the default Flavor and Workload Type for that operating system. Virtual machines created from a virtual machine template now have these details automatically selected.
2.4. Notable technical changes
-
The procedure for installing OpenShift Virtualization in the web console has been streamlined. You can now create the
HyperConverged
custom resource immediately after installing the OpenShift Virtualization Operator by clicking Create HyperConverged. -
Previously, there was a
spec.BareMetalPlatform
field in theHyperConverged
object. This field has been removed. -
The Containerized Data Importer (CDI) configuration parameters have moved from the
CDIConfig
object to theCDI
object. All changes to the CDI configuration must now be made in thespec.config
field of theCDI
object. Istio sidecar injection to transfer pods is now disabled by default. You can override the default option by using the following annotation in the
metadata.annotations
field of aDataVolume
object:sidecar.istio.io/inject: “true”
- Set this option to enable sidecar injection to the transfer pods.
2.5. Known issues
-
A virtual machine instance (VMI) fails to migrate if the
cdrom
drive is set toreadonly: true
in the VMI spec. The following message is displayed:Operation not supported: Cannot migrate empty or read-only disk sdb
. (BZ#1927378) Some Containerized Data Importer (CDI) operations are currently not preallocated when requested. These include:
- Creating blank block disks
- Importing VMWare disk images
If a cloning operation is initiated before the source is available to be cloned, the operation stalls indefinitely. This is because the clone authorization expires before the cloning operation starts. (BZ#1855182)
-
As a workaround, delete the
DataVolume
object that is requesting the clone. When the source is available, recreate theDataVolume
object that you deleted so that the cloning operation can complete successfully.
-
As a workaround, delete the
- The Containerized Data Importer and KubeVirt depend on QEMU which does not support NFS version 3. Therefore, only NFS version 4 is supported. (BZ#1892445)
The name of the Fedora PVC in the
openshift-virtualization-os-images
namespace isfedora
, instead offedora32
. If you populated thefedora32
PVC in OpenShift Virtualization 2.5 or earlier, the virtual machine does not appear in the web console and you cannot use it to clone another virtual machine. (BZ#1913352)-
As a workaround, upload a Fedora image by naming the PVC
fedora
instead offedora32
.
-
As a workaround, upload a Fedora image by naming the PVC
When creating a HPP boot source, the data volume is
pending
with aWaitForFirstConsumer
status if a user creates the boot source using any method except the Upload local file (creates PVC) option. (BZ#1929177)As a workaround, in the Storage → Persistent Volume Claims web console screen, edit the YAML of the underlying PVC of the data volume to add the
cdi.kubevirt.io/storage.bind.immediate.requested: "true"
annotation:metadata: annotations: cdi.kubevirt.io/storage.bind.immediate.requested: "true"
If you use a Fedora image as a boot source, it is no longer attached to a template if the PVC that you used to attach the boot source was previously provisioned. (BZ#1907187) (BZ#1913352)
-
As a workaround, attach a new PVC with the name
fedora
to a template before using it to create virtual machines from boot sources.
-
As a workaround, attach a new PVC with the name
If your OpenShift Container Platform cluster uses OVN-Kubernetes as the default Container Network Interface (CNI) provider, you cannot attach a Linux bridge or bonding to the default interface of a host because of a change in the host network topology of OVN-Kubernetes. (BZ#1885605)
- As a workaround, you can use a secondary network interface connected to your host, or switch to the OpenShift SDN default CNI provider.
Running virtual machines that cannot be live migrated might block an OpenShift Container Platform cluster upgrade. This includes virtual machines that use hostpath-provisioner storage or SR-IOV network interfaces. (BZ#1858777)
As a workaround, you can reconfigure the virtual machines so that they can be powered off during a cluster upgrade. In the
spec
section of the virtual machine configuration file:-
Remove the
evictionStrategy: LiveMigrate
field. See Configuring virtual machine eviction strategy for more information on how to configure eviction strategy. -
Set the
runStrategy
field toAlways
.
-
Remove the
Live migration fails when nodes have different CPU models. Even in cases where nodes have the same physical CPU model, differences introduced by microcode updates have the same effect. This is because the default settings trigger host CPU passthrough behavior, which is incompatible with live migration. (BZ#1760028)
As a workaround, set the default CPU model in the
kubevirt-config
config map, as shown in the following example:NoteYou must make this change before starting the virtual machines that support live migration.
Open the
kubevirt-config
config map for editing by running the following command:$ oc edit configmap kubevirt-config -n openshift-cnv
Edit the config map:
kind: ConfigMap metadata: name: kubevirt-config data: default-cpu-model: "<cpu-model>" 1
- 1
- Replace
<cpu-model>
with the actual CPU model value. You can determine this value by runningoc describe node <node>
for all nodes and looking at thecpu-model-<name>
labels. Select the CPU model that is present on all of your nodes.
If you enter the wrong credentials for the RHV Manager while importing a RHV VM, the Manager might lock the admin user account because the
vm-import-operator
tries repeatedly to connect to the RHV API. (BZ#1887140)To unlock the account, log in to the Manager and enter the following command:
$ ovirt-aaa-jdbc-tool user unlock admin
Chapter 3. Installing OpenShift Virtualization
3.1. Preparing your cluster for OpenShift Virtualization
Review this section before you install OpenShift Virtualization to ensure that your cluster meets the requirements.
You can use any installation method, including user-provisioned, installer-provisioned, or assisted installer, to deploy OpenShift Container Platform. However, the installation method and the cluster topology might affect OpenShift Virtualization functionality, such as snapshots or live migration.
FIPS mode
If you install your cluster in FIPS mode, no additional setup is required for OpenShift Virtualization.
3.1.1. Hardware and operating system requirements
Review the following hardware and operating system requirements for OpenShift Virtualization.
Supported platforms
- On-premise bare metal servers
- Amazon Web Services bare metal instances
Installing OpenShift Virtualization on an AWS bare metal instance is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- Bare metal instances or servers offered by other cloud providers are not supported.
CPU requirements
- Supported by Red Hat Enterprise Linux (RHEL) 8
- Support for Intel 64 or AMD64 CPU extensions
- Intel VT or AMD-V hardware virtualization extensions enabled
- NX (no execute) flag enabled
Storage requirements
- Supported by OpenShift Container Platform
Operating system requirements
Red Hat Enterprise Linux CoreOS (RHCOS) installed on worker nodes
NoteRHEL worker nodes are not supported.
Additional resources
- About RHCOS
- Red Hat Ecosystem Catalog for supported CPUs
- Supported storage
3.1.2. Physical resource overhead requirements
OpenShift Virtualization is an add-on to OpenShift Container Platform and imposes additional overhead that you must account for when planning a cluster. Each cluster machine must accommodate the following overhead requirements in addition to the OpenShift Container Platform requirements. Oversubscribing the physical resources in a cluster can affect performance.
The numbers noted in this documentation are based on Red Hat’s test methodology and setup. These numbers can vary based on your own individual setup and environments.
3.1.2.1. Memory overhead
Calculate the memory overhead values for OpenShift Virtualization by using the equations below.
Cluster memory overhead
Memory overhead per infrastructure node ≈ 150 MiB
Memory overhead per worker node ≈ 360 MiB
Additionally, OpenShift Virtualization environment resources require a total of 2179 MiB of RAM that is spread across all infrastructure nodes.
Virtual machine memory overhead
Memory overhead per virtual machine ≈ (1.002 * requested memory) + 146 MiB \ + 8 MiB * (number of vCPUs) \ 1 + 16 MiB * (number of graphics devices) 2
If your environment includes a Single Root I/O Virtualization (SR-IOV) network device or a Graphics Processing Unit (GPU), allocate 1 GiB additional memory overhead for each device.
3.1.2.2. CPU overhead
Calculate the cluster processor overhead requirements for OpenShift Virtualization by using the equation below. The CPU overhead per virtual machine depends on your individual setup.
Cluster CPU overhead
CPU overhead for infrastructure nodes ≈ 4 cores
OpenShift Virtualization increases the overall utilization of cluster level services such as logging, routing, and monitoring. To account for this workload, ensure that nodes that host infrastructure components have capacity allocated for 4 additional cores (4000 millicores) distributed across those nodes.
CPU overhead for worker nodes ≈ 2 cores + CPU overhead per virtual machine
Each worker node that hosts virtual machines must have capacity for 2 additional cores (2000 millicores) for OpenShift Virtualization management workloads in addition to the CPUs required for virtual machine workloads.
Virtual machine CPU overhead
If dedicated CPUs are requested, there is a 1:1 impact on the cluster CPU overhead requirement. Otherwise, there are no specific rules about how many CPUs a virtual machine requires.
3.1.2.3. Storage overhead
Use the guidelines below to estimate storage overhead requirements for your OpenShift Virtualization environment.
Cluster storage overhead
Aggregated storage overhead per node ≈ 10 GiB
10 GiB is the estimated on-disk storage impact for each node in the cluster when you install OpenShift Virtualization.
Virtual machine storage overhead
Storage overhead per virtual machine depends on specific requests for resource allocation within the virtual machine. The request could be for ephemeral storage on the node or storage resources hosted elsewhere in the cluster. OpenShift Virtualization does not currently allocate any additional ephemeral storage for the running container itself.
3.1.2.4. Example
As a cluster administrator, if you plan to host 10 virtual machines in the cluster, each with 1 GiB of RAM and 2 vCPUs, the memory impact across the cluster is 11.68 GiB. The estimated on-disk storage impact for each node in the cluster is 10 GiB and the CPU impact for worker nodes that host virtual machine workloads is a minimum of 2 cores.
3.1.3. Object maximums
You must consider the following tested object maximums when planning your cluster:
3.1.4. Restricted network environments
If you install OpenShift Virtualization in a restricted environment with no internet connectivity, you must configure Operator Lifecycle Manager for restricted networks.
If you have limited internet connectivity, you can configure proxy support in Operator Lifecycle Manager to access the Red Hat-provided OperatorHub.
3.1.5. Live migration
Live migration has the following requirements:
-
Shared storage with
ReadWriteMany
(RWX) access mode - Sufficient RAM and network bandwidth
- Appropriate CPUs with sufficient capacity on the worker nodes. If the CPUs have different capacities, live migration might be very slow or fail.
3.1.6. Snapshots and cloning
See OpenShift Virtualization storage features for snapshot and cloning requirements.
3.1.7. Cluster high-availability
Automatic high availability for installer-provisioned infrastructure (IPI) is available by deploying machine health checks.
In OpenShift Container Platform clusters installed using installer-provisioned infrastructure and with MachineHealthCheck properly configured, if a node fails the MachineHealthCheck and becomes unavailable to the cluster, it is recycled. What happens next with VMs that ran on the failed node depends on a series of conditions. See About RunStrategies for virtual machines for more detailed information about the potential outcomes and how RunStrategies affect those outcomes.
3.2. Specifying nodes for OpenShift Virtualization components
Specify the nodes where you want to deploy OpenShift Virtualization Operators, workloads, and controllers by configuring node placement rules.
You can configure node placement for some components after installing OpenShift Virtualization, but there must not be virtual machines present if you want to configure node placement for workloads.
3.2.1. About node placement for virtualization components
You might want to customize where OpenShift Virtualization deploys its components to ensure that:
- Virtual machines only deploy on nodes that are intended for virtualization workloads.
- Operators only deploy on infrastructure nodes.
- Certain nodes are unaffected by OpenShift Virtualization. For example, you have workloads unrelated to virtualization running on your cluster, and you want those workloads to be isolated from OpenShift Virtualization.
3.2.1.1. How to apply node placement rules to virtualization components
You can specify node placement rules for a component by editing the corresponding object directly or by using the web console.
-
For the OpenShift Virtualization Operators that Operator Lifecycle Manager (OLM) deploys, edit the OLM
Subscription
object directly. Currently, you cannot configure node placement rules for theSubscription
object by using the web console. -
For components that the OpenShift Virtualization Operators deploy, edit the
HyperConverged
object directly or configure it by using the web console during OpenShift Virtualization installation. For the hostpath provisioner, edit the
HostPathProvisioner
object directly or configure it by using the web console.WarningYou must schedule the hostpath provisioner and the virtualization components on the same nodes. Otherwise, virtualization pods that use the hostpath provisioner cannot run.
Depending on the object, you can use one or more of the following rule types:
nodeSelector
- Allows pods to be scheduled on nodes that are labeled with the key-value pair or pairs that you specify in this field. The node must have labels that exactly match all listed pairs.
affinity
- Enables you to use more expressive syntax to set rules that match nodes with pods. Affinity also allows for more nuance in how the rules are applied. For example, you can specify that a rule is a preference, rather than a hard requirement, so that pods are still scheduled if the rule is not satisfied.
tolerations
- Allows pods to be scheduled on nodes that have matching taints. If a taint is applied to a node, that node only accepts pods that tolerate the taint.
3.2.1.2. Node placement in the OLM Subscription object
To specify the nodes where OLM deploys the OpenShift Virtualization Operators, edit the Subscription
object during OpenShift Virtualization installation. You can include node placement rules in the spec.config
field, as shown in the following example:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: hco-operatorhub
namespace: openshift-cnv
spec:
source: redhat-operators
sourceNamespace: openshift-marketplace
name: kubevirt-hyperconverged
startingCSV: kubevirt-hyperconverged-operator.v2.6.10
channel: "stable"
config: 1
- 1
- The
config
field supportsnodeSelector
andtolerations
, but it does not supportaffinity
.
3.2.1.3. Node placement in the HyperConverged object
To specify the nodes where OpenShift Virtualization deploys its components, you can include the nodePlacement
object in the HyperConverged Cluster custom resource (CR) file that you create during OpenShift Virtualization installation. You can include nodePlacement
under the spec.infra
and spec.workloads
fields, as shown in the following example:
apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: openshift-cnv
spec:
infra:
nodePlacement: 1
...
workloads:
nodePlacement:
...
- 1
- The
nodePlacement
fields supportnodeSelector
,affinity
, andtolerations
fields.
3.2.1.4. Node placement in the HostPathProvisioner object
You can configure node placement rules in the spec.workload
field of the HostPathProvisioner
object that you create when you install the hostpath provisioner.
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
name: hostpath-provisioner
spec:
imagePullPolicy: IfNotPresent
pathConfig:
path: "</path/to/backing/directory>"
useNamingPrefix: false
workload: 1
- 1
- The
workload
field supportsnodeSelector
,affinity
, andtolerations
fields.
3.2.1.5. Additional resources
- Placing pods on specific nodes using node selectors
- Controlling pod placement on nodes using node affinity rules
- Controlling pod placement using node taints
- Installing OpenShift Virtualization using the CLI
- Installing OpenShift Virtualization using the web console
- Configuring local storage for virtual machines
3.2.2. Example manifests
The following example YAML files use nodePlacement
, affinity
, and tolerations
objects to customize node placement for OpenShift Virtualization components.
3.2.2.1. Operator Lifecycle Manager Subscription object
3.2.2.1.1. Example: Node placement with nodeSelector in the OLM Subscription object
In this example, nodeSelector
is configured so that OLM places the OpenShift Virtualization Operators on nodes that are labeled with example.io/example-infra-key = example-infra-value
.
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: hco-operatorhub namespace: openshift-cnv spec: source: redhat-operators sourceNamespace: openshift-marketplace name: kubevirt-hyperconverged startingCSV: kubevirt-hyperconverged-operator.v2.6.10 channel: "stable" config: nodeSelector: example.io/example-infra-key: example-infra-value
3.2.2.1.2. Example: Node placement with tolerations in the OLM Subscription object
In this example, nodes that are reserved for OLM to deploy OpenShift Virtualization Operators are labeled with the key=virtualization:NoSchedule
taint. Only pods with the matching tolerations are scheduled to these nodes.
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: hco-operatorhub namespace: openshift-cnv spec: source: redhat-operators sourceNamespace: openshift-marketplace name: kubevirt-hyperconverged startingCSV: kubevirt-hyperconverged-operator.v2.6.10 channel: "stable" config: tolerations: - key: "key" operator: "Equal" value: "virtualization" effect: "NoSchedule"
3.2.2.2. HyperConverged object
3.2.2.2.1. Example: Node placement with nodeSelector in the HyperConverged Cluster CR
In this example, nodeSelector
is configured so that infrastructure resources are placed on nodes that are labeled with example.io/example-infra-key = example-infra-value
and workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value
.
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: infra: nodePlacement: nodeSelector: example.io/example-infra-key: example-infra-value workloads: nodePlacement: nodeSelector: example.io/example-workloads-key: example-workloads-value
3.2.2.2.2. Example: Node placement with affinity in the HyperConverged Cluster CR
In this example, affinity
is configured so that infrastructure resources are placed on nodes that are labeled with example.io/example-infra-key = example-value
and workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value
. Nodes that have more than eight CPUs are preferred for workloads, but if they are not available, pods are still scheduled.
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: infra: nodePlacement: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: example.io/example-infra-key operator: In values: - example-infra-value workloads: nodePlacement: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: example.io/example-workloads-key operator: In values: - example-workloads-value preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: example.io/num-cpus operator: gt values: - 8
3.2.2.2.3. Example: Node placement with tolerations in the HyperConverged Cluster CR
In this example, nodes that are reserved for OpenShift Virtualization components are labeled with the key=virtualization:NoSchedule
taint. Only pods with the matching tolerations are scheduled to these nodes.
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec: workloads: nodePlacement: tolerations: - key: "key" operator: "Equal" value: "virtualization" effect: "NoSchedule"
3.2.2.3. HostPathProvisioner object
3.2.2.3.1. Example: Node placement with nodeSelector in the HostPathProvisioner object
In this example, nodeSelector
is configured so that workloads are placed on nodes labeled with example.io/example-workloads-key = example-workloads-value
.
apiVersion: hostpathprovisioner.kubevirt.io/v1beta1 kind: HostPathProvisioner metadata: name: hostpath-provisioner spec: imagePullPolicy: IfNotPresent pathConfig: path: "</path/to/backing/directory>" useNamingPrefix: false workload: nodeSelector: example.io/example-workloads-key: example-workloads-value
3.3. Installing OpenShift Virtualization using the web console
Install OpenShift Virtualization to add virtualization functionality to your OpenShift Container Platform cluster.
You can use the OpenShift Container Platform 4.7 web console to subscribe to and deploy the OpenShift Virtualization Operators.
3.3.1. Installing the OpenShift Virtualization Operator
You can install the OpenShift Virtualization Operator from the OpenShift Container Platform web console.
Prerequisites
- Install OpenShift Container Platform 4.7 on your cluster.
-
Log in to the OpenShift Container Platform web console as a user with
cluster-admin
permissions.
Procedure
- Open a browser window and log in to the OpenShift Container Platform web console.
- From the Administrator perspective, click Operators → OperatorHub.
- In the Filter by keyword field, type OpenShift Virtualization.
- Select the OpenShift Virtualization tile.
- Read the information about the Operator and click Install.
On the Install Operator page:
Select stable from the list of available Update Channel options. This ensures that:
- You install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
- When you update OpenShift Container Platform, OpenShift Virtualization automatically updates to the next minor version.
For Installed Namespace, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory
openshift-cnv
namespace, which is automatically created if it does not exist.WarningAttempting to install the OpenShift Virtualization Operator in a namespace other than
openshift-cnv
causes the installation to fail.- For Approval Strategy, ensure that Automatic, which is the default value, is selected. OpenShift Virtualization automatically updates when a new z-stream release is available.
-
Click Install to make the Operator available to the
openshift-cnv
namespace. - When the Operator installs successfully, click Create HyperConverged.
- Optional: Configure Infra and Workloads node placement options for OpenShift Virtualization components.
- Click Create to launch OpenShift Virtualization.
Verification
- Navigate to the Workloads → Pods page and monitor the OpenShift Virtualization pods until they are all Running. After all the pods display the Running state, you can use OpenShift Virtualization.
3.3.2. Next steps
You might want to additionally configure the following components:
- The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace.
- The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first.
Install OpenShift Virtualization to add virtualization functionality to your OpenShift Container Platform cluster. You can subscribe to and deploy the OpenShift Virtualization Operators by using the command line to apply manifests to your cluster.
To specify the nodes where you want OpenShift Virtualization to install its components, configure node placement rules.
3.3.3. Prerequisites
- Install OpenShift Container Platform 4.7 on your cluster.
-
Install the OpenShift CLI (
oc
). -
Log in as a user with
cluster-admin
privileges.
3.3.4. Subscribing to the OpenShift Virtualization catalog by using the CLI
Before you install OpenShift Virtualization, you must subscribe to the OpenShift Virtualization catalog. Subscribing gives the openshift-cnv
namespace access to the OpenShift Virtualization Operators.
To subscribe, configure Namespace
, OperatorGroup
, and Subscription
objects by applying a single manifest to your cluster.
Procedure
Create a YAML file that contains the following manifest:
apiVersion: v1 kind: Namespace metadata: name: openshift-cnv --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: kubevirt-hyperconverged-group namespace: openshift-cnv spec: targetNamespaces: - openshift-cnv --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: hco-operatorhub namespace: openshift-cnv spec: source: redhat-operators sourceNamespace: openshift-marketplace name: kubevirt-hyperconverged startingCSV: kubevirt-hyperconverged-operator.v2.6.10 channel: "stable" 1
- 1
- Using the
stable
channel ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
Create the required
Namespace
,OperatorGroup
, andSubscription
objects for OpenShift Virtualization by running the following command:$ oc apply -f <file name>.yaml
3.3.5. Deploying the OpenShift Virtualization Operator by using the CLI
You can deploy the OpenShift Virtualization Operator by using the oc
CLI.
Prerequisites
-
An active subscription to the OpenShift Virtualization catalog in the
openshift-cnv
namespace.
Procedure
Create a YAML file that contains the following manifest:
apiVersion: hco.kubevirt.io/v1beta1 kind: HyperConverged metadata: name: kubevirt-hyperconverged namespace: openshift-cnv spec:
Deploy the OpenShift Virtualization Operator by running the following command:
$ oc apply -f <file_name>.yaml
Verification
Ensure that OpenShift Virtualization deployed successfully by watching the
PHASE
of the cluster service version (CSV) in theopenshift-cnv
namespace. Run the following command:$ watch oc get csv -n openshift-cnv
The following output displays if deployment was successful:
Example output
NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v2.6.10 OpenShift Virtualization 2.6.10 Succeeded
3.3.6. Next steps
You might want to additionally configure the following components:
- The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace.
- The hostpath provisioner is a local storage provisioner designed for OpenShift Virtualization. If you want to configure local storage for virtual machines, you must enable the hostpath provisioner first.
3.4. Installing the virtctl client
The virtctl
client is a command-line utility for managing OpenShift Virtualization resources. It is available for Linux, macOS, and Windows distributions.
You can install the virtctl
client from the OpenShift Virtualization web console or by enabling the OpenShift Virtualization repository and installing the kubevirt-virtctl
package.
3.4.1. Installing the virtctl client from the web console
You can download the virtctl
client from the Red Hat Customer Portal, which is linked to in your OpenShift Virtualization web console in the Command Line Tools page.
Prerequisites
- You must have an activated OpenShift Container Platform subscription to access the download page on the Customer Portal.
Procedure
- Access the Customer Portal by clicking the icon, which is in the upper-right corner of the web console, and selecting Command Line Tools.
- Ensure you have the appropriate version for your cluster selected from the Version: list.
-
Download the
virtctl
client for your distribution. All downloads are intar.gz
format. Extract the tarball. The following CLI command extracts it into the same directory as the tarball and is applicable for all distributions:
$ tar -xvf <virtctl-version-distribution.arch>.tar.gz
For Linux and macOS:
Navigate the extracted folder hierachy and make the
virtctl
binary executable:$ chmod +x <virtctl-file-name>
Move the
virtctl
binary to a directory on your PATH.To check your path, run:
$ echo $PATH
For Windows users:
-
Navigate the extracted folder hierarchy and double-click the
virtctl
executable file to install the client.
-
Navigate the extracted folder hierarchy and double-click the
3.4.2. Enabling OpenShift Virtualization repositories
Red Hat offers OpenShift Virtualization repositories for both Red Hat Enterprise Linux 8 and Red Hat Enterprise Linux 7:
-
Red Hat Enterprise Linux 8 repository:
cnv-2.6-for-rhel-8-x86_64-rpms
-
Red Hat Enterprise Linux 7 repository:
rhel-7-server-cnv-2.6-rpms
The process for enabling the repository in subscription-manager
is the same in both platforms.
Procedure
Enable the appropriate OpenShift Virtualization repository for your system by running the following command:
# subscription-manager repos --enable <repository>
3.4.3. Installing the virtctl client
Install the virtctl
client from the kubevirt-virtctl
package.
Procedure
Install the
kubevirt-virtctl
package:# yum install kubevirt-virtctl
3.4.4. Additional resources
- Using the CLI tools for OpenShift Virtualization.
3.5. Uninstalling OpenShift Virtualization using the web console
You can uninstall OpenShift Virtualization by using the OpenShift Container Platform web console.
3.5.1. Prerequisites
- You must have OpenShift Virtualization 2.6 installed.
You must delete all virtual machines, virtual machine instances, and data volumes.
ImportantAttempting to uninstall OpenShift Virtualization without deleting these objects results in failure.
3.5.2. Deleting the OpenShift Virtualization Operator Deployment custom resource
To uninstall OpenShift Virtualization, you must first delete the OpenShift Virtualization Operator Deployment custom resource.
Prerequisites
- Create the OpenShift Virtualization Operator Deployment custom resource.
Procedure
-
From the OpenShift Container Platform web console, select
openshift-cnv
from the Projects list. - Navigate to the Operators → Installed Operators page.
- Click OpenShift Virtualization.
- Click the OpenShift Virtualization Operator Deployment tab.
- Click the Options menu in the row containing the kubevirt-hyperconverged custom resource. In the expanded menu, click Delete HyperConverged Cluster.
- Click Delete in the confirmation window.
- Navigate to the Workloads → Pods page to verify that only the Operator pods are running.
Open a terminal window and clean up the remaining resources by running the following command:
$ oc delete apiservices v1alpha3.subresources.kubevirt.io -n openshift-cnv
3.5.3. Deleting the OpenShift Virtualization catalog subscription
To finish uninstalling OpenShift Virtualization, delete the OpenShift Virtualization catalog subscription.
Prerequisites
- An active subscription to the OpenShift Virtualization catalog
Procedure
- Navigate to the Operators → OperatorHub page.
- Search for OpenShift Virtualization and then select it.
- Click Uninstall.
You can now delete the openshift-cnv
namespace.
3.5.4. Deleting a namespace using the web console
You can delete a namespace by using the OpenShift Container Platform web console.
If you do not have permissions to delete the namespace, the Delete Namespace option is not available.
Procedure
- Navigate to Administration → Namespaces.
- Locate the namespace that you want to delete in the list of namespaces.
- On the far right side of the namespace listing, select Delete Namespace from the Options menu .
- When the Delete Namespace pane opens, enter the name of the namespace that you want to delete in the field.
- Click Delete.
3.6. Uninstalling OpenShift Virtualization using the CLI
You can uninstall OpenShift Virtualization by using the OpenShift Container Platform CLI.
3.6.1. Prerequisites
- You must have OpenShift Virtualization 2.6 installed.
You must delete all virtual machines, virtual machine instances, and data volumes.
ImportantAttempting to uninstall OpenShift Virtualization without deleting these objects results in failure.
3.6.2. Deleting OpenShift Virtualization
You can delete OpenShift Virtualization by using the CLI.
Prerequisites
-
Install the OpenShift CLI (
oc
). -
Access to a OpenShift Virtualization cluster using an account with
cluster-admin
permissions.
When you delete the subscription of the OpenShift Virtualization operator in the OLM by using the CLI, the ClusterServiceVersion
(CSV) object is not deleted from the cluster. To completely uninstall OpenShift Virtualization, you must explicitly delete the CSV.
Procedure
Delete the
HyperConverged
custom resource:$ oc delete HyperConverged kubevirt-hyperconverged -n openshift-cnv
Delete the subscription of the OpenShift Virtualization operator in the Operator Lifecycle Manager (OLM):
$ oc delete subscription kubevirt-hyperconverged -n openshift-cnv
Set the cluster service version (CSV) name for OpenShift Virtualization as an environment variable:
$ CSV_NAME=$(oc get csv -n openshift-cnv -o=custom-columns=:metadata.name)
Delete the CSV from the OpenShift Virtualization cluster by specifying the CSV name from the previous step:
$ oc delete csv ${CSV_NAME} -n openshift-cnv
OpenShift Virtualization is uninstalled when a confirmation message indicates that the CSV was deleted successfully:
Example output
clusterserviceversion.operators.coreos.com "kubevirt-hyperconverged-operator.v2.6.10" deleted
Chapter 4. Upgrading OpenShift Virtualization
You can manually upgrade to the next minor version of OpenShift Virtualization and monitor the status of an update by using the web console.
4.1. About upgrading OpenShift Virtualization
4.1.1. How OpenShift Virtualization upgrades work
- You can upgrade to the next minor version of OpenShift Virtualization by using the OpenShift Container Platform web console to change the channel of your Operator subscription.
- You can enable automatic z-stream updates during OpenShift Virtualization installation.
- Updates are delivered via the Marketplace Operator, which is deployed during OpenShift Container Platform installation. The Marketplace Operator makes external Operators available to your cluster.
- The amount of time an update takes to complete depends on your network connection. Most automatic updates complete within fifteen minutes.
4.1.2. How OpenShift Virtualization upgrades affect your cluster
Upgrading does not interrupt virtual machine workloads.
Virtual machine pods are not restarted or migrated during an upgrade. If you need to update the
virt-launcher
pod, you must restart or live migrate the virtual machine.NoteEach virtual machine has a
virt-launcher
pod that runs the virtual machine instance. Thevirt-launcher
pod runs an instance oflibvirt
, which is used to manage the virtual machine process.
- Upgrading does not interrupt network connections.
Data volumes and their associated persistent volume claims are preserved during upgrade.
ImportantIf you have virtual machines running that cannot be live migrated, they might block an OpenShift Container Platform cluster upgrade. This includes virtual machines that use hostpath provisioner storage or SR-IOV network interfaces.
As a workaround, you can reconfigure the virtual machines so that they can be powered off automatically during a cluster upgrade. Remove the
evictionStrategy: LiveMigrate
field and set therunStrategy
field toAlways
.
4.2. Upgrading OpenShift Virtualization to the next minor version
You can manually upgrade OpenShift Virtualization to the next minor version by using the OpenShift Container Platform web console to change the channel of your Operator subscription.
Prerequisites
-
Log in to the cluster as a user with the
cluster-admin
role.
Procedure
- Access the OpenShift Container Platform web console and navigate to Operators → Installed Operators.
- Click OpenShift Virtualization to open the Operator Details page.
- Click the Subscription tab to open the Subscription Overview page.
- In the Channel pane, click the pencil icon on the right side of the version number to open the Change Subscription Update Channel window.
- Select stable. This ensures that you install the version of OpenShift Virtualization that is compatible with your OpenShift Container Platform version.
- Click Save.
Check the status of the upgrade by navigating to Operators → Installed Operators. You can also check the status by running the following
oc
command:$ oc get csv -n openshift-cnv
4.3. Monitoring upgrade status
The best way to monitor OpenShift Virtualization upgrade status is to watch the cluster service version (CSV) PHASE
. You can also monitor the CSV conditions in the web console or by running the command provided here.
The PHASE
and conditions values are approximations that are based on available information.
Prerequisites
-
Log in to the cluster as a user with the
cluster-admin
role. -
Install the OpenShift CLI (
oc
).
Procedure
Run the following command:
$ oc get csv
Review the output, checking the
PHASE
field. For example:Example output
VERSION REPLACES PHASE 2.5.0 kubevirt-hyperconverged-operator.v2.4.3 Installing 2.4.3 Replacing
Optional: Monitor the aggregated status of all OpenShift Virtualization component conditions by running the following command:
$ oc get hco -n openshift-cnv kubevirt-hyperconverged \ -o=jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'
A successful upgrade results in the following output:
Example output
ReconcileComplete True Reconcile completed successfully Available True Reconcile completed successfully Progressing False Reconcile completed successfully Degraded False Reconcile completed successfully Upgradeable True Reconcile completed successfully
4.4. Additional resources
Chapter 5. Additional security privileges granted for kubevirt-controller and virt-launcher
The kubevirt-controller
and virt-launcher pods are granted some SELinux policies and Security Context Constraints privileges that are in addition to typical pod owners. These privileges enable virtual machines to use OpenShift Virtualization features.
5.1. Extended SELinux policies for virt-launcher pods
The container_t
SELinux policy for virt-launcher pods is extended with the following rules:
-
allow process self (tun_socket (relabelfrom relabelto attach_queue))
-
allow process sysfs_t (file (write))
-
allow process hugetlbfs_t (dir (add_name create write remove_name rmdir setattr))
-
allow process hugetlbfs_t (file (create unlink))
These rules enable the following virtualization features:
- Relabel and attach queues to its own TUN sockets, which is required to support network multi-queue. Multi-queue enables network performance to scale as the number of available vCPUs increases.
-
Allows virt-launcher pods to write information to sysfs (
/sys
) files, which is required to enable Single Root I/O Virtualization (SR-IOV). -
Read/write
hugetlbfs
entries, which is required to support huge pages. Huge pages are a method of managing large amounts of memory by increasing the memory page size.
5.2. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account
Security context constraints (SCCs) control permissions for pods. These permissions include actions that a pod, a collection of containers, can perform and what resources it can access. You can use SCCs to define a set of conditions that a pod must run with to be accepted into the system.
The kubevirt-controller
is a cluster controller that creates the virt-launcher pods for virtual machines in the cluster. These virt-launcher pods are granted permissions by the kubevirt-controller
service account.
5.2.1. Additional SCCs granted to the kubevirt-controller service account
The kubevirt-controller
service account is granted additional SCCs and Linux capabilities so that it can create virt-launcher pods with the appropriate permissions. These extended permissions allow virtual machines to take advantage of OpenShift Virtualization features that are beyond the scope of typical pods.
The kubevirt-controller
service account is granted the following SCCs:
-
scc.AllowHostDirVolumePlugin = true
This allows virtual machines to use the hostpath volume plug-in. -
scc.AllowPrivilegedContainer = false
This ensures the virt-launcher pod is not run as a privileged container. -
scc.AllowedCapabilities = []corev1.Capability{"NET_ADMIN", "NET_RAW", "SYS_NICE"}
This provides the following additional Linux capabilitiesNET_ADMIN
,NET_RAW
, andSYS_NICE
.
5.2.2. Viewing the SCC and RBAC definitions for the kubevirt-controller
You can view the SecurityContextConstraints
definition for the kubevirt-controller
by using the oc
tool:
$ oc get scc kubevirt-controller -o yaml
You can view the RBAC definition for the kubevirt-controller
clusterrole by using the oc
tool:
$ oc get clusterrole kubevirt-controller -o yaml
5.3. Additional resources
- The Red Hat Enterprise Linux Virtualization Tuning and Optimization Guide has more information on network multi-queue and huge pages.
-
The
capabilities
man page has more information on the Linux capabilities. -
The
sysfs(5)
man page has more information on sysfs. - The OpenShift Container Platform Authentication guide has more information on Security Context Constraints.
Chapter 6. Using the CLI tools
The two primary CLI tools used for managing resources in the cluster are:
-
The OpenShift Virtualization
virtctl
client -
The OpenShift Container Platform
oc
client
6.1. Prerequisites
-
You must install the
virtctl
client.
6.2. Virtctl client commands
The virtctl
client is a command-line utility for managing OpenShift Virtualization resources. The following table contains the virtctl
commands used throughout the OpenShift Virtualization documentation.
To view a list of options that you can use with a command, run it with the -h
or --help
flag. For example:
$ virtctl image-upload -h
Command | Description |
---|---|
| Start a virtual machine. |
| Stop a virtual machine. |
| Pause a virtual machine or virtual machine instance. The machine state is kept in memory. |
| Unpause a virtual machine or virtual machine instance. |
| Migrate a virtual machine. |
| Restart a virtual machine. |
| Create a service that forwards a designated port of a virtual machine or virtual machine instance and expose the service on the specified port of the node. |
| Connect to a serial console of a virtual machine instance. |
| Open a VNC connection to a virtual machine instance. |
| Upload a virtual machine image to a data volume that already exists. |
| Upload a virtual machine image to a new data volume. |
| Display the client and server version information. |
|
Display a descriptive list of |
| Return a full list of file systems available on the guest machine. |
| Return guest agent information about the operating system. |
| Return a full list of logged-in users on the guest machine. |
6.3. OpenShift Container Platform client commands
The OpenShift Container Platform oc
client is a command-line utility for managing OpenShift Container Platform resources, including the VirtualMachine
(vm
) and VirtualMachineInstance
(vmi
) object types.
You can use the -n <namespace>
flag to specify a different project.
Command | Description |
---|---|
|
Log in to the OpenShift Container Platform cluster as |
| Display a list of objects for the specified object type in the current project. |
| Display details of the specific resource in the current project. |
| Create a resource in the current project from a file name or from stdin. |
| Edit a resource in the current project. |
| Delete a resource in the current project. |
For more comprehensive information on oc
client commands, see the OpenShift Container Platform CLI tools documentation.
Chapter 7. Virtual machines
7.1. Creating virtual machines
Use one of these procedures to create a virtual machine:
- Quick Start guided tour
- Running the wizard
- Pasting a pre-configured YAML file with the virtual machine wizard
- Using the CLI
- Importing a VMware virtual machine or template with the virtual machine wizard
Do not create virtual machines in openshift-*
namespaces. Instead, create a new namespace or use an existing namespace without the openshift
prefix.
When you create virtual machines from the web console, select a virtual machine template that is configured with a boot source. Virtual machine templates with a boot source are labeled as Available boot source or they display a customized label text. Using templates with an available boot source expedites the process of creating virtual machines.
Templates without a boot source are labeled as Boot source required. You can use these templates if you complete the steps for adding a boot source to the virtual machine.
7.1.1. Using a Quick Start to create a virtual machine
The web console provides Quick Starts with instructional guided tours for creating virtual machines. You can access the Quick Starts catalog by selecting the Help menu in the Administrator perspective to view the Quick Starts catalog. When you click on a Quick Start tile and begin the tour, the system guides you through the process.
Tasks in a Quick Start begin with selecting a Red Hat template. Then, you can add a boot source and import the operating system image. Finally, you can save the custom template and use it to create a virtual machine.
Prerequisites
- Access to the website where you can download the URL link for the operating system image.
Procedure
- In the web console, select Quick Starts from the Help menu.
- Click on a tile in the Quick Starts catalog. For example: Creating a Red Hat Linux Enterprise Linux virtual machine.
- Follow the instructions in the guided tour and complete the tasks for importing an operating system image and creating a virtual machine. The Virtual Machines tab displays the virtual machine.
7.1.2. Running the virtual machine wizard to create a virtual machine
The web console features a wizard that guides you through the process of selecting a virtual machine template and creating a virtual machine. Red Hat virtual machine templates are preconfigured with an operating system image, default settings for the operating system, flavor (CPU and memory), and workload type (server). When templates are configured with a boot source, they are labeled with a customized label text or the default label text Available boot source. These templates are then ready to be used for creating virtual machines.
You can select a template from the list of preconfigured templates, review the settings, and create a virtual machine in the Create virtual machine from template wizard. If you choose to customize your virtual machine, the wizard guides you through the General, Networking, Storage, Advanced, and Review steps. All required fields displayed by the wizard are marked by a *.
Create network interface controllers (NICs) and storage disks later and attach them to virtual machines.
Procedure
- Click Workloads → Virtualization from the side menu.
- From the Virtual Machines tab or the Templates tab, click Create and select Virtual Machine with Wizard.
- Select a template that is configured with a boot source.
- Click Next to go to the Review and create step.
- Clear the Start this virtual machine after creation checkbox if you do not want to start the virtual machine now.
- Click Create virtual machine and exit the wizard or continue with the wizard to customize the virtual machine.
Click Customize virtual machine to go to the General step.
- Optional: Edit the Name field to specify a custom name for the virtual machine.
- Optional: In the Description field, add a description.
Click Next to go to the Networking step. A
nic0
NIC is attached by default.- Optional: Click Add Network Interface to create additional NICs.
- Optional: You can remove any or all NICs by clicking the Options menu and selecting Delete. A virtual machine does not need a NIC attached to be created. You can create NICs after the virtual machine has been created.
Click Next to go to the Storage step.
- Optional: Click Add Disk to create additional disks. These disks can be removed by clicking the Options menu and selecting Delete.
- Optional: Click the Options menu to edit the disk and save your changes.
- Click Next to go to the Advanced step and review the details for Cloud-init.
- Click Next to go to the Review step and review the settings for the virtual machine.
- Click Create Virtual Machine.
Click See virtual machine details to view the Overview for this virtual machine.
The virtual machine is listed in the Virtual Machines tab.
Refer to the virtual machine wizard fields section when running the web console wizard.
7.1.2.1. Virtual machine wizard fields
Name | Parameter | Description |
---|---|---|
Name |
The name can contain lowercase letters ( | |
Description | Optional description field. | |
Operating System | The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. | |
Boot Source | Import via URL (creates PVC) | Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. |
Clone existing PVC (creates PVC) | Select an existent persistent volume claim available on the cluster and clone it. | |
Import via Registry (creates PVC) |
Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: | |
PXE (network boot - adds network interface) | Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. | |
Persistent Volume Claim project | Project name that you want to use for cloning the PVC. | |
Persistent Volume Claim name | PVC name that should apply to this virtual machine template if you are cloning an existing PVC. | |
Mount this as a CD-ROM boot source | A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. | |
Flavor | Tiny, Small, Medium, Large, Custom | Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. |
Workload Type | Desktop | A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. |
Server | Balances performance and it is compatible with a wide range of server workloads. | |
High-Performance | A virtual machine configuration that is optimized for high-performance workloads. | |
Start this virtual machine after creation. | This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. |
7.1.2.2. Networking fields
Name | Description |
---|---|
Name | Name for the network interface controller. |
Model | Indicates the model of the network interface controller. Supported values are e1000e and virtio. |
Network | List of available network attachment definitions. |
Type |
List of available binding methods. For the default pod network, |
MAC Address | MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. |
7.1.2.3. Storage fields
Name | Selection | Description |
---|---|---|
Source | Blank (creates PVC) | Create an empty disk. |
Import via URL (creates PVC) | Import content via URL (HTTP or HTTPS endpoint). | |
Use an existing PVC | Use a PVC that is already available in the cluster. | |
Clone existing PVC (creates PVC) | Select an existing PVC available in the cluster and clone it. | |
Import via Registry (creates PVC) | Import content via container registry. | |
Container (ephemeral) | Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. | |
Name |
Name of the disk. The name can contain lowercase letters ( | |
Size | Size of the disk in GiB. | |
Type | Type of disk. Example: Disk or CD-ROM | |
Interface | Type of disk device. Supported interfaces are virtIO, SATA, and SCSI. | |
Storage Class | The storage class that is used to create the disk. | |
Advanced → Volume Mode | Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem. | |
Advanced → Access Mode | Access mode of the persistent volume. Supported access modes are Single User (RWO), Shared Access (RWX), and Read Only (ROX). |
Advanced storage settings
The following advanced storage settings are available for Blank, Import via URL, and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults
config map.
Name | Parameter | Description |
---|---|---|
Volume Mode | Filesystem | Stores the virtual disk on a file system-based volume. |
Block |
Stores the virtual disk directly on the block volume. Only use | |
Access Mode | Single User (RWO) | The disk can be mounted as read/write by a single node. |
Shared Access (RWX) | The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. | |
Read Only (ROX) | The disk can be mounted as read-only by many nodes. |
7.1.2.4. Cloud-init fields
Name | Description |
---|---|
Hostname | Sets a specific hostname for the virtual machine. |
Authorized SSH Keys | The user’s public key that is copied to ~/.ssh/authorized_keys on the virtual machine. |
Custom script | Replaces other options with a field in which you paste a custom cloud-init script. |
For more information on the kubevirt-storage-class-defaults
config map, see Storage defaults for data volumes.
7.1.2.5. Pasting in a pre-configured YAML file to create a virtual machine
Create a virtual machine by writing or pasting a YAML configuration file. A valid example
virtual machine configuration is provided by default whenever you open the YAML edit screen.
If your YAML configuration is invalid when you click Create, an error message indicates the parameter in which the error occurs. Only one error is shown at a time.
Navigating away from the YAML screen while editing cancels any changes to the configuration you have made.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Click Create and select Virtual Machine With YAML.
Write or paste your virtual machine configuration in the editable window.
-
Alternatively, use the
example
virtual machine provided by default in the YAML screen.
-
Alternatively, use the
- Optional: Click Download to download the YAML configuration file in its present state.
- Click Create to create the virtual machine.
The virtual machine is listed in the Virtual Machines tab.
7.1.3. Using the CLI to create a virtual machine
You can create a virtual machine from a virtualMachine
manifest.
Procedure
Edit the
VirtualMachine
manifest for your VM. For example, the following manifest configures a Red Hat Enterprise Linux (RHEL) VM:Example 7.1. Example manifest for a RHEL VM
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: labels: app: <vm_name> 1 name: <vm_name> spec: dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <vm_name> spec: sourceRef: kind: DataSource name: rhel9 namespace: openshift-virtualization-os-images storage: resources: requests: storage: 30Gi running: false template: metadata: labels: kubevirt.io/domain: <vm_name> spec: domain: cpu: cores: 1 sockets: 2 threads: 1 devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default rng: {} features: smm: enabled: true firmware: bootloader: efi: {} resources: requests: memory: 8Gi evictionStrategy: LiveMigrate networks: - name: default pod: {} volumes: - dataVolume: name: <vm_name> name: rootdisk - cloudInitNoCloud: userData: |- #cloud-config user: cloud-user password: '<password>' 2 chpasswd: { expire: False } name: cloudinitdisk
Create a virtual machine by using the manifest file:
$ oc create -f <vm_manifest_file>.yaml
Optional: Start the virtual machine:
$ virtctl start <vm_name>
7.1.4. Virtual machine storage volume types
Storage volume type | Description |
---|---|
ephemeral | A local copy-on-write (COW) image that uses a network volume as a read-only backing store. The backing volume must be a PersistentVolumeClaim. The ephemeral image is created when the virtual machine starts and stores all writes locally. The ephemeral image is discarded when the virtual machine is stopped, restarted, or deleted. The backing volume (PVC) is not mutated in any way. |
persistentVolumeClaim | Attaches an available PV to a virtual machine. Attaching a PV allows for the virtual machine data to persist between sessions. Importing an existing virtual machine disk into a PVC by using CDI and attaching the PVC to a virtual machine instance is the recommended method for importing existing virtual machines into OpenShift Container Platform. There are some requirements for the disk to be used within a PVC. |
dataVolume |
Data volumes build on the
Specify |
cloudInitNoCloud | Attaches a disk that contains the referenced cloud-init NoCloud data source, providing user data and metadata to the virtual machine. A cloud-init installation is required inside the virtual machine disk. |
containerDisk | References an image, such as a virtual machine disk, that is stored in the container image registry. The image is pulled from the registry and attached to the virtual machine as a disk when the virtual machine is launched.
A Only RAW and QCOW2 formats are supported disk types for the container image registry. QCOW2 is recommended for reduced image size. Note
A |
emptyDisk | Creates an additional sparse QCOW2 disk that is tied to the life-cycle of the virtual machine interface. The data survives guest-initiated reboots in the virtual machine but is discarded when the virtual machine stops or is restarted from the web console. The empty disk is used to store application dependencies and data that otherwise exceeds the limited temporary file system of an ephemeral disk. The disk capacity size must also be provided. |
7.1.5. About RunStrategies for virtual machines
A RunStrategy
for virtual machines determines a virtual machine instance’s (VMI) behavior, depending on a series of conditions. The spec.runStrategy
setting exists in the virtual machine configuration process as an alternative to the spec.running
setting. The spec.runStrategy
setting allows greater flexibility for how VMIs are created and managed, in contrast to the spec.running
setting with only true
or false
responses. However, the two settings are mutually exclusive. Only either spec.running
or spec.runStrategy
can be used. An error occurs if both are used.
There are four defined RunStrategies.
Always
-
A VMI is always present when a virtual machine is created. A new VMI is created if the original stops for any reason, which is the same behavior as
spec.running: true
. RerunOnFailure
- A VMI is re-created if the previous instance fails due to an error. The instance is not re-created if the virtual machine stops successfully, such as when it shuts down.
Manual
-
The
start
,stop
, andrestart
virtctl client commands can be used to control the VMI’s state and existence. Halted
-
No VMI is present when a virtual machine is created, which is the same behavior as
spec.running: false
.
Different combinations of the start
, stop
and restart
virtctl commands affect which RunStrategy
is used.
The following table follows a VM’s transition from different states. The first column shows the VM’s initial RunStrategy
. Each additional column shows a virtctl command and the new RunStrategy
after that command is run.
Initial RunStrategy | start | stop | restart |
---|---|---|---|
Always | - | Halted | Always |
RerunOnFailure | - | Halted | RerunOnFailure |
Manual | Manual | Manual | Manual |
Halted | Always | - | - |
In OpenShift Virtualization clusters installed using installer-provisioned infrastructure, when a node fails the MachineHealthCheck and becomes unavailable to the cluster, VMs with a RunStrategy of Always
or RerunOnFailure
are rescheduled on a new node.
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
spec:
RunStrategy: Always 1
template:
...
- 1
- The VMI’s current
RunStrategy
setting.
7.1.6. Additional resources
The
VirtualMachineSpec
definition in the KubeVirt v0.36.5 API Reference provides broader context for the parameters and hierarchy of the virtual machine specification.NoteThe KubeVirt API Reference is the upstream project reference and might contain parameters that are not supported in OpenShift Virtualization.
-
Prepare a container disk before adding it to a virtual machine as a
containerDisk
volume. - See Deploying machine health checks for further details on deploying and enabling machine health checks.
- See Installer-provisioned infrastructure overview for further details on installer-provisioned infrastructure.
- See Configuring the SR-IOV Network Operator for further details on the SR-IOV Network Operator.
7.2. Editing virtual machines
You can update a virtual machine configuration using either the YAML editor in the web console or the OpenShift client on the command line. You can also update a subset of the parameters in the Virtual Machine Details screen.
7.2.1. Editing a virtual machine in the web console
Edit select values of a virtual machine in the web console by clicking the pencil icon next to the relevant field. Other values can be edited using the CLI.
Labels and annotations are editable for both preconfigured Red Hat templates and your custom virtual machine templates. All other values are editable only for custom virtual machine templates that users have created using the Red Hat templates or the Create Virtual Machine Template wizard.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine.
- Click the Details tab.
- Click the pencil icon to make a field editable.
- Make the relevant changes and click Save.
If the virtual machine is running, changes to Boot Order or Flavor will not take effect until you restart the virtual machine.
You can view pending changes by clicking View Pending Changes on the right side of the relevant field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
7.2.2. Editing a virtual machine YAML configuration using the web console
You can edit the YAML configuration of a virtual machine in the web console. Some parameters cannot be modified. If you click Save with an invalid configuration, an error message indicates the parameter that cannot be changed.
If you edit the YAML configuration while the virtual machine is running, changes will not take effect until you restart the virtual machine.
Navigating away from the YAML screen while editing cancels any changes to the configuration you have made.
Procedure
- Click Workloads → Virtualization from the side menu.
- Select a virtual machine.
- Click the YAML tab to display the editable configuration.
- Optional: You can click Download to download the YAML file locally in its current state.
- Edit the file and click Save.
A confirmation message shows that the modification has been successful and includes the updated version number for the object.
7.2.3. Editing a virtual machine YAML configuration using the CLI
Use this procedure to edit a virtual machine YAML configuration using the CLI.
Prerequisites
- You configured a virtual machine with a YAML object configuration file.
-
You installed the
oc
CLI.
Procedure
Run the following command to update the virtual machine configuration:
$ oc edit <object_type> <object_ID>
- Open the object configuration.
- Edit the YAML.
If you edit a running virtual machine, you need to do one of the following:
- Restart the virtual machine.
Run the following command for the new configuration to take effect:
$ oc apply <object_type> <object_ID>
7.2.4. Adding a virtual disk to a virtual machine
Use this procedure to add a virtual disk to a virtual machine.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Disks tab.
In the Add Disk window, specify the Source, Name, Size, Type, Interface, and Storage Class.
-
Optional: In the Advanced list, specify the Volume Mode and Access Mode for the virtual disk. If you do not specify these parameters, the system uses the default values from the
kubevirt-storage-class-defaults
config map.
-
Optional: In the Advanced list, specify the Volume Mode and Access Mode for the virtual disk. If you do not specify these parameters, the system uses the default values from the
- Click Add.
If the virtual machine is running, the new disk is in the pending restart state and will not be attached until you restart the virtual machine.
The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
For more information on the kubevirt-storage-class-defaults
config map, see Storage defaults for data volumes.
7.2.4.1. Storage fields
Name | Selection | Description |
---|---|---|
Source | Blank (creates PVC) | Create an empty disk. |
Import via URL (creates PVC) | Import content via URL (HTTP or HTTPS endpoint). | |
Use an existing PVC | Use a PVC that is already available in the cluster. | |
Clone existing PVC (creates PVC) | Select an existing PVC available in the cluster and clone it. | |
Import via Registry (creates PVC) | Import content via container registry. | |
Container (ephemeral) | Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. | |
Name |
Name of the disk. The name can contain lowercase letters ( | |
Size | Size of the disk in GiB. | |
Type | Type of disk. Example: Disk or CD-ROM | |
Interface | Type of disk device. Supported interfaces are virtIO, SATA, and SCSI. | |
Storage Class | The storage class that is used to create the disk. | |
Advanced → Volume Mode | Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem. | |
Advanced → Access Mode | Access mode of the persistent volume. Supported access modes are Single User (RWO), Shared Access (RWX), and Read Only (ROX). |
Advanced storage settings
The following advanced storage settings are available for Blank, Import via URL, and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults
config map.
Name | Parameter | Description |
---|---|---|
Volume Mode | Filesystem | Stores the virtual disk on a file system-based volume. |
Block |
Stores the virtual disk directly on the block volume. Only use | |
Access Mode | Single User (RWO) | The disk can be mounted as read/write by a single node. |
Shared Access (RWX) | The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. | |
Read Only (ROX) | The disk can be mounted as read-only by many nodes. |
7.2.5. Adding a network interface to a virtual machine
Use this procedure to add a network interface to a virtual machine.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Network Interfaces tab.
- Click Add Network Interface.
- In the Add Network Interface window, specify the Name, Model, Network, Type, and MAC Address of the network interface.
- Click Add.
If the virtual machine is running, the new network interface is in the pending restart state and changes will not take effect until you restart the virtual machine.
The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
7.2.5.1. Networking fields
Name | Description |
---|---|
Name | Name for the network interface controller. |
Model | Indicates the model of the network interface controller. Supported values are e1000e and virtio. |
Network | List of available network attachment definitions. |
Type |
List of available binding methods. For the default pod network, |
MAC Address | MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. |
7.2.6. Editing CD-ROMs for Virtual Machines
Use the following procedure to edit CD-ROMs for virtual machines.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Disks tab.
- Click the Options menu for the CD-ROM that you want to edit and select Edit.
- In the Edit CD-ROM window, edit the fields: Source, Persistent Volume Claim, Name, Type, and Interface.
- Click Save.
7.3. Editing boot order
You can update the values for a boot order list by using the web console or the CLI.
With Boot Order in the Virtual Machine Overview page, you can:
- Select a disk or network interface controller (NIC) and add it to the boot order list.
- Edit the order of the disks or NICs in the boot order list.
- Remove a disk or NIC from the boot order list, and return it back to the inventory of bootable sources.
7.3.1. Adding items to a boot order list in the web console
Add items to a boot order list by using the web console.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Details tab.
- Click the pencil icon that is located on the right side of Boot Order. If a YAML configuration does not exist, or if this is the first time that you are creating a boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file.
- Click Add Source and select a bootable disk or network interface controller (NIC) for the virtual machine.
- Add any additional disks or NICs to the boot order list.
- Click Save.
If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine.
You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
7.3.2. Editing a boot order list in the web console
Edit the boot order list in the web console.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Details tab.
- Click the pencil icon that is located on the right side of Boot Order.
Choose the appropriate method to move the item in the boot order list:
- If you do not use a screen reader, hover over the arrow icon next to the item that you want to move, drag the item up or down, and drop it in a location of your choice.
- If you use a screen reader, press the Up Arrow key or Down Arrow key to move the item in the boot order list. Then, press the Tab key to drop the item in a location of your choice.
- Click Save.
If the virtual machine is running, changes to the boot order list will not take effect until you restart the virtual machine.
You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
7.3.3. Editing a boot order list in the YAML configuration file
Edit the boot order list in a YAML configuration file by using the CLI.
Procedure
Open the YAML configuration file for the virtual machine by running the following command:
$ oc edit vm example
Edit the YAML file and modify the values for the boot order associated with a disk or network interface controller (NIC). For example:
disks: - bootOrder: 1 1 disk: bus: virtio name: containerdisk - disk: bus: virtio name: cloudinitdisk - cdrom: bus: virtio name: cd-drive-1 interfaces: - boot Order: 2 2 macAddress: '02:96:c4:00:00' masquerade: {} name: default
- Save the YAML file.
- Click reload the content to apply the updated boot order values from the YAML file to the boot order list in the web console.
7.3.4. Removing items from a boot order list in the web console
Remove items from a boot order list by using the web console.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Details tab.
- Click the pencil icon that is located on the right side of Boot Order.
- Click the Remove icon next to the item. The item is removed from the boot order list and saved in the list of available boot sources. If you remove all items from the boot order list, the following message displays: No resource selected. VM will attempt to boot from disks by order of appearance in YAML file.
If the virtual machine is running, changes to Boot Order will not take effect until you restart the virtual machine.
You can view pending changes by clicking View Pending Changes on the right side of the Boot Order field. The Pending Changes banner at the top of the page displays a list of all changes that will be applied when the virtual machine restarts.
7.4. Deleting virtual machines
You can delete a virtual machine from the web console or by using the oc
command line interface.
7.4.1. Deleting a virtual machine using the web console
Deleting a virtual machine permanently removes it from the cluster.
When you delete a virtual machine, the data volume it uses is automatically deleted.
Procedure
- In the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
Click the Options menu of the virtual machine that you want to delete and select Delete Virtual Machine.
- Alternatively, click the virtual machine name to open the Virtual Machine Overview screen and click Actions → Delete Virtual Machine.
- In the confirmation pop-up window, click Delete to permanently delete the virtual machine.
7.4.2. Deleting a virtual machine by using the CLI
You can delete a virtual machine by using the oc
command line interface (CLI). The oc
client enables you to perform actions on multiple virtual machines.
When you delete a virtual machine, the data volume it uses is automatically deleted.
Prerequisites
- Identify the name of the virtual machine that you want to delete.
Procedure
Delete the virtual machine by running the following command:
$ oc delete vm <vm_name>
NoteThis command only deletes objects that exist in the current project. Specify the
-n <project_name>
option if the object you want to delete is in a different project or namespace.
7.5. Managing virtual machine instances
If you have standalone virtual machine instances (VMIs) that were created independently outside of the OpenShift Virtualization environment, you can manage them by using the web console or the command-line interface (CLI).
7.5.1. About virtual machine instances
A virtual machine instance (VMI) is a representation of a running virtual machine (VM). When a VMI is owned by a VM or by another object, you manage it through its owner in the web console or by using the oc
command-line interface (CLI).
A standalone VMI is created and started independently with a script, through automation, or by using other methods in the CLI. In your environment, you might have standalone VMIs that were developed and started outside of the OpenShift Virtualization environment. You can continue to manage those standalone VMIs by using the CLI. You can also use the web console for specific tasks associated with standalone VMIs:
- List standalone VMIs and their details.
- Edit labels and annotations for a standalone VMI.
- Delete a standalone VMI.
When you delete a VM, the associated VMI is automatically deleted. You delete a standalone VMI directly because it is not owned by VMs or other objects.
Before you uninstall OpenShift Virtualization, list and view the standalone VMIs by using the CLI or the web console. Then, delete any outstanding VMIs.
7.5.2. Listing all virtual machine instances using the CLI
You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc
command-line interface (CLI).
Procedure
List all VMIs by running the following command:
$ oc get vmis
7.5.3. Listing standalone virtual machine instances using the web console
Using the web console, you can list and view standalone virtual machine instances (VMIs) in your cluster that are not owned by virtual machines (VMs).
VMIs that are owned by VMs or other objects are not displayed in the web console. The web console displays only standalone VMIs. If you want to list all VMIs in your cluster, you must use the CLI.
Procedure
- Click Workloads → Virtualization from the side menu. A list of VMs and standalone VMIs displays. You can identify standalone VMIs by the dark colored badges that display next to the virtual machine instance names.
7.5.4. Editing a standalone virtual machine instance using the web console
You can edit annotations and labels for a standalone virtual machine instance (VMI) using the web console. Other items displayed in the Details page for a standalone VMI are not editable.
Procedure
- Click Workloads → Virtualization from the side menu. A list of virtual machines (VMs) and standalone VMIs displays.
- Click the name of a standalone VMI to open the Virtual Machine Instance Overview screen.
- Click the Details tab.
- Click the pencil icon that is located on the right side of Annotations.
- Make the relevant changes and click Save.
To edit labels for a standalone VMI, click Actions and select Edit Labels. Make the relevant changes and click Save.
7.5.5. Deleting a standalone virtual machine instance using the CLI
You can delete a standalone virtual machine instance (VMI) by using the oc
command-line interface (CLI).
Prerequisites
- Identify the name of the VMI that you want to delete.
Procedure
Delete the VMI by running the following command:
$ oc delete vmi <vmi_name>
7.5.6. Deleting a standalone virtual machine instance using the web console
Delete a standalone virtual machine instance (VMI) from the web console.
Procedure
- In the OpenShift Container Platform web console, click Workloads → Virtualization from the side menu.
Click the ⋮ button of the standalone virtual machine instance (VMI) that you want to delete and select Delete Virtual Machine Instance.
- Alternatively, click the name of the standalone VMI. The Virtual Machine Instance Overview page displays.
- Select Actions → Delete Virtual Machine Instance.
- In the confirmation pop-up window, click Delete to permanently delete the standalone VMI.
7.6. Controlling virtual machine states
You can stop, start, restart, and unpause virtual machines from the web console.
To control virtual machines from the command-line interface (CLI), use the virtctl
client.
7.6.1. Starting a virtual machine
You can start a virtual machine from the web console.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Find the row that contains the virtual machine that you want to start.
Navigate to the appropriate menu for your use case:
To stay on this page, where you can perform actions on multiple virtual machines:
- Click the Options menu located at the far right end of the row.
To view comprehensive information about the selected virtual machine before you start it:
- Access the Virtual Machine Overview screen by clicking the name of the virtual machine.
- Click Actions.
- Select Start Virtual Machine.
- In the confirmation window, click Start to start the virtual machine.
When you start virtual machine that is provisioned from a URL
source for the first time, the virtual machine has a status of Importing while OpenShift Virtualization imports the container from the URL endpoint. Depending on the size of the image, this process might take several minutes.
7.6.2. Restarting a virtual machine
You can restart a running virtual machine from the web console.
To avoid errors, do not restart a virtual machine while it has a status of Importing.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Find the row that contains the virtual machine that you want to restart.
Navigate to the appropriate menu for your use case:
To stay on this page, where you can perform actions on multiple virtual machines:
- Click the Options menu located at the far right end of the row.
To view comprehensive information about the selected virtual machine before you restart it:
- Access the Virtual Machine Overview screen by clicking the name of the virtual machine.
- Click Actions.
- Select Restart Virtual Machine.
- In the confirmation window, click Restart to restart the virtual machine.
7.6.3. Stopping a virtual machine
You can stop a virtual machine from the web console.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Find the row that contains the virtual machine that you want to stop.
Navigate to the appropriate menu for your use case:
To stay on this page, where you can perform actions on multiple virtual machines:
- Click the Options menu located at the far right end of the row.
To view comprehensive information about the selected virtual machine before you stop it:
- Access the Virtual Machine Overview screen by clicking the name of the virtual machine.
- Click Actions.
- Select Stop Virtual Machine.
- In the confirmation window, click Stop to stop the virtual machine.
7.6.4. Unpausing a virtual machine
You can unpause a paused virtual machine from the web console.
Prerequisites
At least one of your virtual machines must have a status of Paused.
NoteYou can pause virtual machines by using the
virtctl
client.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Find the row that contains the virtual machine that you want to unpause.
Navigate to the appropriate menu for your use case:
To stay on this page, where you can perform actions on multiple virtual machines:
- In the Status column, click Paused.
To view comprehensive information about the selected virtual machine before you unpause it:
- Access the Virtual Machine Overview screen by clicking the name of the virtual machine.
- Click the pencil icon that is located on the right side of Status.
- In the confirmation window, click Unpause to unpause the virtual machine.
7.7. Accessing virtual machine consoles
OpenShift Virtualization provides different virtual machine consoles that you can use to accomplish different product tasks. You can access these consoles through the OpenShift Container Platform web console and by using CLI commands.
7.7.1. Accessing virtual machine consoles in the OpenShift Container Platform web console
You can connect to virtual machines by using the serial console or the VNC console in the OpenShift Container Platform web console.
You can connect to Windows virtual machines by using the desktop viewer console, which uses RDP (remote desktop protocol), in the OpenShift Container Platform web console.
7.7.1.1. Connecting to the serial console
Connect to the serial console of a running virtual machine from the Console tab in the Virtual Machine Overview screen of the web console.
Procedure
- In the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview page.
- Click Console. The VNC console opens by default.
- Select Disconnect before switching to ensure that only one console session is open at a time. Otherwise, the VNC console session remains active in the background.
- Click the VNC Console drop-down list and select Serial Console.
- Click Disconnect to end the console session.
- Optional: Open the serial console in a separate window by clicking Open Console in New Window.
7.7.1.2. Connecting to the VNC console
Connect to the VNC console of a running virtual machine from the Console tab in the Virtual Machine Overview screen of the web console.
Procedure
- In the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview page.
- Click the Console tab. The VNC console opens by default.
- Optional: Open the VNC console in a separate window by clicking Open Console in New Window.
- Optional: Send key combinations to the virtual machine by clicking Send Key.
7.7.1.3. Connecting to a Windows virtual machine with RDP
The desktop viewer console, which utilizes the Remote Desktop Protocol (RDP), provides a better console experience for connecting to Windows virtual machines.
To connect to a Windows virtual machine with RDP, download the console.rdp
file for the virtual machine from the Consoles tab in the Virtual Machine Details screen of the web console and supply it to your preferred RDP client.
Prerequisites
-
A running Windows virtual machine with the QEMU guest agent installed. The
qemu-guest-agent
is included in the VirtIO drivers. - A layer-2 NIC attached to the virtual machine.
- An RDP client installed on a machine on the same network as the Windows virtual machine.
Procedure
- In the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a Windows virtual machine to open the Virtual Machine Overview screen.
- Click the Console tab.
- In the Console list, select Desktop Viewer.
- In the Network Interface list, select the layer-2 NIC.
-
Click Launch Remote Desktop to download the
console.rdp
file. Open an RDP client and reference the
console.rdp
file. For example, using remmina:$ remmina --connect /path/to/console.rdp
- Enter the Administrator user name and password to connect to the Windows virtual machine.
7.7.1.4. Copying the SSH command from the web console
Copy the command to access a running virtual machine (VM) via SSH from the Actions list in the web console.
Procedure
- In the OpenShift Container Platform console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview page.
-
From the Actions list, select Copy SSH Command. You can now paste this command onto the OpenShift CLI (
oc
).
7.7.2. Accessing virtual machine consoles by using CLI commands
7.7.2.1. Accessing a virtual machine instance via SSH
You can use SSH to access a virtual machine (VM) after you expose port 22 on it.
The virtctl expose
command forwards a virtual machine instance (VMI) port to a node port and creates a service for enabled access. The following example creates the fedora-vm-ssh
service that forwards traffic from a specific port of cluster nodes to port 22 of the <fedora-vm>
virtual machine.
Prerequisites
- You must be in the same project as the VMI.
-
The VMI you want to access must be connected to the default pod network by using the
masquerade
binding method. - The VMI you want to access must be running.
-
Install the OpenShift CLI (
oc
).
Procedure
Run the following command to create the
fedora-vm-ssh
service:$ virtctl expose vm <fedora-vm> --port=22 --name=fedora-vm-ssh --type=NodePort 1
- 1
<fedora-vm>
is the name of the VM that you run thefedora-vm-ssh
service on.
Check the service to find out which port the service acquired:
$ oc get svc
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE fedora-vm-ssh NodePort 127.0.0.1 <none> 22:32551/TCP 6s
+ In this example, the service acquired the 32551
port.
Log in to the VMI via SSH. Use the
ipAddress
of any of the cluster nodes and the port that you found in the previous step:$ ssh username@<node_IP_address> -p 32551
7.7.2.2. Accessing the serial console of a virtual machine instance
The virtctl console
command opens a serial console to the specified virtual machine instance.
Prerequisites
-
The
virt-viewer
package must be installed. - The virtual machine instance you want to access must be running.
Procedure
Connect to the serial console with
virtctl
:$ virtctl console <VMI>
7.7.2.3. Accessing the graphical console of a virtual machine instances with VNC
The virtctl
client utility can use the remote-viewer
function to open a graphical console to a running virtual machine instance. This capability is included in the virt-viewer
package.
Prerequisites
-
The
virt-viewer
package must be installed. - The virtual machine instance you want to access must be running.
If you use virtctl
via SSH on a remote machine, you must forward the X session to your machine.
Procedure
Connect to the graphical interface with the
virtctl
utility:$ virtctl vnc <VMI>
If the command failed, try using the
-v
flag to collect troubleshooting information:$ virtctl vnc <VMI> -v 4
7.7.2.4. Connecting to a Windows virtual machine with an RDP console
The Remote Desktop Protocol (RDP) provides a better console experience for connecting to Windows virtual machines.
To connect to a Windows virtual machine with RDP, specify the IP address of the attached L2 NIC to your RDP client.
Prerequisites
-
A running Windows virtual machine with the QEMU guest agent installed. The
qemu-guest-agent
is included in the VirtIO drivers. - A layer 2 NIC attached to the virtual machine.
- An RDP client installed on a machine on the same network as the Windows virtual machine.
Procedure
Log in to the OpenShift Virtualization cluster through the
oc
CLI tool as a user with an access token.$ oc login -u <user> https://<cluster.example.com>:8443
Use
oc describe vmi
to display the configuration of the running Windows virtual machine.$ oc describe vmi <windows-vmi-name>
Example output
... spec: networks: - name: default pod: {} - multus: networkName: cnv-bridge name: bridge-net ... status: interfaces: - interfaceName: eth0 ipAddress: 198.51.100.0/24 ipAddresses: 198.51.100.0/24 mac: a0:36:9f:0f:b1:70 name: default - interfaceName: eth1 ipAddress: 192.0.2.0/24 ipAddresses: 192.0.2.0/24 2001:db8::/32 mac: 00:17:a4:77:77:25 name: bridge-net ...
-
Identify and copy the IP address of the layer 2 network interface. This is
192.0.2.0
in the above example, or2001:db8::
if you prefer IPv6. - Open an RDP client and use the IP address copied in the previous step for the connection.
- Enter the Administrator user name and password to connect to the Windows virtual machine.
7.8. Triggering virtual machine failover by resolving a failed node
If a node fails and machine health checks are not deployed on your cluster, virtual machines (VMs) with RunStrategy: Always
configured are not automatically relocated to healthy nodes. To trigger VM failover, you must manually delete the Node
object.
If you installed your cluster by using installer-provisioned infrastructure and you properly configured machine health checks:
- Failed nodes are automatically recycled.
-
Virtual machines with
RunStrategy
set toAlways
orRerunOnFailure
are automatically scheduled on healthy nodes.
7.8.1. Prerequisites
-
A node where a virtual machine was running has the
NotReady
condition. -
The virtual machine that was running on the failed node has
RunStrategy
set toAlways
. -
You have installed the OpenShift CLI (
oc
).
7.8.2. Deleting nodes from a bare metal cluster
When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods.
Procedure
Delete a node from an OpenShift Container Platform cluster running on bare metal by completing the following steps:
Mark the node as unschedulable:
$ oc adm cordon <node_name>
Drain all pods on the node:
$ oc adm drain <node_name> --force=true
This step might fail if the node is offline or unresponsive. Even if the node does not respond, it might still be running a workload that writes to shared storage. To avoid data corruption, power down the physical hardware before you proceed.
Delete the node from the cluster:
$ oc delete node <node_name>
Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node.
- If you powered down the physical hardware, turn it back on so that the node can rejoin the cluster.
7.8.3. Verifying virtual machine failover
After all resources are terminated on the unhealthy node, a new virtual machine instance (VMI) is automatically created on a healthy node for each relocated VM. To confirm that the VMI was created, view all VMIs by using the oc
CLI.
7.8.3.1. Listing all virtual machine instances using the CLI
You can list all virtual machine instances (VMIs) in your cluster, including standalone VMIs and those owned by virtual machines, by using the oc
command-line interface (CLI).
Procedure
List all VMIs by running the following command:
$ oc get vmis
7.9. Installing the QEMU guest agent on virtual machines
The QEMU guest agent is a daemon that runs on the virtual machine and passes information to the host about the virtual machine, users, file systems, and secondary networks.
7.9.1. Installing QEMU guest agent on a Linux virtual machine
The qemu-guest-agent
is widely available and available by default in Red Hat virtual machines. Install the agent and start the service
Procedure
- Access the virtual machine command line through one of the consoles or by SSH.
Install the QEMU guest agent on the virtual machine:
$ yum install -y qemu-guest-agent
Ensure the service is persistent and start it:
$ systemctl enable --now qemu-guest-agent
You can also install and start the QEMU guest agent by using the custom script field in the cloud-init section of the wizard when creating either virtual machines or virtual machines templates in the web console.
7.9.2. Installing QEMU guest agent on a Windows virtual machine
For Windows virtual machines, the QEMU guest agent is included in the VirtIO drivers, which can be installed using one of the following procedures:
7.9.2.1. Installing VirtIO drivers on an existing Windows virtual machine
Install the VirtIO drivers from the attached SATA CD drive to an existing Windows virtual machine.
This procedure uses a generic approach to adding drivers to Windows. The process might differ slightly between versions of Windows. Refer to the installation documentation for your version of Windows for specific installation steps.
Procedure
- Start the virtual machine and connect to a graphical console.
- Log in to a Windows user session.
Open Device Manager and expand Other devices to list any Unknown device.
-
Open the
Device Properties
to identify the unknown device. Right-click the device and select Properties. - Click the Details tab and select Hardware Ids in the Property list.
- Compare the Value for the Hardware Ids with the supported VirtIO drivers.
-
Open the
- Right-click the device and select Update Driver Software.
- Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
- Click Next to install the driver.
- Repeat this process for all the necessary VirtIO drivers.
- After the driver installs, click Close to close the window.
- Reboot the virtual machine to complete the driver installation.
7.9.2.2. Installing VirtIO drivers during Windows installation
Install the VirtIO drivers from the attached SATA CD driver during Windows installation.
This procedure uses a generic approach to the Windows installation and the installation method might differ between versions of Windows. Refer to the documentation for the version of Windows that you are installing.
Procedure
- Start the virtual machine and connect to a graphical console.
- Begin the Windows installation process.
- Select the Advanced installation.
-
The storage destination will not be recognized until the driver is loaded. Click
Load driver
. - The drivers are attached as a SATA CD drive. Click OK and browse the CD drive for the storage driver to load. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
- Repeat the previous two steps for all required drivers.
- Complete the Windows installation.
7.10. Viewing the QEMU guest agent information for virtual machines
When the QEMU guest agent runs on the virtual machine, you can use the web console to view information about the virtual machine, users, file systems, and secondary networks.
7.10.1. Prerequisites
- Install the QEMU guest agent on the virtual machine.
7.10.2. About the QEMU guest agent information in the web console
When the QEMU guest agent is installed, the Details pane within the Virtual Machine Overview tab and the Details tab display information about the hostname, operating system, time zone, and logged in users.
The Virtual Machine Overview shows information about the guest operating system installed on the virtual machine. The Details tab displays a table with information for logged in users. The Disks tab displays a table with information for file systems.
If the QEMU guest agent is not installed, the Virtual Machine Overview tab and the Details tab display information about the operating system that was specified when the virtual machine was created.
7.10.3. Viewing the QEMU guest agent information in the web console
You can use the web console to view information for virtual machines that is passed by the QEMU guest agent to the host.
Procedure
- Click Workloads → Virtual Machines from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine name to open the Virtual Machine Overview screen and view the Details pane.
- Click Logged in users to view the Details tab that shows information for users.
- Click the Disks tab to view information about the file systems.
7.11. Managing config maps, secrets, and service accounts in virtual machines
You can use secrets, config maps, and service accounts to pass configuration data to virtual machines. For example, you can:
- Give a virtual machine access to a service that requires credentials by adding a secret to the virtual machine.
- Store non-confidential configuration data in a config map so that a pod or another object can consume the data.
- Allow a component to access the API server by associating a service account with that component.
OpenShift Virtualization exposes secrets, config maps, and service accounts as virtual machine disks so that you can use them across platforms without additional overhead.
7.11.1. Adding a secret, config map, or service account to a virtual machine
Add a secret, config map, or service account to a virtual machine by using the OpenShift Container Platform web console.
Prerequisites
- The secret, config map, or service account that you want to add must exist in the same namespace as the target virtual machine.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Environment tab.
- Click Select a resource and select a secret, config map, or service account from the list. A six character serial number is automatically generated for the selected resource.
- Click Save.
- Optional. Add another object by clicking Add Config Map, Secret or Service Account.
- You can reset the form to the last saved state by clicking Reload.
- The Environment resources are added to the virtual machine as disks. You can mount the secret, config map, or service account as you would mount any other disk.
- If the virtual machine is running, changes will not take effect until you restart the virtual machine. The newly added resources are marked as pending changes for both the Environment and Disks tab in the Pending Changes banner at the top of the page.
Verification
- From the Virtual Machine Overview page, click the Disks tab.
- Check to ensure that the secret, config map, or service account is included in the list of disks.
Optional. Choose the appropriate method to apply your changes:
- If the virtual machine is running, restart the virtual machine by clicking Actions → Restart Virtual Machine.
- If the virtual machine is stopped, start the virtual machine by clicking Actions → Start Virtual Machine.
You can now mount the secret, config map, or service account as you would mount any other disk.
7.11.2. Removing a secret, config map, or service account from a virtual machine
Remove a secret, config map, or service account from a virtual machine by using the OpenShift Container Platform web console.
Prerequisites
- You must have at least one secret, config map, or service account that is attached to a virtual machine.
Procedure
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click the Environment tab.
- Find the item that you want to delete in the list, and click Remove on the right side of the item.
- Click Save.
You can reset the form to the last saved state by clicking Reload.
Verification
- From the Virtual Machine Overview page, click the Disks tab.
- Check to ensure that the secret, config map, or service account that you removed is no longer included in the list of disks.
7.11.3. Additional resources
7.12. Installing VirtIO driver on an existing Windows virtual machine
7.12.1. Understanding VirtIO drivers
VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines to run in OpenShift Virtualization. The supported drivers are available in the container-native-virtualization/virtio-win
container disk of the Red Hat Ecosystem Catalog.
The container-native-virtualization/virtio-win
container disk must be attached to the virtual machine as a SATA CD drive to enable driver installation. You can install VirtIO drivers during Windows installation on the virtual machine or added to an existing Windows installation.
After the drivers are installed, the container-native-virtualization/virtio-win
container disk can be removed from the virtual machine.
See also: Installing Virtio drivers on a new Windows virtual machine.
7.12.2. Supported VirtIO drivers for Microsoft Windows virtual machines
Driver name | Hardware ID | Description |
---|---|---|
viostor |
VEN_1AF4&DEV_1001 | The block driver. Sometimes displays as an SCSI Controller in the Other devices group. |
viorng |
VEN_1AF4&DEV_1005 | The entropy source driver. Sometimes displays as a PCI Device in the Other devices group. |
NetKVM |
VEN_1AF4&DEV_1000 | The network driver. Sometimes displays as an Ethernet Controller in the Other devices group. Available only if a VirtIO NIC is configured. |
7.12.3. Adding VirtIO drivers container disk to a virtual machine
OpenShift Virtualization distributes VirtIO drivers for Microsoft Windows as a container disk, which is available from the Red Hat Ecosystem Catalog. To install these drivers to a Windows virtual machine, attach the container-native-virtualization/virtio-win
container disk to the virtual machine as a SATA CD drive in the virtual machine configuration file.
Prerequisites
-
Download the
container-native-virtualization/virtio-win
container disk from the Red Hat Ecosystem Catalog. This is not mandatory, because the container disk will be downloaded from the Red Hat registry if it not already present in the cluster, but it can reduce installation time.
Procedure
Add the
container-native-virtualization/virtio-win
container disk as acdrom
disk in the Windows virtual machine configuration file. The container disk will be downloaded from the registry if it is not already present in the cluster.spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
- 1
- OpenShift Virtualization boots virtual machine disks in the order defined in the
VirtualMachine
configuration file. You can either define other disks for the virtual machine before thecontainer-native-virtualization/virtio-win
container disk or use the optionalbootOrder
parameter to ensure the virtual machine boots from the correct disk. If you specify thebootOrder
for a disk, it must be specified for all disks in the configuration.
The disk is available once the virtual machine has started:
-
If you add the container disk to a running virtual machine, use
oc apply -f <vm.yaml>
in the CLI or reboot the virtual machine for the changes to take effect. -
If the virtual machine is not running, use
virtctl start <vm>
.
-
If you add the container disk to a running virtual machine, use
After the virtual machine has started, the VirtIO drivers can be installed from the attached SATA CD drive.
7.12.4. Installing VirtIO drivers on an existing Windows virtual machine
Install the VirtIO drivers from the attached SATA CD drive to an existing Windows virtual machine.
This procedure uses a generic approach to adding drivers to Windows. The process might differ slightly between versions of Windows. Refer to the installation documentation for your version of Windows for specific installation steps.
Procedure
- Start the virtual machine and connect to a graphical console.
- Log in to a Windows user session.
Open Device Manager and expand Other devices to list any Unknown device.
-
Open the
Device Properties
to identify the unknown device. Right-click the device and select Properties. - Click the Details tab and select Hardware Ids in the Property list.
- Compare the Value for the Hardware Ids with the supported VirtIO drivers.
-
Open the
- Right-click the device and select Update Driver Software.
- Click Browse my computer for driver software and browse to the attached SATA CD drive, where the VirtIO drivers are located. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
- Click Next to install the driver.
- Repeat this process for all the necessary VirtIO drivers.
- After the driver installs, click Close to close the window.
- Reboot the virtual machine to complete the driver installation.
7.12.5. Removing the VirtIO container disk from a virtual machine
After installing all required VirtIO drivers to the virtual machine, the container-native-virtualization/virtio-win
container disk no longer needs to be attached to the virtual machine. Remove the container-native-virtualization/virtio-win
container disk from the virtual machine configuration file.
Procedure
Edit the configuration file and remove the
disk
and thevolume
.$ oc edit vm <vm-name>
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
- Reboot the virtual machine for the changes to take effect.
7.13. Installing VirtIO driver on a new Windows virtual machine
7.13.1. Prerequisites
- Windows installation media accessible by the virtual machine, such as importing an ISO into a data volume and attaching it to the virtual machine.
7.13.2. Understanding VirtIO drivers
VirtIO drivers are paravirtualized device drivers required for Microsoft Windows virtual machines to run in OpenShift Virtualization. The supported drivers are available in the container-native-virtualization/virtio-win
container disk of the Red Hat Ecosystem Catalog.
The container-native-virtualization/virtio-win
container disk must be attached to the virtual machine as a SATA CD drive to enable driver installation. You can install VirtIO drivers during Windows installation on the virtual machine or added to an existing Windows installation.
After the drivers are installed, the container-native-virtualization/virtio-win
container disk can be removed from the virtual machine.
See also: Installing VirtIO driver on an existing Windows virtual machine.
7.13.3. Supported VirtIO drivers for Microsoft Windows virtual machines
Driver name | Hardware ID | Description |
---|---|---|
viostor |
VEN_1AF4&DEV_1001 | The block driver. Sometimes displays as an SCSI Controller in the Other devices group. |
viorng |
VEN_1AF4&DEV_1005 | The entropy source driver. Sometimes displays as a PCI Device in the Other devices group. |
NetKVM |
VEN_1AF4&DEV_1000 | The network driver. Sometimes displays as an Ethernet Controller in the Other devices group. Available only if a VirtIO NIC is configured. |
7.13.4. Adding VirtIO drivers container disk to a virtual machine
OpenShift Virtualization distributes VirtIO drivers for Microsoft Windows as a container disk, which is available from the Red Hat Ecosystem Catalog. To install these drivers to a Windows virtual machine, attach the container-native-virtualization/virtio-win
container disk to the virtual machine as a SATA CD drive in the virtual machine configuration file.
Prerequisites
-
Download the
container-native-virtualization/virtio-win
container disk from the Red Hat Ecosystem Catalog. This is not mandatory, because the container disk will be downloaded from the Red Hat registry if it not already present in the cluster, but it can reduce installation time.
Procedure
Add the
container-native-virtualization/virtio-win
container disk as acdrom
disk in the Windows virtual machine configuration file. The container disk will be downloaded from the registry if it is not already present in the cluster.spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 1 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
- 1
- OpenShift Virtualization boots virtual machine disks in the order defined in the
VirtualMachine
configuration file. You can either define other disks for the virtual machine before thecontainer-native-virtualization/virtio-win
container disk or use the optionalbootOrder
parameter to ensure the virtual machine boots from the correct disk. If you specify thebootOrder
for a disk, it must be specified for all disks in the configuration.
The disk is available once the virtual machine has started:
-
If you add the container disk to a running virtual machine, use
oc apply -f <vm.yaml>
in the CLI or reboot the virtual machine for the changes to take effect. -
If the virtual machine is not running, use
virtctl start <vm>
.
-
If you add the container disk to a running virtual machine, use
After the virtual machine has started, the VirtIO drivers can be installed from the attached SATA CD drive.
7.13.5. Installing VirtIO drivers during Windows installation
Install the VirtIO drivers from the attached SATA CD driver during Windows installation.
This procedure uses a generic approach to the Windows installation and the installation method might differ between versions of Windows. Refer to the documentation for the version of Windows that you are installing.
Procedure
- Start the virtual machine and connect to a graphical console.
- Begin the Windows installation process.
- Select the Advanced installation.
-
The storage destination will not be recognized until the driver is loaded. Click
Load driver
. - The drivers are attached as a SATA CD drive. Click OK and browse the CD drive for the storage driver to load. The drivers are arranged hierarchically according to their driver type, operating system, and CPU architecture.
- Repeat the previous two steps for all required drivers.
- Complete the Windows installation.
7.13.6. Removing the VirtIO container disk from a virtual machine
After installing all required VirtIO drivers to the virtual machine, the container-native-virtualization/virtio-win
container disk no longer needs to be attached to the virtual machine. Remove the container-native-virtualization/virtio-win
container disk from the virtual machine configuration file.
Procedure
Edit the configuration file and remove the
disk
and thevolume
.$ oc edit vm <vm-name>
spec: domain: devices: disks: - name: virtiocontainerdisk bootOrder: 2 cdrom: bus: sata volumes: - containerDisk: image: container-native-virtualization/virtio-win name: virtiocontainerdisk
- Reboot the virtual machine for the changes to take effect.
7.14. Advanced virtual machine management
7.14.1. Automating management tasks
You can automate OpenShift Virtualization management tasks by using Red Hat Ansible Automation Platform. Learn the basics by using an Ansible Playbook to create a new virtual machine.
7.14.1.1. About Red Hat Ansible Automation
Ansible is an automation tool used to configure systems, deploy software, and perform rolling updates. Ansible includes support for OpenShift Virtualization, and Ansible modules enable you to automate cluster management tasks such as template, persistent volume claim, and virtual machine operations.
Ansible provides a way to automate OpenShift Virtualization management, which you can also accomplish by using the oc
CLI tool or APIs. Ansible is unique because it allows you to integrate KubeVirt modules with other Ansible modules.
7.14.1.2. Automating virtual machine creation
You can use the kubevirt_vm
Ansible Playbook to create virtual machines in your OpenShift Container Platform cluster using Red Hat Ansible Automation Platform.
Prerequisites
- Red Hat Ansible Engine version 2.8 or newer
Procedure
Edit an Ansible Playbook YAML file so that it includes the
kubevirt_vm
task:kubevirt_vm: namespace: name: cpu_cores: memory: disks: - name: volume: containerDisk: image: disk: bus:
NoteThis snippet only includes the
kubevirt_vm
portion of the playbook.Edit the values to reflect the virtual machine you want to create, including the
namespace
, the number ofcpu_cores
, thememory
, and thedisks
. For example:kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio
If you want the virtual machine to boot immediately after creation, add
state: running
to the YAML file. For example:kubevirt_vm: namespace: default name: vm1 state: running 1 cpu_cores: 1
- 1
- Changing this value to
state: absent
deletes the virtual machine, if it already exists.
Run the
ansible-playbook
command, using your playbook’s file name as the only argument:$ ansible-playbook create-vm.yaml
Review the output to determine if the play was successful:
Example output
(...) TASK [Create my first VM] ************************************************************************ changed: [localhost] PLAY RECAP ******************************************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
If you did not include
state: running
in your playbook file and you want to boot the VM now, edit the file so that it includesstate: running
and run the playbook again:$ ansible-playbook create-vm.yaml
To verify that the virtual machine was created, try to access the VM console.
7.14.1.3. Example: Ansible Playbook for creating virtual machines
You can use the kubevirt_vm
Ansible Playbook to automate virtual machine creation.
The following YAML file is an example of the kubevirt_vm
playbook. It includes sample values that you must replace with your own information if you run the playbook.
--- - name: Ansible Playbook 1 hosts: localhost connection: local tasks: - name: Create my first VM kubevirt_vm: namespace: default name: vm1 cpu_cores: 1 memory: 64Mi disks: - name: containerdisk volume: containerDisk: image: kubevirt/cirros-container-disk-demo:latest disk: bus: virtio
Additional information
7.14.2. Using EFI mode for virtual machines
You can boot a virtual machine (VM) in Extensible Firmware Interface (EFI) mode.
7.14.2.1. About EFI mode for virtual machines
Extensible Firmware Interface (EFI), like legacy BIOS, initializes hardware components and operating system image files when a computer starts. EFI supports more modern features and customization options than BIOS, enabling faster boot times.
It stores all the information about initialization and startup in a file with a .efi
extension, which is stored on a special partition called EFI System Partition (ESP). The ESP also contains the boot loader programs for the operating system that is installed on the computer.
OpenShift Virtualization only supports a virtual machine (VM) with Secure Boot when using EFI mode. If Secure Boot is not enabled, the VM crashes repeatedly. However, the VM might not support Secure Boot. Before you boot a VM, verify that it supports Secure Boot by checking the VM settings.
7.14.2.2. Booting virtual machines in EFI mode
You can configure a virtual machine to boot in EFI mode by editing the VM or VMI manifest.
Prerequisites
-
Install the OpenShift CLI (
oc
).
Procedure
Create a YAML file that defines a VM object or a Virtual Machine Instance (VMI) object. Use the firmware stanza of the example YAML file:
Booting in EFI mode with secure boot active
apiversion: kubevirt.io/v1 kind: VirtualMachineInstance metadata: labels: special: vmi-secureboot name: vmi-secureboot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk features: acpi: {} smm: enabled: true 1 firmware: bootloader: efi: secureBoot: true 2 ...
- 1
- OpenShift Virtualization requires System Management Mode (
SMM
) to be enabled for Secure Boot in EFI mode to occur. - 2
- OpenShift Virtualization only supports a virtual machine (VM) with Secure Boot when using EFI mode. If Secure Boot is not enabled, the VM crashes repeatedly. However, the VM might not support Secure Boot. Before you boot a VM, verify that it supports Secure Boot by checking the VM settings.
Apply the manifest to your cluster by running the following command:
$ oc create -f <file_name>.yaml
7.14.3. Configuring PXE booting for virtual machines
PXE booting, or network booting, is available in OpenShift Virtualization. Network booting allows a computer to boot and load an operating system or other program without requiring a locally attached storage device. For example, you can use it to choose your desired OS image from a PXE server when deploying a new host.
7.14.3.1. Prerequisites
- A Linux bridge must be connected.
- The PXE server must be connected to the same VLAN as the bridge.
7.14.3.2. PXE booting with a specified MAC address
As an administrator, you can boot a client over the network by first creating a NetworkAttachmentDefinition
object for your PXE network. Then, reference the network attachment definition in your virtual machine instance configuration file before you start the virtual machine instance. You can also specify a MAC address in the virtual machine instance configuration file, if required by the PXE server.
Prerequisites
- A Linux bridge must be connected.
- The PXE server must be connected to the same VLAN as the bridge.
Procedure
Configure a PXE network on the cluster:
Create the network attachment definition file for PXE network
pxe-net-conf
:apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: pxe-net-conf spec: config: '{ "cniVersion": "0.3.1", "name": "pxe-net-conf", "plugins": [ { "type": "cnv-bridge", "bridge": "br1", "vlan": 1 1 }, { "type": "cnv-tuning" 2 } ] }'
NoteThe virtual machine instance will be attached to the bridge
br1
through an access port with the requested VLAN.
Create the network attachment definition by using the file you created in the previous step:
$ oc create -f pxe-net-conf.yaml
Edit the virtual machine instance configuration file to include the details of the interface and network.
Specify the network and MAC address, if required by the PXE server. If the MAC address is not specified, a value is assigned automatically. However, note that at this time, MAC addresses assigned automatically are not persistent.
Ensure that
bootOrder
is set to1
so that the interface boots first. In this example, the interface is connected to a network called<pxe-net>
:interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1
NoteBoot order is global for interfaces and disks.
Assign a boot device number to the disk to ensure proper booting after operating system provisioning.
Set the disk
bootOrder
value to2
:devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2
Specify that the network is connected to the previously created network attachment definition. In this scenario,
<pxe-net>
is connected to the network attachment definition called<pxe-net-conf>
:networks: - name: default pod: {} - name: pxe-net multus: networkName: pxe-net-conf
Create the virtual machine instance:
$ oc create -f vmi-pxe-boot.yaml
Example output
virtualmachineinstance.kubevirt.io "vmi-pxe-boot" created
Wait for the virtual machine instance to run:
$ oc get vmi vmi-pxe-boot -o yaml | grep -i phase phase: Running
View the virtual machine instance using VNC:
$ virtctl vnc vmi-pxe-boot
- Watch the boot screen to verify that the PXE boot is successful.
Log in to the virtual machine instance:
$ virtctl console vmi-pxe-boot
Verify the interfaces and MAC address on the virtual machine and that the interface connected to the bridge has the specified MAC address. In this case, we used
eth1
for the PXE boot, without an IP address. The other interface,eth0
, got an IP address from OpenShift Container Platform.$ ip addr
Example output
... 3. eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether de:00:00:00:00:de brd ff:ff:ff:ff:ff:ff
7.14.3.3. Template: Virtual machine instance configuration file for PXE booting
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: creationTimestamp: null labels: special: vmi-pxe-boot name: vmi-pxe-boot spec: domain: devices: disks: - disk: bus: virtio name: containerdisk bootOrder: 2 - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default - bridge: {} name: pxe-net macAddress: de:00:00:00:00:de bootOrder: 1 machine: type: "" resources: requests: memory: 1024M networks: - name: default pod: {} - multus: networkName: pxe-net-conf name: pxe-net terminationGracePeriodSeconds: 0 volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin name: cloudinitdisk status: {}
7.14.3.4. OpenShift Virtualization networking glossary
OpenShift Virtualization provides advanced networking functionality by using custom resources and plug-ins.
The following terms are used throughout OpenShift Virtualization documentation:
- Container Network Interface (CNI)
- a Cloud Native Computing Foundation project, focused on container network connectivity. OpenShift Virtualization uses CNI plug-ins to build upon the basic Kubernetes networking functionality.
- Multus
- a "meta" CNI plug-in that allows multiple CNIs to exist so that a pod or virtual machine can use the interfaces it needs.
- Custom resource definition (CRD)
- a Kubernetes API resource that allows you to define custom resources, or an object defined by using the CRD API resource.
- Network attachment definition
- a CRD introduced by the Multus project that allows you to attach pods, virtual machines, and virtual machine instances to one or more networks.
- Preboot eXecution Environment (PXE)
- an interface that enables an administrator to boot a client machine from a server over the network. Network booting allows you to remotely load operating systems and other software onto the client.
7.14.4. Managing guest memory
If you want to adjust guest memory settings to suit a specific use case, you can do so by editing the guest’s YAML configuration file. OpenShift Virtualization allows you to configure guest memory overcommitment and disable guest memory overhead accounting.
The following procedures increase the chance that virtual machine processes will be killed due to memory pressure. Proceed only if you understand the risks.
7.14.4.1. Configuring guest memory overcommitment
If your virtual workload requires more memory than available, you can use memory overcommitment to allocate all or most of the host’s memory to your virtual machine instances (VMIs). Enabling memory overcommitment means that you can maximize resources that are normally reserved for the host.
For example, if the host has 32 GB RAM, you can use memory overcommitment to fit 8 virtual machines (VMs) with 4 GB RAM each. This allocation works under the assumption that the virtual machines will not use all of their memory at the same time.
Memory overcommitment increases the potential for virtual machine processes to be killed due to memory pressure (OOM killed).
The potential for a VM to be OOM killed varies based on your specific configuration, node memory, available swap space, virtual machine memory consumption, the use of kernel same-page merging (KSM), and other factors.
Procedure
To explicitly tell the virtual machine instance that it has more memory available than was requested from the cluster, edit the virtual machine configuration file and set
spec.domain.memory.guest
to a higher value thanspec.domain.resources.requests.memory
. This process is called memory overcommitment.In this example,
1024M
is requested from the cluster, but the virtual machine instance is told that it has2048M
available. As long as there is enough free memory available on the node, the virtual machine instance will consume up to 2048M.kind: VirtualMachine spec: template: domain: resources: requests: memory: 1024M memory: guest: 2048M
NoteThe same eviction rules as those for pods apply to the virtual machine instance if the node is under memory pressure.
Create the virtual machine:
$ oc create -f <file_name>.yaml
7.14.4.2. Disabling guest memory overhead accounting
A small amount of memory is requested by each virtual machine instance in addition to the amount that you request. This additional memory is used for the infrastructure that wraps each VirtualMachineInstance
process.
Though it is not usually advisable, it is possible to increase the virtual machine instance density on the node by disabling guest memory overhead accounting.
Disabling guest memory overhead accounting increases the potential for virtual machine processes to be killed due to memory pressure (OOM killed).
The potential for a VM to be OOM killed varies based on your specific configuration, node memory, available swap space, virtual machine memory consumption, the use of kernel same-page merging (KSM), and other factors.
Procedure
To disable guest memory overhead accounting, edit the YAML configuration file and set the
overcommitGuestOverhead
value totrue
. This parameter is disabled by default.kind: VirtualMachine spec: template: domain: resources: overcommitGuestOverhead: true requests: memory: 1024M
NoteIf
overcommitGuestOverhead
is enabled, it adds the guest overhead to memory limits, if present.Create the virtual machine:
$ oc create -f <file_name>.yaml
7.14.5. Using huge pages with virtual machines
You can use huge pages as backing memory for virtual machines in your cluster.
7.14.5.1. Prerequisites
- Nodes must have pre-allocated huge pages configured.
7.14.5.2. What huge pages do
Memory is managed in blocks known as pages. On most systems, a page is 4Ki. 1Mi of memory is equal to 256 pages; 1Gi of memory is 256,000 pages, and so on. CPUs have a built-in memory management unit that manages a list of these pages in hardware. The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software-based address translation, resulting in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size.
A huge page is a memory page that is larger than 4Ki. On x86_64 architectures, there are two common huge page sizes: 2Mi and 1Gi. Sizes vary on other architectures. To use huge pages, code must be written so that applications are aware of them. Transparent Huge Pages (THP) attempt to automate the management of huge pages without application knowledge, but they have limitations. In particular, they are limited to 2Mi page sizes. THP can lead to performance degradation on nodes with high memory utilization or fragmentation due to defragmenting efforts of THP, which can lock memory pages. For this reason, some applications may be designed to (or recommend) usage of pre-allocated huge pages instead of THP.
In OpenShift Virtualization, virtual machines can be configured to consume pre-allocated huge pages.
7.14.5.3. Configuring huge pages for virtual machines
You can configure virtual machines to use pre-allocated huge pages by including the memory.hugepages.pageSize
and resources.requests.memory
parameters in your virtual machine configuration.
The memory request must be divisible by the page size. For example, you cannot request 500Mi
memory with a page size of 1Gi
.
The memory layouts of the host and the guest OS are unrelated. Huge pages requested in the virtual machine manifest apply to QEMU. Huge pages inside the guest can only be configured based on the amount of available memory of the virtual machine instance.
If you edit a running virtual machine, the virtual machine must be rebooted for the changes to take effect.
Prerequisites
- Nodes must have pre-allocated huge pages configured.
Procedure
In your virtual machine configuration, add the
resources.requests.memory
andmemory.hugepages.pageSize
parameters to thespec.domain
. The following configuration snippet is for a virtual machine that requests a total of4Gi
memory with a page size of1Gi
:kind: VirtualMachine ... spec: domain: resources: requests: memory: "4Gi" 1 memory: hugepages: pageSize: "1Gi" 2 ...
Apply the virtual machine configuration:
$ oc apply -f <virtual_machine>.yaml
7.14.6. Enabling dedicated resources for virtual machines
To improve performance, you can dedicate node resources, such as CPU, to a virtual machine.
7.14.6.1. About dedicated resources
When you enable dedicated resources for your virtual machine, your virtual machine’s workload is scheduled on CPUs that will not be used by other processes. By using dedicated resources, you can improve the performance of the virtual machine and the accuracy of latency predictions.
7.14.6.2. Prerequisites
-
The CPU Manager must be configured on the node. Verify that the node has the
cpumanager = true
label before scheduling virtual machine workloads. - The virtual machine must be powered off.
7.14.6.3. Enabling dedicated resources for a virtual machine
You can enable dedicated resources for a virtual machine in the Details tab. Virtual machines that were created by using a Red Hat template or the wizard can be enabled with dedicated resources.
Procedure
- Click Workloads → Virtual Machines from the side menu.
- Select a virtual machine to open the Virtual Machine tab.
- Click the Details tab.
- Click the pencil icon to the right of the Dedicated Resources field to open the Dedicated Resources window.
- Select Schedule this workload with dedicated resources (guaranteed policy).
- Click Save.
7.14.7. Scheduling virtual machines
You can schedule a virtual machine (VM) on a node by ensuring that the VM’s CPU model and policy attribute are matched for compatibility with the CPU models and policy attributes supported by the node.
7.14.7.1. Understanding policy attributes
You can schedule a virtual machine (VM) by specifying a policy attribute and a CPU feature that is matched for compatibility when the VM is scheduled on a node. A policy attribute specified for a VM determines how that VM is scheduled on a node.
Policy attribute | Description |
---|---|
force | The VM is forced to be scheduled on a node. This is true even if the host CPU does not support the VM’s CPU. |
require | Default policy that applies to a VM if the VM is not configured with a specific CPU model and feature specification. If a node is not configured to support CPU node discovery with this default policy attribute or any one of the other policy attributes, VMs are not scheduled on that node. Either the host CPU must support the VM’s CPU or the hypervisor must be able to emulate the supported CPU model. |
optional | The VM is added to a node if that VM is supported by the host’s physical machine CPU. |
disable | The VM cannot be scheduled with CPU node discovery. |
forbid | The VM is not scheduled even if the feature is supported by the host CPU and CPU node discovery is enabled. |
7.14.7.2. Setting a policy attribute and CPU feature
You can set a policy attribute and CPU feature for each virtual machine (VM) to ensure that it is scheduled on a node according to policy and feature. The CPU feature that you set is verified to ensure that it is supported by the host CPU or emulated by the hypervisor.
Procedure
Edit the
domain
spec of your VM configuration file. The following example sets the CPU feature and therequire
policy for a virtual machine instance (VMI):apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: myvmi spec: domain: cpu: features: - name: apic 1 policy: require 2
7.14.7.3. Scheduling virtual machines with the supported CPU model
You can configure a CPU model for a virtual machine (VM) or a virtual machine instance (VMI) to schedule it on a node where its CPU model is supported.
Procedure
Edit the
domain
spec of your virtual machine configuration file. The following example shows a specific CPU model defined for a VMI:apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: Conroe 1
- 1
- CPU model for the VMI.
7.14.7.4. Scheduling virtual machines with the host model
When the CPU model for a virtual machine (VM) is set to host-model
, the VM inherits the CPU model of the node where it is scheduled.
Procedure
Edit the
domain
spec of your VM configuration file. The following example showshost-model
being specified for the virtual machine instance (VMI):apiVersion: kubevirt/v1alpha3 kind: VirtualMachineInstance metadata: name: myvmi spec: domain: cpu: model: host-model 1
- 1
- The VM or VMI that inherits the CPU model of the node where it is scheduled.
7.15. Importing virtual machines
7.15.1. TLS certificates for data volume imports
7.15.1.1. Adding TLS certificates for authenticating data volume imports
TLS certificates for registry or HTTPS endpoints must be added to a config map to import data from these sources. This config map must be present in the namespace of the destination data volume.
Create the config map by referencing the relative file path for the TLS certificate.
Procedure
Ensure you are in the correct namespace. The config map can only be referenced by data volumes if it is in the same namespace.
$ oc get ns
Create the config map:
$ oc create configmap <configmap-name> --from-file=</path/to/file/ca.pem>
7.15.1.2. Example: Config map created from a TLS certificate
The following example is of a config map created from ca.pem
TLS certificate.
apiVersion: v1 kind: ConfigMap metadata: name: tls-certs data: ca.pem: | -----BEGIN CERTIFICATE----- ... <base64 encoded cert> ... -----END CERTIFICATE-----
7.15.2. Importing virtual machine images with data volumes
Use the Containerized Data Importer (CDI) to import a virtual machine image into a persistent volume claim (PVC) by using a data volume. You can attach a data volume to a virtual machine for persistent storage.
The virtual machine image can be hosted at an HTTP or HTTPS endpoint, or built into a container disk and stored in a container registry.
When you import a disk image into a PVC, the disk image is expanded to use the full storage capacity that is requested in the PVC. To use this space, the disk partitions and file system(s) in the virtual machine might need to be expanded.
The resizing procedure varies based on the operating system installed on the virtual machine. Refer to the operating system documentation for details.
7.15.2.1. Prerequisites
- If the endpoint requires a TLS certificate, the certificate must be included in a config map in the same namespace as the data volume and referenced in the data volume configuration.
To import a container disk:
- You might need to prepare a container disk from a virtual machine image and store it in your container registry before importing it.
-
If the container registry does not have TLS, you must add the registry to the
cdi-insecure-registries
config map before you can import a container disk from it.
- You might need to define a storage class or prepare CDI scratch space for this operation to complete successfully.
7.15.2.2. CDI supported operations matrix
This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space.
Content types | HTTP | HTTPS | HTTP basic auth | Registry | Upload |
---|---|---|---|---|---|
KubeVirt(QCOW2) |
✓ QCOW2 |
✓ QCOW2** |
✓ QCOW2 |
✓ QCOW2* |
✓ QCOW2* |
KubeVirt (RAW) |
✓ RAW |
✓ RAW |
✓ RAW |
✓ RAW* |
✓ RAW* |
✓ Supported operation
□ Unsupported operation
* Requires scratch space
** Requires scratch space if a custom certificate authority is required
7.15.2.3. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.15.2.4. Importing a virtual machine image into storage by using a data volume
You can import a virtual machine image into storage by using a data volume.
The virtual machine image can be hosted at an HTTP or HTTPS endpoint or the image can be built into a container disk and stored in a container registry.
You specify the data source for the image in a VirtualMachine
configuration file. When the virtual machine is created, the data volume with the virtual machine image is imported into storage.
Prerequisites
To import a virtual machine image you must have the following:
-
A virtual machine disk image in RAW, ISO, or QCOW2 format, optionally compressed by using
xz
orgz
. - An HTTP or HTTPS endpoint where the image is hosted, along with any authentication credentials needed to access the data source.
-
A virtual machine disk image in RAW, ISO, or QCOW2 format, optionally compressed by using
- To import a container disk, you must have a virtual machine image built into a container disk and stored in a container registry, along with any authentication credentials needed to access the data source.
- If the virtual machine must communicate with servers that use self-signed certificates or certificates not signed by the system CA bundle, you must create a config map in the same namespace as the data volume.
Procedure
If your data source requires authentication, create a
Secret
manifest, specifying the data source credentials, and save it asendpoint-secret.yaml
:apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3
Apply the
Secret
manifest:$ oc apply -f endpoint-secret.yaml
Edit the
VirtualMachine
manifest, specifying the data source for the virtual machine image you want to import, and save it asvm-fedora-datavolume.yaml
:apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: creationTimestamp: null labels: kubevirt.io/vm: vm-fedora-datavolume name: vm-fedora-datavolume 1 spec: dataVolumeTemplates: - metadata: creationTimestamp: null name: fedora-dv 2 spec: storage: resources: requests: storage: 10Gi storageClassName: local source: http: 3 url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 4 secretRef: endpoint-secret 5 certConfigMap: "" 6 status: {} running: true template: metadata: creationTimestamp: null labels: kubevirt.io/vm: vm-fedora-datavolume spec: domain: devices: disks: - disk: bus: virtio name: datavolumedisk1 machine: type: "" resources: requests: memory: 1.5Gi terminationGracePeriodSeconds: 60 volumes: - dataVolume: name: fedora-dv name: datavolumedisk1 status: {}
- 1
- Specify the name of the virtual machine.
- 2
- Specify the name of the data volume.
- 3
- Specify
http
for an HTTP or HTTPS endpoint. Specifyregistry
for a container disk image imported from a registry. - 4
- The source of the virtual machine image you want to import. This example references a virtual machine image at an HTTPS endpoint. An example of a container registry endpoint is
url: "docker://kubevirt/fedora-cloud-container-disk-demo:latest"
. - 5
- Required if you created a
Secret
for the data source. - 6
- Optional: Specify a CA certificate config map.
Create the virtual machine:
$ oc create -f vm-fedora-datavolume.yaml
NoteThe
oc create
command creates the data volume and the virtual machine. The CDI controller creates an underlying PVC with the correct annotation and the import process begins. When the import is complete, the data volume status changes toSucceeded
. You can start the virtual machine.Data volume provisioning happens in the background, so there is no need to monitor the process.
Verification
The importer pod downloads the virtual machine image or container disk from the specified URL and stores it on the provisioned PV. View the status of the importer pod by running the following command:
$ oc get pods
Monitor the data volume until its status is
Succeeded
by running the following command:$ oc describe dv fedora-dv 1
- 1
- Specify the data volume name that you defined in the
VirtualMachine
manifest.
Verify that provisioning is complete and that the virtual machine has started by accessing its serial console:
$ virtctl console vm-fedora-datavolume
7.15.2.5. Additional resources
- Configure preallocation mode to improve write performance for data volume operations.
7.15.3. Importing virtual machine images into block storage with data volumes
You can import an existing virtual machine image into your OpenShift Container Platform cluster. OpenShift Virtualization uses data volumes to automate the import of data and the creation of an underlying persistent volume claim (PVC).
When you import a disk image into a PVC, the disk image is expanded to use the full storage capacity that is requested in the PVC. To use this space, the disk partitions and file system(s) in the virtual machine might need to be expanded.
The resizing procedure varies based on the operating system that is installed on the virtual machine. Refer to the operating system documentation for details.
7.15.3.1. Prerequisites
- If you require scratch space according to the CDI supported operations matrix, you must first define a storage class or prepare CDI scratch space for this operation to complete successfully.
7.15.3.2. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.15.3.3. About block persistent volumes
A block persistent volume (PV) is a PV that is backed by a raw block device. These volumes do not have a file system and can provide performance benefits for virtual machines by reducing overhead.
Raw block volumes are provisioned by specifying volumeMode: Block
in the PV and persistent volume claim (PVC) specification.
7.15.3.4. Creating a local block persistent volume
Create a local block persistent volume (PV) on a node by populating a file and mounting it as a loop device. You can then reference this loop device in a PV manifest as a Block
volume and use it as a block device for a virtual machine image.
Procedure
-
Log in as
root
to the node on which to create the local PV. This procedure usesnode01
for its examples. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file
loop10
with a size of 2Gb (20 100Mb blocks):$ dd if=/dev/zero of=<loop10> bs=100M count=20
Mount the
loop10
file as a loop device.$ losetup </dev/loop10>d3 <loop10> 1 2
Create a
PersistentVolume
manifest that references the mounted loop device.kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4
Create the block PV.
# oc create -f <local-block-pv10.yaml>1
- 1
- The file name of the persistent volume created in the previous step.
7.15.3.5. Importing a virtual machine image into block storage by using a data volume
You can import a virtual machine image into block storage by using a data volume. You reference the data volume in a VirtualMachine
manifest before you create a virtual machine.
Prerequisites
-
A virtual machine disk image in RAW, ISO, or QCOW2 format, optionally compressed by using
xz
orgz
. - An HTTP or HTTPS endpoint where the image is hosted, along with any authentication credentials needed to access the data source.
Procedure
If your data source requires authentication, create a
Secret
manifest, specifying the data source credentials, and save it asendpoint-secret.yaml
:apiVersion: v1 kind: Secret metadata: name: endpoint-secret 1 labels: app: containerized-data-importer type: Opaque data: accessKeyId: "" 2 secretKey: "" 3
Apply the
Secret
manifest:$ oc apply -f endpoint-secret.yaml
Create a
DataVolume
manifest, specifying the data source for the virtual machine image andBlock
forstorage.volumeMode
.apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: import-pv-datavolume 1 spec: storageClassName: local 2 source: http: url: "https://mirror.arizona.edu/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2" 3 secretRef: endpoint-secret 4 storage: volumeMode: Block 5 resources: requests: storage: 10Gi
- 1
- Specify the name of the data volume.
- 2
- Optional: Set the storage class or omit it to accept the cluster default.
- 3
- Specify the HTTP or HTTPS URL of the image to import.
- 4
- Required if you created a
Secret
for the data source. - 5
- The volume mode and access mode are detected automatically for known storage provisioners. Otherwise, specify
Block
.
Create the data volume to import the virtual machine image:
$ oc create -f import-pv-datavolume.yaml
You can reference this data volume in a VirtualMachine
manifest before you create a virtual machine.
7.15.3.6. CDI supported operations matrix
This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space.
Content types | HTTP | HTTPS | HTTP basic auth | Registry | Upload |
---|---|---|---|---|---|
KubeVirt(QCOW2) |
✓ QCOW2 |
✓ QCOW2** |
✓ QCOW2 |
✓ QCOW2* |
✓ QCOW2* |
KubeVirt (RAW) |
✓ RAW |
✓ RAW |
✓ RAW |
✓ RAW* |
✓ RAW* |
✓ Supported operation
□ Unsupported operation
* Requires scratch space
** Requires scratch space if a custom certificate authority is required
7.15.3.7. Additional resources
- Configure preallocation mode to improve write performance for data volume operations.
7.15.4. Importing a single Red Hat Virtualization virtual machine
You can import a single Red Hat Virtualization (RHV) virtual machine into OpenShift Virtualization by using the VM Import wizard or the CLI.
7.15.4.1. OpenShift Virtualization storage feature matrix
The following table describes the OpenShift Virtualization storage types that support VM import.
RHV VM import | |
---|---|
OpenShift Container Storage: RBD block-mode volumes | Yes |
OpenShift Virtualization hostpath provisioner | No |
Other multi-node writable storage | Yes [1] |
Other single-node writable storage | Yes [2] |
- PVCs must request a ReadWriteMany access mode.
- PVCs must request a ReadWriteOnce access mode.
7.15.4.2. Prerequisites for importing a virtual machine
Importing a virtual machine from Red Hat Virtualization (RHV) into OpenShift Virtualization has the following prerequisites:
- You must have admin user privileges.
Storage:
- The OpenShift Virtualization local and shared persistent storage classes must support VM import.
- If you are using Ceph RBD block-mode volumes and the available storage space is too small for the virtual disk, the import process bar stops at 75% for more than 20 minutes and the migration does not succeed. No error message is displayed in the web console. BZ#1910019
Networks:
- The RHV and OpenShift Virtualization networks must either have the same name or be mapped to each other.
-
The RHV VM network interface must be
e1000
,rtl8139
, orvirtio
.
VM disks:
-
The disk interface must be
sata
,virtio_scsi
, orvirtio
. - The disk must not be configured as a direct LUN.
-
The disk status must not be
illegal
orlocked
. -
The storage type must be
image
. - SCSI reservation must be disabled.
-
ScsiGenericIO
must be disabled.
-
The disk interface must be
VM configuration:
- If the VM uses GPU resources, the nodes providing the GPUs must be configured.
- The VM must not be configured for vGPU resources.
-
The VM must not have snapshots with disks in an
illegal
state. - The VM must not have been created with OpenShift Container Platform and subsequently added to RHV.
- The VM must not be configured for USB devices.
-
The watchdog model must not be
diag288
.
7.15.4.3. Importing a virtual machine with the VM Import wizard
You can import a single virtual machine with the VM Import wizard.
Procedure
- In the web console, click Workloads → Virtual Machines.
- Click Create Virtual Machine and select Import with Wizard.
- Select Red Hat Virtualization (RHV) from the Provider list.
Select Connect to New Instance or a saved RHV instance.
If you select Connect to New Instance, fill in the following fields:
-
API URL: For example,
https://<RHV_Manager_FQDN>/ovirt-engine/api
CA certificate: Click Browse to upload the RHV Manager CA certificate or paste the CA certificate into the field.
View the CA certificate by running the following command:
$ openssl s_client -connect <RHV_Manager_FQDN>:443 -showcerts < /dev/null
The CA certificate is the second certificate in the output.
-
Username: RHV Manager user name, for example,
ocpadmin@internal
- Password: RHV Manager password
-
API URL: For example,
- If you select a saved RHV instance, the wizard connects to the RHV instance using the saved credentials.
Click Check and Save and wait for the connection to complete.
NoteThe connection details are stored in a secret. If you add a provider with an incorrect URL, user name, or password, click Workloads → Secrets and delete the provider secret.
- Select a cluster and a virtual machine.
- Click Next.
- In the Review screen, review your settings.
- Optional: You can select Start virtual machine on creation.
Click Edit to update the following settings:
- General → Name: The VM name is limited to 63 characters.
General → Description: Optional description of the VM.
Storage Class: Select NFS or ocs-storagecluster-ceph-rbd.
If you select ocs-storagecluster-ceph-rbd, you must set the Volume Mode of the disk to Block.
- Advanced → Volume Mode: Select Block.
- Advanced → Volume Mode: Select Block.
- Networking → Network: You can select a network from a list of available network attachment definition objects.
Click Import or Review and Import, if you have edited the import settings.
A Successfully created virtual machine message and a list of resources created for the virtual machine are displayed. The virtual machine appears in Workloads → Virtual Machines.
Virtual machine wizard fields
Name | Parameter | Description |
---|---|---|
Name |
The name can contain lowercase letters ( | |
Description | Optional description field. | |
Operating System | The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. | |
Boot Source | Import via URL (creates PVC) | Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. |
Clone existing PVC (creates PVC) | Select an existent persistent volume claim available on the cluster and clone it. | |
Import via Registry (creates PVC) |
Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: | |
PXE (network boot - adds network interface) | Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. | |
Persistent Volume Claim project | Project name that you want to use for cloning the PVC. | |
Persistent Volume Claim name | PVC name that should apply to this virtual machine template if you are cloning an existing PVC. | |
Mount this as a CD-ROM boot source | A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. | |
Flavor | Tiny, Small, Medium, Large, Custom | Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. |
Workload Type | Desktop | A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. |
Server | Balances performance and it is compatible with a wide range of server workloads. | |
High-Performance | A virtual machine configuration that is optimized for high-performance workloads. | |
Start this virtual machine after creation. | This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. |
Networking fields
Name | Description |
---|---|
Name | Name for the network interface controller. |
Model | Indicates the model of the network interface controller. Supported values are e1000e and virtio. |
Network | List of available network attachment definitions. |
Type |
List of available binding methods. For the default pod network, |
MAC Address | MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. |
Storage fields
Name | Selection | Description |
---|---|---|
Source | Blank (creates PVC) | Create an empty disk. |
Import via URL (creates PVC) | Import content via URL (HTTP or HTTPS endpoint). | |
Use an existing PVC | Use a PVC that is already available in the cluster. | |
Clone existing PVC (creates PVC) | Select an existing PVC available in the cluster and clone it. | |
Import via Registry (creates PVC) | Import content via container registry. | |
Container (ephemeral) | Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. | |
Name |
Name of the disk. The name can contain lowercase letters ( | |
Size | Size of the disk in GiB. | |
Type | Type of disk. Example: Disk or CD-ROM | |
Interface | Type of disk device. Supported interfaces are virtIO, SATA, and SCSI. | |
Storage Class | The storage class that is used to create the disk. | |
Advanced → Volume Mode | Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem. |
Advanced storage settings
Name | Parameter | Description |
---|---|---|
Volume Mode | Filesystem | Stores the virtual disk on a file system-based volume. |
Block |
Stores the virtual disk directly on the block volume. Only use | |
Access Mode [1] | Single User (RWO) | The disk can be mounted as read/write by a single node. |
Shared Access (RWX) | The disk can be mounted as read/write by many nodes. | |
Read Only (ROX) | The disk can be mounted as read-only by many nodes. |
- You can change the access mode by using the command line interface.
7.15.4.4. Importing a virtual machine with the CLI
You can import a virtual machine with the CLI by creating the Secret
and VirtualMachineImport
custom resources (CRs). The Secret
CR stores the RHV Manager credentials and CA certificate. The VirtualMachineImport
CR defines the parameters of the VM import process.
Optional: You can create a ResourceMapping
CR that is separate from the VirtualMachineImport
CR. A ResourceMapping
CR provides greater flexibility, for example, if you import additional RHV VMs.
The default target storage class must be NFS. Cinder does not support RHV VM import.
Procedure
Create the
Secret
CR by running the following command:$ cat <<EOF | oc create -f - apiVersion: v1 kind: Secret metadata: name: rhv-credentials namespace: default 1 type: Opaque stringData: ovirt: | apiUrl: <api_endpoint> 2 username: ocpadmin@internal password: 3 caCert: | -----BEGIN CERTIFICATE----- 4 -----END CERTIFICATE----- EOF
- 1
- Optional. You can specify a different namespace in all the CRs.
- 2
- Specify the API endpoint of the RHV Manager, for example,
\"https://www.example.com:8443/ovirt-engine/api"
- 3
- Specify the password for
ocpadmin@internal
. - 4
- Specify the RHV Manager CA certificate. You can obtain the CA certificate by running the following command:
$ openssl s_client -connect :443 -showcerts < /dev/null
Optional: Create a
ResourceMapping
CR if you want to separate the resource mapping from theVirtualMachineImport
CR by running the following command:$ cat <<EOF | kubectl create -f - apiVersion: v2v.kubevirt.io/v1alpha1 kind: ResourceMapping metadata: name: resourcemapping_example namespace: default spec: ovirt: networkMappings: - source: name: <rhv_logical_network>/<vnic_profile> 1 target: name: <target_network> 2 type: pod storageMappings: 3 - source: name: <rhv_storage_domain> 4 target: name: <target_storage_class> 5 volumeMode: <volume_mode> 6 EOF
- 1
- Specify the RHV logical network and vNIC profile.
- 2
- Specify the OpenShift Virtualization network.
- 3
- If storage mappings are specified in both the
ResourceMapping
and theVirtualMachineImport
CRs, theVirtualMachineImport
CR takes precedence. - 4
- Specify the RHV storage domain.
- 5
- Specify
nfs
orocs-storagecluster-ceph-rbd
. - 6
- If you specified the
ocs-storagecluster-ceph-rbd
storage class, you must specifyBlock
as the volume mode.
Create the
VirtualMachineImport
CR by running the following command:$ cat <<EOF | oc create -f - apiVersion: v2v.kubevirt.io/v1beta1 kind: VirtualMachineImport metadata: name: vm-import namespace: default spec: providerCredentialsSecret: name: rhv-credentials namespace: default # resourceMapping: 1 # name: resourcemapping-example # namespace: default targetVmName: vm_example 2 startVm: true source: ovirt: vm: id: <source_vm_id> 3 name: <source_vm_name> 4 cluster: name: <source_cluster_name> 5 mappings: 6 networkMappings: - source: name: <source_logical_network>/<vnic_profile> 7 target: name: <target_network> 8 type: pod storageMappings: 9 - source: name: <source_storage_domain> 10 target: name: <target_storage_class> 11 accessMode: <volume_access_mode> 12 diskMappings: - source: id: <source_vm_disk_id> 13 target: name: <target_storage_class> 14 EOF
- 1
- If you create a
ResourceMapping
CR, uncomment theresourceMapping
section. - 2
- Specify the target VM name.
- 3
- Specify the source VM ID, for example,
80554327-0569-496b-bdeb-fcbbf52b827b
. You can obtain the VM ID by enteringhttps://www.example.com/ovirt-engine/api/vms/
in a web browser on the Manager machine to list all VMs. Locate the VM you want to import and its corresponding VM ID. You do not need to specify a VM name or cluster name. - 4
- If you specify the source VM name, you must also specify the source cluster. Do not specify the source VM ID.
- 5
- If you specify the source cluster, you must also specify the source VM name. Do not specify the source VM ID.
- 6
- If you create a
ResourceMapping
CR, comment out themappings
section. - 7
- Specify the logical network and vNIC profile of the source VM.
- 8
- Specify the OpenShift Virtualization network.
- 9
- If storage mappings are specified in both the
ResourceMapping
and theVirtualMachineImport
CRs, theVirtualMachineImport
CR takes precedence. - 10
- Specify the source storage domain.
- 11
- Specify the target storage class.
- 12
- Specify
ReadWriteOnce
,ReadWriteMany
, orReadOnlyMany
. If no access mode is specified, {virt} determines the correct volume access mode based on the Host → Migration mode setting of the RHV VM or on the virtual disk access mode:-
If the RHV VM migration mode is
Allow manual and automatic migration
, the default access mode isReadWriteMany
. -
If the RHV virtual disk access mode is
ReadOnly
, the default access mode isReadOnlyMany
. -
For all other settings, the default access mode is
ReadWriteOnce
.
-
If the RHV VM migration mode is
- 13
- Specify the source VM disk ID, for example,
8181ecc1-5db8-4193-9c92-3ddab3be7b05
. You can obtain the disk ID by enteringhttps://www.example.com/ovirt-engine/api/vms/vm23
in a web browser on the Manager machine and reviewing the VM details. - 14
- Specify the target storage class.
Follow the progress of the virtual machine import to verify that the import was successful:
$ oc get vmimports vm-import -n default
The output indicating a successful import resembles the following example:
Example output
... status: conditions: - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Validation completed successfully reason: ValidationCompleted status: "True" type: Valid - lastHeartbeatTime: "2020-07-22T08:58:52Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: 'VM specifies IO Threads: 1, VM has NUMA tune mode specified: interleave' reason: MappingRulesVerificationReportedWarnings status: "True" type: MappingRulesVerified - lastHeartbeatTime: "2020-07-22T08:58:56Z" lastTransitionTime: "2020-07-22T08:58:52Z" message: Copying virtual machine disks reason: CopyingDisks status: "True" type: Processing dataVolumes: - name: fedora32-b870c429-11e0-4630-b3df-21da551a48c0 targetVmName: fedora32
7.15.4.4.1. Creating a config map for importing a VM
You can create a config map to map the Red Hat Virtualization (RHV) virtual machine operating system to an OpenShift Virtualization template if you want to override the default vm-import-controller
mapping or to add additional mappings.
The default vm-import-controller
config map contains the following RHV operating systems and their corresponding common OpenShift Virtualization templates.
RHV VM operating system | OpenShift Virtualization template |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Procedure
In a web browser, identify the REST API name of the RHV VM operating system by navigating to
http://<RHV_Manager_FQDN>/ovirt-engine/api/vms/<VM_ID>
. The operating system name appears in the<os>
section of the XML output, as shown in the following example:... <os> ... <type>rhel_8x64</type> </os>
View a list of the available OpenShift Virtualization templates:
$ oc get templates -n openshift --show-labels | tr ',' '\n' | grep os.template.kubevirt.io | sed -r 's#os.template.kubevirt.io/(.*)=.*#\1#g' | sort -u
Example output
fedora31 fedora32 ... rhel8.1 rhel8.2 ...
- If an OpenShift Virtualization template that matches the RHV VM operating system does not appear in the list of available templates, create a template with the OpenShift Virtualization web console.
Create a config map to map the RHV VM operating system to the OpenShift Virtualization template:
$ cat <<EOF | oc create -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default 1 data: guestos2common: | "Red Hat Enterprise Linux Server": "rhel" "CentOS Linux": "centos" "Fedora": "fedora" "Ubuntu": "ubuntu" "openSUSE": "opensuse" osinfo2common: | "<rhv-operating-system>": "<vm-template>" 2 EOF
Config map example
$ cat <<EOF | oc apply -f - apiVersion: v1 kind: ConfigMap metadata: name: os-configmap namespace: default data: osinfo2common: | "other_linux": "fedora31" EOF
Verify that the custom config map was created:
$ oc get cm -n default os-configmap -o yaml
Patch the
vm-import-controller-config
config map to apply the new config map:$ oc patch configmap vm-import-controller-config -n openshift-cnv --patch '{ "data": { "osConfigMap.name": "os-configmap", "osConfigMap.namespace": "default" 1 } }'
- 1
- Update the namespace if you changed it in the config map.
Verify that the template appears in the OpenShift Virtualization web console:
- Click Workloads → Virtualization from the side menu.
- Click the Virtual Machine Templates tab and find the template in the list.
7.15.4.5. Troubleshooting a virtual machine import
7.15.4.5.1. Logs
You can check the VM Import Controller pod log for errors.
Procedure
View the VM Import Controller pod name by running the following command:
$ oc get pods -n <namespace> | grep import 1
- 1
- Specify the namespace of your imported virtual machine.
Example output
vm-import-controller-f66f7d-zqkz7 1/1 Running 0 4h49m
View the VM Import Controller pod log by running the following command:
$ oc logs <vm-import-controller-f66f7d-zqkz7> -f -n <namespace> 1
- 1
- Specify the VM Import Controller pod name and the namespace.
7.15.4.5.2. Error messages
The following error message might appear:
The following error message is displayed in the VM Import Controller pod log and the progress bar stops at 10% if the OpenShift Virtualization storage PV is not suitable:
Failed to bind volumes: provisioning failed for PVC
You must use a compatible storage class. The Cinder storage class is not supported.
7.15.4.5.3. Known issues
- If you are using Ceph RBD block-mode volumes and the available storage space is too small for the virtual disk, the import process bar stops at 75% for more than 20 minutes and the migration does not succeed. No error message is displayed in the web console. BZ#1910019
7.15.5. Importing a single VMware virtual machine or template
You can import a VMware vSphere 6.5, 6.7, or 7.0 VM or VM template into OpenShift Virtualization by using the VM Import wizard.
If you import a VM template, OpenShift Virtualization creates a virtual machine based on the template.
7.15.5.1. OpenShift Virtualization storage feature matrix
The following table describes the OpenShift Virtualization storage types that support VM import.
VMware VM import | |
---|---|
OpenShift Container Storage: RBD block-mode volumes | Yes |
OpenShift Virtualization hostpath provisioner | Yes |
Other multi-node writable storage | Yes [1] |
Other single-node writable storage | Yes [2] |
- PVCs must request a ReadWriteMany access mode.
- PVCs must request a ReadWriteOnce access mode.
7.15.5.2. Preparing a VDDK image
The import process uses the VMware Virtual Disk Development Kit (VDDK) to copy the VMware virtual disk.
You can download the VDDK SDK, create a VDDK image, upload the image to an image registry, and add it to the v2v-vmware
config map.
You can configure either an internal OpenShift Container Platform image registry or a secure external image registry for the VDDK image. The registry must be accessible to your OpenShift Virtualization environment.
Storing the VDDK image in a public registry might violate the terms of the VMware license.
7.15.5.2.1. Configuring an internal image registry
You can configure the internal OpenShift Container Platform image registry on bare metal by updating the Image Registry Operator configuration.
You can access the registry directly, from within the OpenShift Container Platform cluster, or externally, by exposing the registry with a route.
Changing the image registry’s management state
To start the image registry, you must change the Image Registry Operator configuration’s managementState
from Removed
to Managed
.
Procedure
Change
managementState
Image Registry Operator configuration fromRemoved
toManaged
. For example:$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
Configuring registry storage for bare metal and other manual installations
As a cluster administrator, following installation you must configure your registry to use storage.
Prerequisites
- Cluster administrator permissions.
- A cluster that uses manually-provisioned Red Hat Enterprise Linux CoreOS (RHCOS) nodes, such as bare metal.
Persistent storage provisioned for your cluster, such as Red Hat OpenShift Container Storage.
ImportantOpenShift Container Platform supports
ReadWriteOnce
access for image registry storage when you have only one replica. To deploy an image registry that supports high availability with two or more replicas,ReadWriteMany
access is required.- Must have 100Gi capacity.
Procedure
To configure your registry to use storage, change the
spec.storage.pvc
in theconfigs.imageregistry/cluster
resource.NoteWhen using shared storage, review your security settings to prevent outside access.
Verify that you do not have a registry pod:
$ oc get pod -n openshift-image-registry -l docker-registry=default
Example output
No resourses found in openshift-image-registry namespace
NoteIf you do have a registry pod in your output, you do not need to continue with this procedure.
Check the registry configuration:
$ oc edit configs.imageregistry.operator.openshift.io
Example output
storage: pvc: claim:
Leave the
claim
field blank to allow the automatic creation of animage-registry-storage
PVC.Check the
clusteroperator
status:$ oc get clusteroperator image-registry
Example output
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE image-registry 4.7 True False False 6h50m
Ensure that your registry is set to managed to enable building and pushing of images.
Run:
$ oc edit configs.imageregistry/cluster
Then, change the line
managementState: Removed
to
managementState: Managed
Accessing registry directly from the cluster
You can access the registry from inside the cluster.
Procedure
Access the registry from the cluster by using internal routes:
Access the node by getting the node’s name:
$ oc get nodes
$ oc debug nodes/<node_name>
To enable access to tools such as
oc
andpodman
on the node, run the following command:sh-4.2# chroot /host
Log in to the container image registry by using your access token:
sh-4.2# oc login -u kubeadmin -p <password_from_install_log> https://api-int.<cluster_name>.<base_domain>:6443
sh-4.2# podman login -u kubeadmin -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000
You should see a message confirming login, such as:
Login Succeeded!
NoteYou can pass any value for the user name; the token contains all necessary information. Passing a user name that contains colons will result in a login failure.
Since the Image Registry Operator creates the route, it will likely be similar to
default-route-openshift-image-registry.<cluster_name>
.Perform
podman pull
andpodman push
operations against your registry:ImportantYou can pull arbitrary images, but if you have the system:registry role added, you can only push images to the registry in your project.
In the following examples, use:
Component Value <registry_ip>
172.30.124.220
<port>
5000
<project>
openshift
<image>
image
<tag>
omitted (defaults to
latest
)Pull an arbitrary image:
sh-4.2# podman pull name.io/image
Tag the new image with the form
<registry_ip>:<port>/<project>/<image>
. The project name must appear in this pull specification for OpenShift Container Platform to correctly place and later access the image in the registry:sh-4.2# podman tag name.io/image image-registry.openshift-image-registry.svc:5000/openshift/image
NoteYou must have the
system:image-builder
role for the specified project, which allows the user to write or push an image. Otherwise, thepodman push
in the next step will fail. To test, you can create a new project to push the image.Push the newly tagged image to your registry:
sh-4.2# podman push image-registry.openshift-image-registry.svc:5000/openshift/image
Exposing a secure registry manually
Instead of logging in to the OpenShift Container Platform registry from within the cluster, you can gain external access to it by exposing it with a route. This allows you to log in to the registry from outside the cluster using the route address, and to tag and push images to an existing project by using the route host.
Prerequisites:
The following prerequisites are automatically performed:
- Deploy the Registry Operator.
- Deploy the Ingress Operator.
Procedure
You can expose the route by using DefaultRoute
parameter in the configs.imageregistry.operator.openshift.io
resource or by using custom routes.
To expose the registry using DefaultRoute
:
Set
DefaultRoute
toTrue
:$ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
Log in with
podman
:$ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
$ podman login -u kubeadmin -p $(oc whoami -t) --tls-verify=false $HOST 1
- 1
--tls-verify=false
is needed if the cluster’s default certificate for routes is untrusted. You can set a custom, trusted certificate as the default certificate with the Ingress Operator.
To expose the registry using custom routes:
Create a secret with your route’s TLS keys:
$ oc create secret tls public-route-tls \ -n openshift-image-registry \ --cert=</path/to/tls.crt> \ --key=</path/to/tls.key>
This step is optional. If you do not create a secret, the route uses the default TLS configuration from the Ingress Operator.
On the Registry Operator:
spec: routes: - name: public-routes hostname: myregistry.mycorp.organization secretName: public-route-tls ...
NoteOnly set
secretName
if you are providing a custom TLS configuration for the registry’s route.
7.15.5.2.2. Configuring an external image registry
If you use an external image registry for the VDDK image, you can add the external image registry’s certificate authorities to the OpenShift Container Platform cluster.
Optionally, you can create a pull secret from your Docker credentials and add it to your service account.
Adding certificate authorities to the cluster
You can add certificate authorities (CA) to the cluster for use when pushing and pulling images with the following procedure.
Prerequisites
- You must have cluster administrator privileges.
-
You must have access to the public certificates of the registry, usually a
hostname/ca.crt
file located in the/etc/docker/certs.d/
directory.
Procedure
Create a
ConfigMap
in theopenshift-config
namespace containing the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure the key in theConfigMap
is the hostname of the registry in thehostname[..port]
format:$ oc create configmap registry-cas -n openshift-config \ --from-file=myregistry.corp.com..5000=/etc/docker/certs.d/myregistry.corp.com:5000/ca.crt \ --from-file=otherregistry.com=/etc/docker/certs.d/otherregistry.com/ca.crt
Update the cluster image configuration:
$ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
Allowing pods to reference images from other secured registries
The .dockercfg
$HOME/.docker/config.json
file for Docker clients is a Docker credentials file that stores your authentication information if you have previously logged into a secured or insecure registry.
To pull a secured container image that is not from OpenShift Container Platform’s internal registry, you must create a pull secret from your Docker credentials and add it to your service account.
Procedure
If you already have a
.dockercfg
file for the secured registry, you can create a secret from that file by running:$ oc create secret generic <pull_secret_name> \ --from-file=.dockercfg=<path/to/.dockercfg> \ --type=kubernetes.io/dockercfg
Or if you have a
$HOME/.docker/config.json
file:$ oc create secret generic <pull_secret_name> \ --from-file=.dockerconfigjson=<path/to/.docker/config.json> \ --type=kubernetes.io/dockerconfigjson
If you do not already have a Docker credentials file for the secured registry, you can create a secret by running:
$ oc create secret docker-registry <pull_secret_name> \ --docker-server=<registry_server> \ --docker-username=<user_name> \ --docker-password=<password> \ --docker-email=<email>
To use a secret for pulling images for pods, you must add the secret to your service account. The name of the service account in this example should match the name of the service account the pod uses. The default service account is
default
:$ oc secrets link default <pull_secret_name> --for=pull
7.15.5.2.3. Creating and using a VDDK image
You can download the VMware Virtual Disk Development Kit (VDDK), build a VDDK image, and push the VDDK image to your image registry. You then add the VDDK image to the v2v-vmware
config map.
Prerequisites
- You must have access to an OpenShift Container Platform internal image registry or a secure external registry.
Procedure
Create and navigate to a temporary directory:
$ mkdir /tmp/<dir_name> && cd /tmp/<dir_name>
- In a browser, navigate to VMware code and click SDKs.
- Under Compute Virtualization, click Virtual Disk Development Kit (VDDK).
- Select the VDDK version that corresponds to your VMware vSphere version, for example, VDDK 7.0 for vSphere 7.0, click Download, and then save the VDDK archive in the temporary directory.
Extract the VDDK archive:
$ tar -xzf VMware-vix-disklib-<version>.x86_64.tar.gz
Create a
Dockerfile
:$ cat > Dockerfile <<EOF FROM busybox:latest COPY vmware-vix-disklib-distrib /vmware-vix-disklib-distrib RUN mkdir -p /opt ENTRYPOINT ["cp", "-r", "/vmware-vix-disklib-distrib", "/opt"] EOF
Build the image:
$ podman build . -t <registry_route_or_server_path>/vddk:<tag> 1
- 1
- Specify your image registry:
-
For an internal OpenShift Container Platform registry, use the internal registry route, for example,
image-registry.openshift-image-registry.svc:5000/openshift/vddk:<tag>
. -
For an external registry, specify the server name, path, and tag, for example,
server.example.com:5000/vddk:<tag>
.
-
For an internal OpenShift Container Platform registry, use the internal registry route, for example,
Push the image to the registry:
$ podman push <registry_route_or_server_path>/vddk:<tag>
- Ensure that the image is accessible to your OpenShift Virtualization environment.
Edit the
v2v-vmware
config map in the openshift-cnv project:$ oc edit configmap v2v-vmware -n openshift-cnv
Add the
vddk-init-image
parameter to thedata
stanza:... data: vddk-init-image: <registry_route_or_server_path>/vddk:<tag>
7.15.5.3. Importing a virtual machine with the VM Import wizard
You can import a single virtual machine with the VM Import wizard.
You can also import a VM template. If you import a VM template, OpenShift Virtualization creates a virtual machine based on the template.
Prerequisites
- You must have admin user privileges.
- The VMware Virtual Disk Development Kit (VDDK) image must be in an image registry that is accessible to your OpenShift Virtualization environment.
-
The VDDK image must be added to the
v2v-vmware
config map. - The VM must be powered off.
- Virtual disks must be connected to IDE or SCSI controllers. If virtual disks are connected to a SATA controller, you can change them to IDE controllers and then migrate the VM.
- The OpenShift Virtualization local and shared persistent storage classes must support VM import.
The OpenShift Virtualization storage must be large enough to accommodate the virtual disk.
WarningIf you are using Ceph RBD block-mode volumes, the storage must be large enough to accommodate the virtual disk. If the disk is too large for the available storage, the import process fails and the PV that is used to copy the virtual disk is not released. You will not be able to import another virtual machine or to clean up the storage because there are insufficient resources to support object deletion. To resolve this situation, you must add more object storage devices to the storage back end.
The OpenShift Virtualization egress network policy must allow the following traffic:
Destination Protocol Port VMware ESXi hosts
TCP
443
VMware ESXi hosts
TCP
902
VMware vCenter
TCP
5840
Procedure
- In the web console, click Workloads → Virtual Machines.
- Click Create Virtual Machine and select Import with Wizard.
- Select VMware from the Provider list.
Select Connect to New Instance or a saved vCenter instance.
- If you select Connect to New Instance, enter the vCenter hostname, Username, and Password.
- If you select a saved vCenter instance, the wizard connects to the vCenter instance using the saved credentials.
Click Check and Save and wait for the connection to complete.
NoteThe connection details are stored in a secret. If you add a provider with an incorrect hostname, user name, or password, click Workloads → Secrets and delete the provider secret.
- Select a virtual machine or a template.
- Click Next.
- In the Review screen, review your settings.
Click Edit to update the following settings:
General:
- Description
- Operating System
- Flavor
- Memory
- CPUs
- Workload Profile
Networking:
- Name
- Model
- Network
- Type
- MAC Address
Storage: Click the Options menu of the VM disk and select Edit to update the following fields:
- Name
- Source: For example, Import Disk.
- Size
- Interface
Storage Class: Select NFS or ocs-storagecluster-ceph-rbd (ceph-rbd).
If you select ocs-storagecluster-ceph-rbd, you must set the Volume Mode of the disk to Block.
Other storage classes might work, but they are not officially supported.
- Advanced → Volume Mode: Select Block.
- Advanced → Access Mode
Advanced → Cloud-init:
- Form: Enter the Hostname and Authenticated SSH Keys.
-
Custom script: Enter the
cloud-init
script in the text field.
- Advanced → Virtual Hardware: You can attach a virtual CD-ROM to the imported virtual machine.
Click Import or Review and Import, if you have edited the import settings.
A Successfully created virtual machine message and a list of resources created for the virtual machine are displayed. The virtual machine appears in Workloads → Virtual Machines.
Virtual machine wizard fields
Name | Parameter | Description |
---|---|---|
Name |
The name can contain lowercase letters ( | |
Description | Optional description field. | |
Operating System | The primary operating system that is selected for the virtual machine in the template. You cannot edit this field when creating a virtual machine from a template. | |
Boot Source | Import via URL (creates PVC) | Import content from an image available from an HTTP or HTTPS endpoint. Example: Obtaining a URL link from the web page with the operating system image. |
Clone existing PVC (creates PVC) | Select an existent persistent volume claim available on the cluster and clone it. | |
Import via Registry (creates PVC) |
Provision virtual machine from a bootable operating system container located in a registry accessible from the cluster. Example: | |
PXE (network boot - adds network interface) | Boot an operating system from a server on the network. Requires a PXE bootable network attachment definition. | |
Persistent Volume Claim project | Project name that you want to use for cloning the PVC. | |
Persistent Volume Claim name | PVC name that should apply to this virtual machine template if you are cloning an existing PVC. | |
Mount this as a CD-ROM boot source | A CD-ROM requires an additional disk for installing the operating system. Select the checkbox to add a disk and customize it later. | |
Flavor | Tiny, Small, Medium, Large, Custom | Presets that determine the amount of CPU and memory allocated to the virtual machine. The presets displayed for Flavor are determined by the operating system. |
Workload Type | Desktop | A virtual machine configuration for use on a desktop. Ideal for consumption on a small scale. Recommended for use with the web console. |
Server | Balances performance and it is compatible with a wide range of server workloads. | |
High-Performance | A virtual machine configuration that is optimized for high-performance workloads. | |
Start this virtual machine after creation. | This checkbox is selected by default and the virtual machine starts running after creation. Clear the checkbox if you do not want the virtual machine to start when it is created. |
Cloud-init fields
Name | Description |
---|---|
Hostname | Sets a specific hostname for the virtual machine. |
Authorized SSH Keys | The user’s public key that is copied to ~/.ssh/authorized_keys on the virtual machine. |
Custom script | Replaces other options with a field in which you paste a custom cloud-init script. |
Networking fields
Name | Description |
---|---|
Name | Name for the network interface controller. |
Model | Indicates the model of the network interface controller. Supported values are e1000e and virtio. |
Network | List of available network attachment definitions. |
Type |
List of available binding methods. For the default pod network, |
MAC Address | MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. |
Storage fields
Name | Selection | Description |
---|---|---|
Source | Blank (creates PVC) | Create an empty disk. |
Import via URL (creates PVC) | Import content via URL (HTTP or HTTPS endpoint). | |
Use an existing PVC | Use a PVC that is already available in the cluster. | |
Clone existing PVC (creates PVC) | Select an existing PVC available in the cluster and clone it. | |
Import via Registry (creates PVC) | Import content via container registry. | |
Container (ephemeral) | Upload content from a container located in a registry accessible from the cluster. The container disk should be used only for read-only filesystems such as CD-ROMs or temporary virtual machines. | |
Name |
Name of the disk. The name can contain lowercase letters ( | |
Size | Size of the disk in GiB. | |
Type | Type of disk. Example: Disk or CD-ROM | |
Interface | Type of disk device. Supported interfaces are virtIO, SATA, and SCSI. | |
Storage Class | The storage class that is used to create the disk. | |
Advanced → Volume Mode | Defines whether the persistent volume uses a formatted file system or raw block state. Default is Filesystem. | |
Advanced → Access Mode | Access mode of the persistent volume. Supported access modes are Single User (RWO), Shared Access (RWX), and Read Only (ROX). |
Advanced storage settings
The following advanced storage settings are available for Blank, Import via URL, and Clone existing PVC disks. These parameters are optional. If you do not specify these parameters, the system uses the default values from the kubevirt-storage-class-defaults
config map.
Name | Parameter | Description |
---|---|---|
Volume Mode | Filesystem | Stores the virtual disk on a file system-based volume. |
Block |
Stores the virtual disk directly on the block volume. Only use | |
Access Mode | Single User (RWO) | The disk can be mounted as read/write by a single node. |
Shared Access (RWX) | The disk can be mounted as read/write by many nodes. Note This is required for some features, such as live migration of virtual machines between nodes. | |
Read Only (ROX) | The disk can be mounted as read-only by many nodes. |
7.15.5.3.1. Updating the NIC name of an imported virtual machine
You must update the NIC name of a virtual machine that you imported from VMware to conform to OpenShift Virtualization naming conventions.
Procedure
- Log in to the virtual machine.
-
Navigate to the
/etc/sysconfig/network-scripts
directory. Rename the network configuration file:
$ mv vmnic0 ifcfg-eth0 1
- 1
- The first network configuration file is named
ifcfg-eth0
. Additional network configuration files are numbered sequentially, for example,ifcfg-eth1
,ifcfg-eth2
.
Update the
NAME
andDEVICE
parameters in the network configuration file:NAME=eth0 DEVICE=eth0
Restart the network:
$ systemctl restart network
7.15.5.4. Troubleshooting a virtual machine import
7.15.5.4.1. Logs
You can check the V2V Conversion pod log for errors.
Procedure
View the V2V Conversion pod name by running the following command:
$ oc get pods -n <namespace> | grep v2v 1
- 1
- Specify the namespace of your imported virtual machine.
Example output
kubevirt-v2v-conversion-f66f7d-zqkz7 1/1 Running 0 4h49m
View the V2V Conversion pod log by running the following command:
$ oc logs <kubevirt-v2v-conversion-f66f7d-zqkz7> -f -n <namespace> 1
- 1
- Specify the VM Conversion pod name and the namespace.
7.15.5.4.2. Error messages
The following error messages might appear:
If the VMware VM is not shut down before import, the imported virtual machine displays the error message,
Readiness probe failed
in the OpenShift Container Platform console and the V2V Conversion pod log displays the following error message:INFO - have error: ('virt-v2v error: internal error: invalid argument: libvirt domain ‘v2v_migration_vm_1’ is running or paused. It must be shut down in order to perform virt-v2v conversion',)"
The following error message is displayed in the OpenShift Container Platform console if a non-admin user tries to import a VM:
Could not load config map vmware-to-kubevirt-os in kube-public namespace Restricted Access: configmaps "vmware-to-kubevirt-os" is forbidden: User cannot get resource "configmaps" in API group "" in the namespace "kube-public"
Only an admin user can import a VM.
7.16. Cloning virtual machines
7.16.1. Enabling user permissions to clone data volumes across namespaces
The isolating nature of namespaces means that users cannot by default clone resources between namespaces.
To enable a user to clone a virtual machine to another namespace, a user with the cluster-admin
role must create a new cluster role. Bind this cluster role to a user to enable them to clone virtual machines to the destination namespace.
7.16.1.1. Prerequisites
-
Only a user with the
cluster-admin
role can create cluster roles.
7.16.1.2. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.16.1.3. Creating RBAC resources for cloning data volumes
Create a new cluster role that enables permissions for all actions for the datavolumes
resource.
Procedure
Create a
ClusterRole
manifest:apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: <datavolume-cloner> 1 rules: - apiGroups: ["cdi.kubevirt.io"] resources: ["datavolumes/source"] verbs: ["*"]
- 1
- Unique name for the cluster role.
Create the cluster role in the cluster:
$ oc create -f <datavolume-cloner.yaml> 1
- 1
- The file name of the
ClusterRole
manifest created in the previous step.
Create a
RoleBinding
manifest that applies to both the source and destination namespaces and references the cluster role created in the previous step.apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: <allow-clone-to-user> 1 namespace: <Source namespace> 2 subjects: - kind: ServiceAccount name: default namespace: <Destination namespace> 3 roleRef: kind: ClusterRole name: datavolume-cloner 4 apiGroup: rbac.authorization.k8s.io
Create the role binding in the cluster:
$ oc create -f <datavolume-cloner.yaml> 1
- 1
- The file name of the
RoleBinding
manifest created in the previous step.
7.16.2. Cloning a virtual machine disk into a new data volume
You can clone the persistent volume claim (PVC) of a virtual machine disk into a new data volume by referencing the source PVC in your data volume configuration file.
Cloning operations between different volume modes are not supported. The volumeMode
values must match in both the source and target specifications.
For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block
to a PV with volumeMode: Filesystem
, the operation fails with an error message.
7.16.2.1. Prerequisites
- Users need additional permissions to clone the PVC of a virtual machine disk into another namespace.
7.16.2.2. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.16.2.3. Cloning the persistent volume claim of a virtual machine disk into a new data volume
You can clone a persistent volume claim (PVC) of an existing virtual machine disk into a new data volume. The new data volume can then be used for a new virtual machine.
When a data volume is created independently of a virtual machine, the lifecycle of the data volume is independent of the virtual machine. If the virtual machine is deleted, neither the data volume nor its associated PVC is deleted.
Prerequisites
- Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it.
-
Install the OpenShift CLI (
oc
).
Procedure
- Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC.
Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC, and the size of the new data volume.
For example:
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4
Start cloning the PVC by creating the data volume:
$ oc create -f <cloner-datavolume>.yaml
NoteData volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones.
7.16.2.4. Template: Data volume clone configuration file
example-clone-dv.yaml
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: "example-clone-dv" spec: source: pvc: name: source-pvc namespace: example-ns pvc: accessModes: - ReadWriteOnce resources: requests: storage: "1G"
7.16.2.5. CDI supported operations matrix
This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space.
Content types | HTTP | HTTPS | HTTP basic auth | Registry | Upload |
---|---|---|---|---|---|
KubeVirt(QCOW2) |
✓ QCOW2 |
✓ QCOW2** |
✓ QCOW2 |
✓ QCOW2* |
✓ QCOW2* |
KubeVirt (RAW) |
✓ RAW |
✓ RAW |
✓ RAW |
✓ RAW* |
✓ RAW* |
✓ Supported operation
□ Unsupported operation
* Requires scratch space
** Requires scratch space if a custom certificate authority is required
7.16.3. Cloning a virtual machine by using a data volume template
You can create a new virtual machine by cloning the persistent volume claim (PVC) of an existing VM. By including a dataVolumeTemplate
in your virtual machine configuration file, you create a new data volume from the original PVC.
Cloning operations between different volume modes are not supported. The volumeMode
values must match in both the source and target specifications.
For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block
to a PV with volumeMode: Filesystem
, the operation fails with an error message.
7.16.3.1. Prerequisites
- Users need additional permissions to clone the PVC of a virtual machine disk into another namespace.
7.16.3.2. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.16.3.3. Creating a new virtual machine from a cloned persistent volume claim by using a data volume template
You can create a virtual machine that clones the persistent volume claim (PVC) of an existing virtual machine into a data volume. Reference a dataVolumeTemplate
in the virtual machine manifest and the source
PVC is cloned to a data volume, which is then automatically used for the creation of the virtual machine.
When a data volume is created as part of the data volume template of a virtual machine, the lifecycle of the data volume is then dependent on the virtual machine. If the virtual machine is deleted, the data volume and associated PVC are also deleted.
Prerequisites
- Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it.
-
Install the OpenShift CLI (
oc
).
Procedure
- Examine the virtual machine you want to clone to identify the name and namespace of the associated PVC.
Create a YAML file for a
VirtualMachine
object. The following virtual machine example clonesmy-favorite-vm-disk
, which is located in thesource-namespace
namespace. The2Gi
data volume calledfavorite-clone
is created frommy-favorite-vm-disk
.For example:
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: kubevirt.io/vm: vm-dv-clone name: vm-dv-clone 1 spec: running: false template: metadata: labels: kubevirt.io/vm: vm-dv-clone spec: domain: devices: disks: - disk: bus: virtio name: root-disk resources: requests: memory: 64M volumes: - dataVolume: name: favorite-clone name: root-disk dataVolumeTemplates: - metadata: name: favorite-clone spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi source: pvc: namespace: "source-namespace" name: "my-favorite-vm-disk"
- 1
- The virtual machine to create.
Create the virtual machine with the PVC-cloned data volume:
$ oc create -f <vm-clone-datavolumetemplate>.yaml
7.16.3.4. Template: Data volume virtual machine configuration file
example-dv-vm.yaml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: example-vm
name: example-vm
spec:
dataVolumeTemplates:
- metadata:
name: example-dv
spec:
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1G
source:
http:
url: "" 1
running: false
template:
metadata:
labels:
kubevirt.io/vm: example-vm
spec:
domain:
cpu:
cores: 1
devices:
disks:
- disk:
bus: virtio
name: example-dv-disk
machine:
type: q35
resources:
requests:
memory: 1G
terminationGracePeriodSeconds: 0
volumes:
- dataVolume:
name: example-dv
name: example-dv-disk
- 1
- The
HTTP
source of the image you want to import, if applicable.
7.16.3.5. CDI supported operations matrix
This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space.
Content types | HTTP | HTTPS | HTTP basic auth | Registry | Upload |
---|---|---|---|---|---|
KubeVirt(QCOW2) |
✓ QCOW2 |
✓ QCOW2** |
✓ QCOW2 |
✓ QCOW2* |
✓ QCOW2* |
KubeVirt (RAW) |
✓ RAW |
✓ RAW |
✓ RAW |
✓ RAW* |
✓ RAW* |
✓ Supported operation
□ Unsupported operation
* Requires scratch space
** Requires scratch space if a custom certificate authority is required
7.16.4. Cloning a virtual machine disk into a new block storage data volume
You can clone the persistent volume claim (PVC) of a virtual machine disk into a new block data volume by referencing the source PVC in your data volume configuration file.
Cloning operations between different volume modes are not supported. The volumeMode
values must match in both the source and target specifications.
For example, if you attempt to clone from a persistent volume (PV) with volumeMode: Block
to a PV with volumeMode: Filesystem
, the operation fails with an error message.
7.16.4.1. Prerequisites
- Users need additional permissions to clone the PVC of a virtual machine disk into another namespace.
7.16.4.2. About data volumes
DataVolume
objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OpenShift Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared.
7.16.4.3. About block persistent volumes
A block persistent volume (PV) is a PV that is backed by a raw block device. These volumes do not have a file system and can provide performance benefits for virtual machines by reducing overhead.
Raw block volumes are provisioned by specifying volumeMode: Block
in the PV and persistent volume claim (PVC) specification.
7.16.4.4. Creating a local block persistent volume
Create a local block persistent volume (PV) on a node by populating a file and mounting it as a loop device. You can then reference this loop device in a PV manifest as a Block
volume and use it as a block device for a virtual machine image.
Procedure
-
Log in as
root
to the node on which to create the local PV. This procedure usesnode01
for its examples. Create a file and populate it with null characters so that it can be used as a block device. The following example creates a file
loop10
with a size of 2Gb (20 100Mb blocks):$ dd if=/dev/zero of=<loop10> bs=100M count=20
Mount the
loop10
file as a loop device.$ losetup </dev/loop10>d3 <loop10> 1 2
Create a
PersistentVolume
manifest that references the mounted loop device.kind: PersistentVolume apiVersion: v1 metadata: name: <local-block-pv10> annotations: spec: local: path: </dev/loop10> 1 capacity: storage: <2Gi> volumeMode: Block 2 storageClassName: local 3 accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Delete nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <node01> 4
Create the block PV.
# oc create -f <local-block-pv10.yaml>1
- 1
- The file name of the persistent volume created in the previous step.
7.16.4.5. Cloning the persistent volume claim of a virtual machine disk into a new data volume
You can clone a persistent volume claim (PVC) of an existing virtual machine disk into a new data volume. The new data volume can then be used for a new virtual machine.
When a data volume is created independently of a virtual machine, the lifecycle of the data volume is independent of the virtual machine. If the virtual machine is deleted, neither the data volume nor its associated PVC is deleted.
Prerequisites
- Determine the PVC of an existing virtual machine disk to use. You must power down the virtual machine that is associated with the PVC before you can clone it.
-
Install the OpenShift CLI (
oc
). - At least one available block persistent volume (PV) that is the same size as or larger than the source PVC.
Procedure
- Examine the virtual machine disk you want to clone to identify the name and namespace of the associated PVC.
Create a YAML file for a data volume that specifies the name of the new data volume, the name and namespace of the source PVC,
volumeMode: Block
so that an available block PV is used, and the size of the new data volume.For example:
apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: <cloner-datavolume> 1 spec: source: pvc: namespace: "<source-namespace>" 2 name: "<my-favorite-vm-disk>" 3 pvc: accessModes: - ReadWriteOnce resources: requests: storage: <2Gi> 4 volumeMode: Block 5
- 1
- The name of the new data volume.
- 2
- The namespace where the source PVC exists.
- 3
- The name of the source PVC.
- 4
- The size of the new data volume. You must allocate enough space, or the cloning operation fails. The size must be the same as or larger than the source PVC.
- 5
- Specifies that the destination is a block PV
Start cloning the PVC by creating the data volume:
$ oc create -f <cloner-datavolume>.yaml
NoteData volumes prevent a virtual machine from starting before the PVC is prepared, so you can create a virtual machine that references the new data volume while the PVC clones.
7.16.4.6. CDI supported operations matrix
This matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space.
Content types | HTTP | HTTPS | HTTP basic auth | Registry | Upload |
---|---|---|---|---|---|
KubeVirt(QCOW2) |
✓ QCOW2 |
✓ QCOW2** |
✓ QCOW2 |
✓ QCOW2* |
✓ QCOW2* |
KubeVirt (RAW) |
✓ RAW |
✓ RAW |
✓ RAW |
✓ RAW* |
✓ RAW* |
✓ Supported operation
□ Unsupported operation
* Requires scratch space
** Requires scratch space if a custom certificate authority is required
7.17. Virtual machine networking
7.17.1. Configuring the virtual machine for the default pod network
You can connect a virtual machine to the default internal pod network by configuring its network interface to use the masquerade
binding mode.
The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. It is not enabled by default. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace.
7.17.1.1. Configuring masquerade mode from the command line
You can use masquerade mode to hide a virtual machine’s outgoing traffic behind the pod IP address. Masquerade mode uses Network Address Translation (NAT) to connect virtual machines to the pod network backend through a Linux bridge.
Enable masquerade mode and allow traffic to enter the virtual machine by editing your virtual machine configuration file.
Prerequisites
- The virtual machine must be configured to use DHCP to acquire IPv4 addresses. The examples below are configured to use DHCP.
Procedure
Edit the
interfaces
spec of your virtual machine configuration file:kind: VirtualMachine spec: domain: devices: interfaces: - name: default masquerade: {} 1 ports: - port: 80 2 networks: - name: default pod: {}
- 1
- Connect using masquerade mode.
- 2
- Optional: List the ports that you want to expose from the virtual machine, each specified by the
port
field. Theport
value must be a number between 0 and 65536. When theports
array is not used, all ports in the valid range are open to incoming traffic. In this example, incoming traffic is allowed on port80
.
NotePorts 49152 and 49153 are reserved for use by the libvirt platform and all other incoming traffic to these ports is dropped.
Create the virtual machine:
$ oc create -f <vm-name>.yaml
7.17.1.2. Creating a service from a virtual machine
Create a service from a running virtual machine by first creating a Service
object to expose the virtual machine.
The ClusterIP
service type exposes the virtual machine internally, within the cluster. The NodePort
or LoadBalancer
service types expose the virtual machine externally, outside of the cluster.
This procedure presents an example of how to create, connect to, and expose a Service
object of type: ClusterIP
as a virtual machine-backed service.
ClusterIP
is the default service type
, if the service type
is not specified.
Procedure
Edit the virtual machine YAML as follows:
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-ephemeral namespace: example-namespace spec: running: false template: metadata: labels: special: key 1 spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin
- 1
- Add the label
special: key
in thespec.template.metadata.labels
section.
NoteLabels on a virtual machine are passed through to the pod. The labels on the
VirtualMachine
configuration, for examplespecial: key
, must match the labels in theService
YAMLselector
attribute, which you create later in this procedure.- Save the virtual machine YAML to apply your changes.
Edit the
Service
YAML to configure the settings necessary to create and expose theService
object:apiVersion: v1 kind: Service metadata: name: vmservice 1 namespace: example-namespace 2 spec: ports: - port: 27017 protocol: TCP targetPort: 22 3 selector: special: key 4 type: ClusterIP 5
- 1
- Specify the
name
of the service you are creating and exposing. - 2
- Specify
namespace
in themetadata
section of theService
YAML that corresponds to thenamespace
you specify in the virtual machine YAML. - 3
- Add
targetPort: 22
, exposing the service on SSH port22
. - 4
- In the
spec
section of theService
YAML, addspecial: key
to theselector
attribute, which corresponds to thelabels
you added in the virtual machine YAML configuration file. - 5
- In the
spec
section of theService
YAML, addtype: ClusterIP
for a ClusterIP service. To create and expose other types of services externally, outside of the cluster, such asNodePort
andLoadBalancer
, replacetype: ClusterIP
withtype: NodePort
ortype: LoadBalancer
, as appropriate.
-
Save the
Service
YAML to store the service configuration. Create the
ClusterIP
service:$ oc create -f <service_name>.yaml
- Start the virtual machine. If the virtual machine is already running, restart it.
Query the
Service
object to verify it is available and is configured with typeClusterIP
.Verification
Run the
oc get service
command, specifying thenamespace
that you reference in the virtual machine andService
YAML files.$ oc get service -n example-namespace
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vmservice ClusterIP 172.30.3.149 <none> 27017/TCP 2m
-
As shown from the output,
vmservice
is running. -
The
TYPE
displays asClusterIP
, as you specified in theService
YAML.
-
As shown from the output,
Establish a connection to the virtual machine that you want to use to back your service. Connect from an object inside the cluster, such as another virtual machine.
Edit the virtual machine YAML as follows:
apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: vm-connect namespace: example-namespace spec: running: false template: spec: domain: devices: disks: - name: containerdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - masquerade: {} name: default resources: requests: memory: 1024M networks: - name: default pod: {} volumes: - name: containerdisk containerDisk: image: kubevirt/fedora-cloud-container-disk-demo - name: cloudinitdisk cloudInitNoCloud: userData: | #!/bin/bash echo "fedora" | passwd fedora --stdin
Run the
oc create
command to create a second virtual machine, wherefile.yaml
is the name of the virtual machine YAML:$ oc create -f <file.yaml>
- Start the virtual machine.
Connect to the virtual machine by running the following
virtctl
command:$ virtctl -n example-namespace console <new-vm-name>
NoteFor service type
LoadBalancer
, use thevinagre
client to connect your virtual machine by using the public IP and port. External ports are dynamically allocated when using service typeLoadBalancer
.Run the
ssh
command to authenticate the connection, where172.30.3.149
is the ClusterIP of the service andfedora
is the user name of the virtual machine:$ ssh fedora@172.30.3.149 -p 27017
Verification
- You receive the command prompt of the virtual machine backing the service you want to expose. You now have a service backed by a running virtual machine.
7.17.2. Attaching a virtual machine to a Linux bridge network
By default, OpenShift Virtualization is installed with a single, internal pod network.
You must create a Linux bridge network attachment definition (NAD) in order to connect to additional networks.
To attach a virtual machine to an additional network:
- Create a Linux bridge node network configuration policy.
- Create a Linux bridge network attachment definition.
- Configure the virtual machine, enabling the virtual machine to recognize the network attachment definition.
For more information about scheduling, interface types, and other node networking activities, see the node networking section.
7.17.2.1. Connecting to the network through the network attachment definition
7.17.2.1.1. Creating a Linux bridge node network configuration policy
Use a NodeNetworkConfigurationPolicy
manifest YAML file to create the Linux bridge.
Procedure
Create the
NodeNetworkConfigurationPolicy
manifest. This example includes sample values that you must replace with your own information.apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br1-eth1-policy 1 spec: desiredState: interfaces: - name: br1 2 description: Linux bridge with eth1 as a port 3 type: linux-bridge 4 state: up 5 ipv4: enabled: false 6 bridge: options: stp: enabled: false 7 port: - name: eth1 8
- 1
- Name of the policy.
- 2
- Name of the interface.
- 3
- Optional: Human-readable description of the interface.
- 4
- The type of interface. This example creates a bridge.
- 5
- The requested state for the interface after creation.
- 6
- Disables IPv4 in this example.
- 7
- Disables STP in this example.
- 8
- The node NIC to which the bridge is attached.
7.17.2.2. Creating a Linux bridge network attachment definition
7.17.2.2.1. Prerequisites
- A Linux bridge must be configured and attached on every node. See the node networking section for more information.
Configuring IP address management (IPAM) in a network attachment definition for virtual machines is not supported.
7.17.2.2.2. Creating a Linux bridge network attachment definition in the web console
The network attachment definition is a custom resource that exposes layer-2 devices to a specific namespace in your OpenShift Virtualization cluster.
Network administrators can create network attachment definitions to provide existing layer-2 networking to pods and virtual machines.
Procedure
- In the web console, click Networking → Network Attachment Definitions.
Click Create Network Attachment Definition.
NoteThe network attachment definition must be in the same namespace as the pod or virtual machine.
- Enter a unique Name and optional Description.
- Click the Network Type list and select CNV Linux bridge.
- Enter the name of the bridge in the Bridge Name field.
- Optional: If the resource has VLAN IDs configured, enter the ID numbers in the VLAN Tag Number field.
- Optional: Select MAC Spoof Check to enable MAC spoof filtering. This feature provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod.
Click Create.
NoteA Linux bridge network attachment definition is the most efficient method for connecting a virtual machine to a VLAN.
7.17.2.2.3. Creating a Linux bridge network attachment definition in the CLI
As a network administrator, you can configure a network attachment definition of type cnv-bridge
to provide layer-2 networking to pods and virtual machines.
Prerequisites
-
The node must support nftables and the
nft
binary must be deployed to enable MAC spoof check.
Procedure
- Create a network attachment definition in the same namespace as the virtual machine.
Add the virtual machine to the network attachment definition, as in the following example:
apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: <bridge-network> 1 annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/<bridge-interface> 2 spec: config: '{ "cniVersion": "0.3.1", "name": "<bridge-network>", 3 "type": "cnv-bridge", 4 "bridge": "<bridge-interface>", 5 "macspoofchk": true, 6 "vlan": 1 7 }'
- 1
- The name for the
NetworkAttachmentDefinition
object. - 2
- Optional: Annotation key-value pair for node selection, where
bridge-interface
must match the name of a bridge configured on some nodes. If you add this annotation to your network attachment definition, your virtual machine instances will only run on the nodes that have thebridge-interface
bridge connected. - 3
- The name for the configuration. It is recommended to match the configuration name to the
name
value of the network attachment definition. - 4
- The actual name of the Container Network Interface (CNI) plug-in that provides the network for this network attachment definition. Do not change this field unless you want to use a different CNI.
- 5
- The name of the Linux bridge configured on the node.
- 6
- Optional: Flag to enable MAC spoof check. When set to
true
, you cannot change the MAC address of the pod or guest interface. This attribute provides security against a MAC spoofing attack by allowing only a single MAC address to exit the pod. - 7
- Optional: The VLAN tag. No additional VLAN configuration is required on the node network configuration policy.
NoteA Linux bridge network attachment definition is the most efficient method for connecting a virtual machine to a VLAN.
Create the network attachment definition:
$ oc create -f <network-attachment-definition.yaml> 1
- 1
- Where
<network-attachment-definition.yaml>
is the file name of the network attachment definition manifest.
Verification
Verify that the network attachment definition was created by running the following command:
$ oc get network-attachment-definition <bridge-network>
7.17.2.3. Configuring the virtual machine for a Linux bridge network
7.17.2.3.1. Creating a NIC for a virtual machine in the web console
Create and attach additional NICs to a virtual machine from the web console.
Procedure
- In the correct project in the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine to open the Virtual Machine Overview screen.
- Click Network Interfaces to display the NICs already attached to the virtual machine.
- Click Add Network Interface to create a new slot in the list.
- Use the Network drop-down list to select the network attachment definition for the additional network.
- Fill in the Name, Model, Type, and MAC Address for the new NIC.
- Click Add to save and attach the NIC to the virtual machine.
7.17.2.3.2. Networking fields
Name | Description |
---|---|
Name | Name for the network interface controller. |
Model | Indicates the model of the network interface controller. Supported values are e1000e and virtio. |
Network | List of available network attachment definitions. |
Type |
List of available binding methods. For the default pod network, |
MAC Address | MAC address for the network interface controller. If a MAC address is not specified, one is assigned automatically. |
7.17.2.3.3. Attaching a virtual machine to an additional network in the CLI
Attach a virtual machine to an additional network by adding a bridge interface and specifying a network attachment definition in the virtual machine configuration.
This procedure uses a YAML file to demonstrate editing the configuration and applying the updated file to the cluster. You can alternatively use the oc edit <object> <name>
command to edit an existing virtual machine.
Prerequisites
- Shut down the virtual machine before editing the configuration. If you edit a running virtual machine, you must restart the virtual machine for the changes to take effect.
Procedure
- Create or edit a configuration of a virtual machine that you want to connect to the bridge network.
Add the bridge interface to the
spec.template.spec.domain.devices.interfaces
list and the network attachment definition to thespec.template.spec.networks
list. This example adds a bridge interface calledbridge-net
that connects to thea-bridge-network
network attachment definition:apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: name: <example-vm> spec: template: spec: domain: devices: interfaces: - masquerade: {} name: <default> - bridge: {} name: <bridge-net> 1 ... networks: - name: <default> pod: {} - name: <bridge-net> 2 multus: networkName: <a-bridge-network> 3 ...
- 1
- The name of the bridge interface.
- 2
- The name of the network. This value must match the
name
value of the correspondingspec.template.spec.domain.devices.interfaces
entry. - 3
- The name of the network attachment definition, prefixed by the namespace where it exists. The namespace must be either the
default
namespace or the same namespace where the VM is to be created. In this case,multus
is used. Multus is a cloud network interface (CNI) plug-in that allows multiple CNIs to exist so that a pod or virtual machine can use the interfaces it needs.
Apply the configuration:
$ oc apply -f <example-vm.yaml>
- Optional: If you edited a running virtual machine, you must restart it for the changes to take effect.
7.17.3. Configuring IP addresses for virtual machines
You can configure either dynamically or statically provisioned IP addresses for virtual machines.
Prerequisites
- The virtual machine must connect to an external network.
- You must have a DHCP server available on the additional network to configure a dynamic IP for the virtual machine.
7.17.3.1. Configuring an IP address for a new virtual machine using cloud-init
You can use cloud-init to configure an IP address when you create a virtual machine. The IP address can be dynamically or statically provisioned.
Procedure
Create a virtual machine configuration and include the cloud-init network details in the
spec.volumes.cloudInitNoCloud.networkData
field of the virtual machine configuration:To configure a dynamic IP, specify the interface name and the
dhcp4
boolean:kind: VirtualMachine spec: ... volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 dhcp4: true 2
To configure a static IP, specify the interface name and the IP address:
kind: VirtualMachine spec: ... volumes: - cloudInitNoCloud: networkData: | version: 2 ethernets: eth1: 1 addresses: - 10.10.10.14/24 2
7.17.4. Configuring an SR-IOV network device for virtual machines
You can configure a Single Root I/O Virtualization (SR-IOV) device for virtual machines in your cluster. This process is similar but not identical to configuring an SR-IOV device for OpenShift Container Platform.
Live migration is not supported for virtual machines that are attached to an SR-IOV network interface.
7.17.4.1. Prerequisites
- You must have installed the SR-IOV Operator.
- You must have configured the SR-IOV Operator.
7.17.4.2. Automated discovery of SR-IOV network devices
The SR-IOV Network Operator searches your cluster for SR-IOV capable network devices on worker nodes. The Operator creates and updates a SriovNetworkNodeState custom resource (CR) for each worker node that provides a compatible SR-IOV network device.
The CR is assigned the same name as the worker node. The status.interfaces
list provides information about the network devices on a node.
Do not modify a SriovNetworkNodeState
object. The Operator creates and manages these resources automatically.
7.17.4.2.1. Example SriovNetworkNodeState object
The following YAML is an example of a SriovNetworkNodeState
object created by the SR-IOV Network Operator:
An SriovNetworkNodeState object
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState metadata: name: node-25 1 namespace: openshift-sriov-network-operator ownerReferences: - apiVersion: sriovnetwork.openshift.io/v1 blockOwnerDeletion: true controller: true kind: SriovNetworkNodePolicy name: default spec: dpConfigVersion: "39824" status: interfaces: 2 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f0 pciAddress: "0000:18:00.0" totalvfs: 8 vendor: 15b3 - deviceID: "1017" driver: mlx5_core mtu: 1500 name: ens785f1 pciAddress: "0000:18:00.1" totalvfs: 8 vendor: 15b3 - deviceID: 158b driver: i40e mtu: 1500 name: ens817f0 pciAddress: 0000:81:00.0 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens817f1 pciAddress: 0000:81:00.1 totalvfs: 64 vendor: "8086" - deviceID: 158b driver: i40e mtu: 1500 name: ens803f0 pciAddress: 0000:86:00.0 totalvfs: 64 vendor: "8086" syncStatus: Succeeded
7.17.4.3. Configuring SR-IOV network devices
The SR-IOV Network Operator adds the SriovNetworkNodePolicy.sriovnetwork.openshift.io
CustomResourceDefinition to OpenShift Container Platform. You can configure an SR-IOV network device by creating a SriovNetworkNodePolicy custom resource (CR).
When applying the configuration specified in a SriovNetworkNodePolicy
object, the SR-IOV Operator might drain the nodes, and in some cases, reboot nodes.
It might take several minutes for a configuration change to apply.
Prerequisites
-
You installed the OpenShift CLI (
oc
). -
You have access to the cluster as a user with the
cluster-admin
role. - You have installed the SR-IOV Network Operator.
- You have enough available nodes in your cluster to handle the evicted workload from drained nodes.
- You have not selected any control plane nodes for SR-IOV network device configuration.
Procedure
-
Create an
SriovNetworkNodePolicy
object, and then save the YAML in the<name>-sriov-node-network.yaml
file. Replace<name>
with the name for this configuration.
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 nodeSelector: feature.node.kubernetes.io/network-sriov.capable: "true" 4 priority: <priority> 5 mtu: <mtu> 6 numVfs: <num> 7 nicSelector: 8 vendor: "<vendor_code>" 9 deviceID: "<device_id>" 10 pfNames: ["<pf_name>", ...] 11 rootDevices: ["<pci_bus_id>", "..."] 12 deviceType: vfio-pci 13 isRdma: false 14
- 1
- Specify a name for the CR object.
- 2
- Specify the namespace where the SR-IOV Operator is installed.
- 3
- Specify the resource name of the SR-IOV device plug-in. You can create multiple
SriovNetworkNodePolicy
objects for a resource name. - 4
- Specify the node selector to select which nodes are configured. Only SR-IOV network devices on selected nodes are configured. The SR-IOV Container Network Interface (CNI) plug-in and device plug-in are deployed only on selected nodes.
- 5
- Optional: Specify an integer value between
0
and99
. A smaller number gets higher priority, so a priority of10
is higher than a priority of99
. The default value is99
. - 6
- Optional: Specify a value for the maximum transmission unit (MTU) of the virtual function. The maximum MTU value can vary for different NIC models.
- 7
- Specify the number of the virtual functions (VF) to create for the SR-IOV physical network device. For an Intel network interface controller (NIC), the number of VFs cannot be larger than the total VFs supported by the device. For a Mellanox NIC, the number of VFs cannot be larger than
128
. - 8
- The
nicSelector
mapping selects the Ethernet device for the Operator to configure. You do not need to specify values for all the parameters. It is recommended to identify the Ethernet adapter with enough precision to minimize the possibility of selecting an Ethernet device unintentionally. If you specifyrootDevices
, you must also specify a value forvendor
,deviceID
, orpfNames
. If you specify bothpfNames
androotDevices
at the same time, ensure that they point to an identical device. - 9
- Optional: Specify the vendor hex code of the SR-IOV network device. The only allowed values are either
8086
or15b3
. - 10
- Optional: Specify the device hex code of SR-IOV network device. The only allowed values are
158b
,1015
,1017
. - 11
- Optional: The parameter accepts an array of one or more physical function (PF) names for the Ethernet device.
- 12
- The parameter accepts an array of one or more PCI bus addresses for the physical function of the Ethernet device. Provide the address in the following format:
0000:02:00.1
. - 13
- The
vfio-pci
driver type is required for virtual functions in OpenShift Virtualization. - 14
- Optional: Specify whether to enable remote direct memory access (RDMA) mode. For a Mellanox card, set
isRdma
tofalse
. The default value isfalse
.NoteIf
isRDMA
flag is set totrue
, you can continue to use the RDMA enabled VF as a normal network device. A device can be used in either mode.-
Optional: Label the SR-IOV capable cluster nodes with
SriovNetworkNodePolicy.Spec.NodeSelector
if they are not already labeled. For more information about labeling nodes, see "Understanding how to update labels on nodes".
-
Optional: Label the SR-IOV capable cluster nodes with
Create the
SriovNetworkNodePolicy
object:$ oc create -f <name>-sriov-node-network.yaml
where
<name>
specifies the name for this configuration.After applying the configuration update, all the pods in
sriov-network-operator
namespace transition to theRunning
status.To verify that the SR-IOV network device is configured, enter the following command. Replace
<node_name>
with the name of a node with the SR-IOV network device that you just configured.$ oc get sriovnetworknodestates -n openshift-sriov-network-operator <node_name> -o jsonpath='{.status.syncStatus}'
7.17.4.4. Next steps
7.17.5. Defining an SR-IOV network
You can create a network attachment for a Single Root I/O Virtualization (SR-IOV) device for virtual machines.
After the network is defined, you can attach virtual machines to the SR-IOV network.
7.17.5.1. Prerequisites
- You must have configured an SR-IOV device for virtual machines.
7.17.5.2. Configuring SR-IOV additional network
You can configure an additional network that uses SR-IOV hardware by creating a SriovNetwork
object. When you create a SriovNetwork
object, the SR-IOV Operator automatically creates a NetworkAttachmentDefinition
object.
Users can then attach virtual machines to the SR-IOV network by specifying the network in the virtual machine configurations.
Do not modify or delete a SriovNetwork
object if it is attached to any pods or virtual machines in the running
state.
Prerequisites
-
Install the OpenShift CLI (
oc
). -
Log in as a user with
cluster-admin
privileges.
Procedure
-
Create the following
SriovNetwork
object, and then save the YAML in the<name>-sriov-network.yaml
file. Replace<name>
with a name for this additional network.
apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: <name> 1 namespace: openshift-sriov-network-operator 2 spec: resourceName: <sriov_resource_name> 3 networkNamespace: <target_namespace> 4 vlan: <vlan> 5 spoofChk: "<spoof_check>" 6 linkState: <link_state> 7 maxTxRate: <max_tx_rate> 8 minTxRate: <min_rx_rate> 9 vlanQoS: <vlan_qos> 10 trust: "<trust_vf>" 11 capabilities: <capabilities> 12
- 1
- Replace
<name>
with a name for the object. The SR-IOV Network Operator creates aNetworkAttachmentDefinition
object with same name. - 2
- Specify the namespace where the SR-IOV Network Operator is installed.
- 3
- Replace
<sriov_resource_name>
with the value for the.spec.resourceName
parameter from theSriovNetworkNodePolicy
object that defines the SR-IOV hardware for this additional network. - 4
- Replace
<target_namespace>
with the target namespace for the SriovNetwork. Only pods or virtual machines in the target namespace can attach to the SriovNetwork. - 5
- Optional: Replace
<vlan>
with a Virtual LAN (VLAN) ID for the additional network. The integer value must be from0
to4095
. The default value is0
. - 6
- Optional: Replace
<spoof_check>
with the spoof check mode of the VF. The allowed values are the strings"on"
and"off"
.ImportantYou must enclose the value you specify in quotes or the CR is rejected by the SR-IOV Network Operator.
- 7
- Optional: Replace
<link_state>
with the link state of virtual function (VF). Allowed value areenable
,disable
andauto
. - 8
- Optional: Replace
<max_tx_rate>
with a maximum transmission rate, in Mbps, for the VF. - 9
- Optional: Replace
<min_tx_rate>
with a minimum transmission rate, in Mbps, for the VF. This value should always be less than or equal to Maximum transmission rate.NoteIntel NICs do not support the
minTxRate
parameter. For more information, see BZ#1772847. - 10
- Optional: Replace
<vlan_qos>
with an IEEE 802.1p priority level for the VF. The default value is0
. - 11
- Optional: Replace
<trust_vf>
with the trust mode of the VF. The allowed values are the strings"on"
and"off"
.ImportantYou must enclose the value you specify in quotes or the CR is rejected by the SR-IOV Network Operator.
- 12
- Optional: Replace
<capabilities>
with the capabilities to configure for this network.
To create the object, enter the following command. Replace
<name>
with a name for this additional network.$ oc create -f <name>-sriov-network.yaml
Optional: To confirm that the
NetworkAttachmentDefinition
object associated with theSriovNetwork
object that you created in the previous step exists, enter the following command. Replace<namespace>
with the namespace you specified in theSriovNetwork
object.$ oc get net-attach-def -n <namespace>
7.17.5.3. Next steps
7.17.6. Attaching a virtual machine to an SR-IOV network
You can attach a virtual machine to use a Single Root I/O Virtualization (SR-IOV) network as a secondary network.
7.17.6.1. Prerequisites
- You must have configured an SR-IOV device for virtual machines.
- You must have defined an SR-IOV network.
7.17.6.2. Attaching a virtual machine to an SR-IOV network
You can attach the virtual machine to the SR-IOV network by including the network details in the virtual machine configuration.
Procedure
Include the SR-IOV network details in the
spec.domain.devices.interfaces
andspec.networks
of the virtual machine configuration:kind: VirtualMachine ... spec: domain: devices: interfaces: - name: <default> 1 masquerade: {} 2 - name: <nic1> 3 sriov: {} networks: - name: <default> 4 pod: {} - name: <nic1> 5 multus: networkName: <sriov-network> 6 ...
- 1
- A unique name for the interface that is connected to the pod network.
- 2
- The
masquerade
binding to the default pod network. - 3
- A unique name for the SR-IOV interface.
- 4
- The name of the pod network interface. This must be the same as the
interfaces.name
that you defined earlier. - 5
- The name of the SR-IOV interface. This must be the same as the
interfaces.name
that you defined earlier. - 6
- The name of the SR-IOV network attachment definition.
Apply the virtual machine configuration:
$ oc apply -f <vm-sriov.yaml> 1
- 1
- The name of the virtual machine YAML file.
7.17.7. Viewing the IP address of NICs on a virtual machine
You can view the IP address for a network interface controller (NIC) by using the web console or the oc
client. The QEMU guest agent displays additional information about the virtual machine’s secondary networks.
7.17.7.1. Viewing the IP address of a virtual machine interface in the CLI
The network interface configuration is included in the oc describe vmi <vmi_name>
command.
You can also view the IP address information by running ip addr
on the virtual machine, or by running oc get vmi <vmi_name> -o yaml
.
Procedure
Use the
oc describe
command to display the virtual machine interface configuration:$ oc describe vmi <vmi_name>
Example output
... Interfaces: Interface Name: eth0 Ip Address: 10.244.0.37/24 Ip Addresses: 10.244.0.37/24 fe80::858:aff:fef4:25/64 Mac: 0a:58:0a:f4:00:25 Name: default Interface Name: v2 Ip Address: 1.1.1.7/24 Ip Addresses: 1.1.1.7/24 fe80::f4d9:70ff:fe13:9089/64 Mac: f6:d9:70:13:90:89 Interface Name: v1 Ip Address: 1.1.1.1/24 Ip Addresses: 1.1.1.1/24 1.1.1.2/24 1.1.1.4/24 2001:de7:0:f101::1/64 2001:db8:0:f101::1/64 fe80::1420:84ff:fe10:17aa/64 Mac: 16:20:84:10:17:aa
7.17.7.2. Viewing the IP address of a virtual machine interface in the web console
The IP information displays in the Virtual Machine Overview screen for the virtual machine.
Procedure
- In the OpenShift Virtualization console, click Workloads → Virtualization from the side menu.
- Click the Virtual Machines tab.
- Select a virtual machine name to open the Virtual Machine Overview screen.
The information for each attached NIC is displayed under IP Address.
7.17.8. Using a MAC address pool for virtual machines
The KubeMacPool component provides a MAC address pool service for virtual machine NICs in designated namespaces. Enable a MAC address pool in a namespace by applying the KubeMacPool label to that namespace.
7.17.8.1. About KubeMacPool
If you enable the KubeMacPool component for a namespace, virtual machine NICs in that namespace are allocated MAC addresses from a MAC address pool. This ensures that the NIC is assigned a unique MAC address that does not conflict with the MAC address of another virtual machine.
Virtual machine instances created from that virtual machine retain the assigned MAC address across reboots.
KubeMacPool does not handle virtual machine instances created independently from a virtual machine.
KubeMacPool is disabled by default. Enable a MAC address pool for a namespace by applying the KubeMacPool label to the namespace.
7.17.8.2. Enabling a MAC address pool for a namespace in the CLI
Enable a MAC address pool for virtual machines in a namespace by applying the mutatevirtualmachines.kubemacpool.io=allocate
label to the namespace.
Procedure
Add the KubeMacPool label to the namespace. The following example adds the KubeMacPool label to two namespaces,
<namespace1>
and<namespace2>
:$ oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io=allocate
7.17.8.3. Disabling a MAC address pool for a namespace in the CLI
Disable a MAC address pool for virtual machines in a namespace by removing the mutatevirtualmachines.kubemacpool.io
label.
Procedure
Remove the KubeMacPool label from the namespace. The following example removes the KubeMacPool label from two namespaces,
<namespace1>
and<namespace2>
:$ oc label namespace <namespace1> <namespace2> mutatevirtualmachines.kubemacpool.io-
7.18. Virtual machine disks
7.18.1. Storage features
Use the following table to determine feature availability for local and shared persistent storage in OpenShift Virtualization.
7.18.1.1. OpenShift Virtualization storage feature matrix
Virtual machine live migration | Host-assisted virtual machine disk cloning | Storage-assisted virtual machine disk cloning | Virtual machine snapshots | |
---|---|---|---|---|
OpenShift Container Storage: RBD block-mode volumes | Yes | Yes | Yes | Yes |