Virtualization

• scc.AllowHostDirVolumePlugin = true

  
  This allows virtual machines to use the hostpath volume plugin.

• scc.AllowPrivilegedContainer = false

  
  This ensures the

  virt-launcher

  pod is not run as a privileged container.

• scc.AllowedCapabilities = []corev1.Capability{"SYS_NICE", "NET_BIND_SERVICE"}

  • SYS_NICE

    allows setting the CPU affinity.

  • NET_BIND_SERVICE

    allows DHCP and Slirp operations.

• Red Hat Enterprise Linux CoreOS workers. In the Microsoft SVVP Catalog, they are named Red Hat OpenShift Container Platform 4.19.
• Intel and AMD CPUs.

• You can now prevent the inadvertent deletion of a virtual machine (VM) by enabling delete protection for the VM. You can also disable delete protection that has been set for a VM.

  As a cluster administrator, you can prevent users from enabling VM delete protection by removing the option at the cluster level.

• You can now choose to expand virtual machines (VMs) using instance types and preferences. For more information, see Using InstancetypeReferencePolicy to expand VirtualMachines in the Red Hat Customer Portal.
• Red Hat Enterprise Linux (RHEL) 10 is added as a certified guest operating system. For a complete list of supported guest operating systems, see Certified Guest Operating Systems in OpenShift Virtualization.

• You can now update the machine type of multiple virtual machines (VMs) at the same time from the OpenShift CLI (

  oc

  ).

• You can now configure a NodeNetworkConfigurationPolicy manifest to enable the Link Layer Discovery Protocol (LLDP) listener for all ethernet ports in your OpenShift Container Platform cluster.
• You can now create a new Node Network Configuration Policy (NNCP) in the topology view of the cluster and see its graphical representation in real time. Clicking Create on the Node network configuration page opens a form for configuring the elements of the new NNCP, and the NNCP is also displayed as a diagram.

• You can now use the OVN-Kubernetes localnet network topology to connect a VM to a secondary user-defined network.

• You can now use a PVC as the source of a custom

  DataImportCron

  in the

  dataImportCronTemplates

  section of the

  HyperConverged

  custom resource (CR). See Managing automatic boot source updates for more information.

• You can now configure multiple IOThreads for virtual machines that use fast storage, such as SSD (solid-state drive) or NVMe (non-volatile memory express). This improves I/O performance by enabling multiple threads for disk access.

• On the VirtualMachines page, you can now see the summary of CPU, memory, and storage usage by your VMs. To restrict this summary to the VMs in a specific project, select the project name in the tree view.

• On the VirtualMachines page, you can now navigate between your VMs by using the tree view.

• An attempt to stop, restart, or pause a VM or multiple VMs now displays a confirmation dialog.

• You can now access the commands of the Options menu from the tree view by right-clicking the VM. If you right-click a project and select a command, the action is applied to all VMs in the project.
• You can now perform bulk actions on multiple virtual machines (VMs), including adding or removing labels, viewing the number of VMs selected for deletion, and moving VMs to a folder within the same namespace.

• On the VirtualMachines page, you can now use the tree view to organize VMs in folders and drag and drop VMs to these folders.

• You can now search for virtual machines by fields such as name, project, description, labels, date created, vCPU, and memory. You can also save frequently used search queries.

• The following alerts for the OpenShift Virtualization Operator are now included in the OpenShift Container Platform runbooks:

  • HAControlPlaneDown

  • HighCPUWorkload

  • NodeNetworkInterfaceDown

• New metrics are now available and improve the observability of virtual machines (VMs) and virtual machine instances (VMIs). You can use these metrics to monitor the following VM lifecycle events, resource usage, and migration details:

  • Migration metrics
  • vNIC networking information metrics
  • Allocated storage size metrics for running and stopped VMs

  In addition, the following VM and VMI metadata metrics are now available:

  • The

    pod_name

    label in

    kubevirt_vmi_info

  • UID in VM and VMI metrics
  • The VM creation date

  For a complete list of virtualization metrics, see KubeVirt components metrics.

• VirtualMachines that use instance types and preferences no longer have their specification mutated at runtime to include derived metadata, such as

  revisionName

  . This metadata is now stored in the

  status

  field to preserve the declarative VM specification and ensure compatibility.

• In OpenShift Virtualization 4.19, the default permissions for live migration have changed to improve cluster security. Users must now be explicitly granted the

  kubevirt.io:migrate

  cluster role to create, delete, or update live migration requests. Previously, namespace administrators had these permissions by default. For more information, see About live migration permissions.

• The RHEL 8

  kubevirt-virtctl

  RPM is deprecated. Download the

  virtctl

  binary from the OpenShift Container Platform web console instead of using the command line. The RPM will be removed in a future release.
• The

  OperatorConditionsUnhealthy

  alert is deprecated. You can safely silence it.

• The following

  HyperConverged

  custom resource (CR) fields have been deprecated and copied from their original location under the

  spec.featureGates

  fields to a new location in the

  spec

  field, where they can be used if needed:

  • DeployVmConsoleProxy

  • EnableApplicationAwareQuota

  • EnableCommonBootImageImport

  If used in the

  spec.featureGates

  location, the old fields are ignored.

New and changed features

• Installing OpenShift Virtualization on Oracle Cloud Infrastructure (OCI) is now generally available. For more information, see OpenShift Virtualization and Oracle Cloud Infrastructure known issues and limitations in the Red Hat Knowledgebase, and Installing OpenShift Virtualization on OCI on GitHub.

New and changed features

• By using the OpenShift Container Platform web console, you can now migrate VMs in bulk from one storage class to another storage class.

  This feature requires both OpenShift Virtualization 4.19.4 or greater and Migration Toolkit for Containers (MTC) 1.8.9 or greater.

New and changed features

• Support for s390x architecture is now generally available. You can use OpenShift Virtualization on an OpenShift Container Platform cluster that has been deployed in one or more logical partitions (LPARs) on IBM Z® and IBM® LinuxONE (s390x architecture) systems. For more information, see IBM Z and IBM LinuxONE compatibility.

New and changed features

• With the new IBM Fusion Access for SAN, you can now deploy VMs on a scalable, clustered file system in Red Hat OpenShift Virtualization. Fusion Access for SAN offers access to consolidated, block-level data storage. It presents storage devices such as disk arrays to the operating system as if they were direct-attached storage.

  The Fusion Access for SAN Operator is available in the OpenShift Container Platform Operator hub.

  See About IBM Fusion Access for SAN for more information.

Known issues

• When a file system in Fusion Access for SAN has two local disks and one local disk fails, both local disks move to the

  Unknown

  state, with no indication which of the local disks failed. (OCPNAS-56)

• When creating more than one file system for VM storage in Fusion Access for SAN, deleting the initial primary file system results in all of the remaining file systems becoming unusable. You cannot migrate or restart any of the VMs running on the remaining file systems, and you cannot create new VMs on the remaining file systems.

  To determine which file system is the primary file system, run the following command:

  $ oc get cso -n ibm-spectrum-scale-csi ibm-spectrum-scale-csi -o jsonpath='{.spec.clusters[*].primary.primaryFs}'

  (OCPNAS-61)

• When a disruption occurs between the worker nodes in a Fusion Access for SAN storage cluster and the shared LUNs they are connected to, the VMs on the storage cluster pause and cannot be unpaused even after the service was restored. The only way to recover the VM is to restart it. (OCPNAS-62)

• Storage live migration from ODF to Fusion Access for SAN using MTC (v1.8.6) only works when the target access mode is specified as

  RWO

  . However, Fusion Access for SAN uses

  filesystem/RWX

  by default.

  When you migrate from ODF to Fusion Access for SAN (RWO) you receive the following error in the VM logs:

  message: 'cannot migrate VMI: PVC dv-fedora000-mig-hwtp is not shared, live migration
    requires that all PVCs must be shared (using ReadWriteMany access mode)'
  reason: DisksNotLiveMigratable

  This results in the VM being inaccessible when the worker node is not available.

  (OCPNAS-77)

• When you create a new file system in Fusion Access for SAN with the same name as an existing file system, an error appears, and the Create file system button is stuck displaying a loading spinner. If you reload the page, it lists only the original file system. However, if you try to create another new file system, the LUNs you selected for the second file system no longer appear as available. (OCPNAS-81)

• If a Fusion Access for SAN file system is filled to its maximum capacity, the

  mmhealth state

  of the file system custom resource (CR) becomes

  Degraded

  . This is caused by the

  no_disk_space_warn

  event. After freeing disk space, you can once again use the file system, but the file system keeps the

  Degraded

  status. (OCPNAS-110)

• When using a multipath LUN in Fusion Access for SAN, removing a local disk does not remove the partition. (OCPNAS-124)

  • As a workaround, run the following commands on one of the nodes:

    $ multipath -f <device>

    $ multipath -r

    Running these commands on one of the nodes fixes all of the nodes.

• LUNs used to create a file system in Fusion Access for SAN still appear as available for use until the file system moves from the

  Creating

  state to the

  Healthy

  state. This can result in users creating an additional file system with LUNs that that are already in use. After the first file system shifts to the

  Healthy

  state, the LUNs disappear from the second file system. (OCPNAS-126)

• Fusion Access for SAN formats disks with existing partitions that are not Fusion Access for SAN related. When attempting to add a new iSCSI target with an existing partition and data, Fusion Access for SAN automatically formats the share without warning. (OCPNAS-143)

• Deleting a second file system in Fusion Access for SAN results in the following error:

  Your focus-trap must have at least one container with at least one tabbable node in it at all times.

  (OCPNAS-163) ** As a workaround, reload the page and delete the second file system.

• If your credentials for the image registry used to install Fusion Access for SAN change, you must delete the

  kmm-registry-push-pull-secret

  pull secret in the

  ibm-fusion-access

  namespace.Then you must restart the

  fusion-access-operator-controller-manager

  pod in the

  ibm-fusion-access

  namespace. (OCPNAS-170)

• If you change the KMM settings that trigger a rebuild while the Fusion Access for SAN storage cluster is running and using the kernel modules, KMM cannot unload the modules, resulting in an error. (OCPNAS-172)

• When backing up VMs with OADP datamover on a Fusion Access for SAN storage cluster, the process remains in the

  Pending

  state for a long time before shifting to the

  Bound

  state and beginning the backup. The process might even remain in

  Pending

  until it times out completely. (OCPNAS-175)

• When creating a file system, it may take over twenty minutes for the Status of the new file system to change from Creating to Healthy. During that time, the Status appears stuck in Creating, and the following error message appears when you click on the status:

  Failed to create filesystem. Check the operator log for more details.

  This error is not correct.

  (OCPNAS-184)

Prerequisites

• Each host in your cluster must be registered with Red Hat Subscription Manager (RHSM) and have an active OpenShift Container Platform subscription.

Prerequisites

• The PVC volume mode must be

  FileSystem

  .

• The PVC must be large enough to contain the memory dump.

  The formula for calculating the PVC size is

  (VMMemorySize + 100Mi) * FileSystemOverhead

  , where

  100Mi

  is the memory dump overhead.

• You must enable the hot plug feature gate in the

  HyperConverged

  custom resource by running the following command:

  $ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
    --type json -p '[{"op": "add", "path": "/spec/featureGates", \
    "value": "HotplugVolumes"}]'

• You can install the cluster by using installer-provisioned infrastructure, ensuring that you specify bare-metal instance types for the worker nodes. For example, you can use the

  c5n.metal

  type value for a machine based on x86_64 architecture. You specify bare-metal instance types by editing the

  install-config.yaml

  file.

  For more information, see the OpenShift Container Platform documentation about installing on AWS.

• There is no change to how you access VMs by using the

  virtctl

  CLI tool or the OpenShift Container Platform web console.

• You can expose VMs by using a

  NodePort

  or

  LoadBalancer

  service.
  Note

  The load balancer approach is preferable because OpenShift Container Platform automatically creates the load balancer in AWS and manages its lifecycle. A security group is also created for the load balancer, and you can use annotations to attach existing security groups. When you remove the service, OpenShift Container Platform removes the load balancer and its associated resources.

• You cannot use Single Root I/O Virtualization (SR-IOV) or bridge Container Network Interface (CNI) networks, including virtual LAN (VLAN). If your application requires a flat layer 2 network or control over the IP pool, consider using OVN-Kubernetes secondary overlay networks.

• You can use any storage solution that is certified by the storage vendor to work with the underlying platform.

  Important

  AWS bare metal, Red Hat OpenShift Service on AWS, and Red Hat OpenShift Service on AWS classic architecture clusters might have different supported storage solutions. Ensure that you confirm support with your storage vendor.

• Using Amazon Elastic File System (EFS) or Amazon Elastic Block Store (EBS) with OpenShift Virtualization might cause performance and functionality limitations as shown in the following table:

  Expand

  Table 4.1. EFS and EBS performance and functionality limitations

  Feature	EBS volume			EFS volume	Shared storage solutions
  	gp2	gp3	io2
  VM live migration	Not available	Not available	Available	Available	Available
  Fast VM creation by using cloning	Available			Not available	Available
  VM backup and restore by using snapshots	Available			Not available	Available

  Show more

  Consider using CSI storage, which supports ReadWriteMany (RWX), cloning, and snapshots to enable live migration, fast VM creation, and VM snapshots capabilities.

• HCPs for OpenShift Virtualization are not currently supported on AWS infrastructure.

• Memory hot plugging and hot unplugging
• Node Health Check Operator
• SR-IOV Operator
• PCI passthrough
• OpenShift Virtualization cluster checkup framework
• OpenShift Virtualization on a cluster installed in FIPS mode
• IPv6
• IBM® Storage scale
• Hosted control planes for OpenShift Virtualization
• VM pages using HugePages

• virtual Trusted Platform Module (vTPM) devices
• UEFI mode for VMs
• USB host passthrough
• Configuring virtual GPUs
• Creating and managing Windows VMs
• Hyper-V

• When deleting a virtual machine by using the web console, the grace period option is ignored.
• When configuring the default CPU model, the

  spec.defaultCPUModel

  value is

  "gen15b"

  for an IBM Z cluster.
• When configuring a downward metrics device, if you use a VM preference, the

  spec.preference.name

  value must be set to

  rhel.9.s390x

  or another available preference with the format

  *.s390x

  .
• When creating virtual machines from instance types, you are not allowed to set

  spec.domain.memory.maxGuest

  because memory hot plugging is not supported on IBM Z®.
• Prometheus queries for VM guests could have inconsistent outcome in comparison to

  x86

  .

• Supported by Red Hat Enterprise Linux (RHEL) 9.

  See Red Hat Ecosystem Catalog for supported CPUs.

  Note

  If your worker nodes have different CPUs, live migration failures might occur because different CPUs have different capabilities. You can mitigate this issue by ensuring that your worker nodes have CPUs with the appropriate capacity and by configuring node affinity rules for your virtual machines.

  See Configuring a required node affinity rule for details.

• Support for AMD and Intel 64-bit architectures (x86-64-v2).
• Support for Intel 64 or AMD64 CPU extensions.
• Intel VT or AMD-V hardware virtualization extensions enabled.
• NX (no execute) flag enabled.

• Red Hat Enterprise Linux CoreOS (RHCOS) installed on worker nodes.

  See About RHCOS for details.

  Note

  RHEL worker nodes are not supported.

• Supported by OpenShift Container Platform. See Optimizing storage.
• You must create a default OpenShift Virtualization or OpenShift Container Platform storage class. The purpose of this is to address the unique storage needs of VM workloads and offer optimized performance, reliability, and user experience. If both OpenShift Virtualization and OpenShift Container Platform default storage classes exist, the OpenShift Virtualization class takes precedence when creating VM disks.

• If the storage provisioner supports snapshots, you must associate a

  VolumeSnapshotClass

  object with the default storage class.

Prerequisites

• Install OpenShift Container Platform 4.19 on your cluster.
• Log in to the OpenShift Container Platform web console as a user with

  cluster-admin

  permissions.

Verification

• Navigate to the Workloads → Pods page and monitor the OpenShift Virtualization pods until they are all Running. After all the pods display the Running state, you can use OpenShift Virtualization.

Prerequisites

• You have access to an OpenShift Container Platform cluster using an account with

  cluster-admin

  permissions.

Prerequisites

• You have access to the OpenShift Container Platform cluster web console using an account with

  cluster-admin

  permissions.

Prerequisites

• You have access to the OpenShift Container Platform cluster using an account with

  cluster-admin

  permissions.