Chapter 19. Creating nested virtual machines

Tested architectures

• The L0 host must be an Intel, AMD, IBM POWER9, or IBM Z system. Nested virtualization currently does not work on other architectures, such as ARM.

• Both the L0 host and the L1 VM use RHEL 8
• The L2 VM uses RHEL 8, or RHEL 7 with a rhel-alt kernel.
• The L1 VM and L2 VM are not running in POWER8 compatibility mode.

Hypervisor limitations

• Currently, Red Hat tests nesting only on RHEL-KVM. When RHEL is used as the L0 hypervisor, you can use RHEL or Windows as the L1 hypervisor.
• When using an L1 RHEL 8 VM on a non-KVM L0 hypervisor, such as VMware ESXi or Amazon Web Services (AWS), creating L2 VMs in the RHEL 8 guest operating system might work, but is not tested.

Feature limitations

• Use of L2 VMs as hypervisors and creating L3 guests has not been properly tested and is not expected to work.
• Migrating VMs currently does not work on AMD systems if nested virtualization has been enabled on the L0 host.

• On an IBM Z system, huge-page backing storage and nested virtualization cannot be used at the same time.

  subs="+quotes,attributes"]

• Some features available on the L0 host might be unavailable for the L1 hypervisor.

  For example, on IBM POWER 9 hardware, the External Interrupt Virtualization Engine (XIVE) does not work. However, L1 VMs can use the emulated XIVE interrupt controller to start L2 VMs.

Prerequisites

• An L0 RHEL 8 host running an L1 virtual machine (VM).
• The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the vmx and ept flags, creating L2 VMs is possible. This is generally the case on Intel Xeon v3 cores and later.

• Ensure that nested virtualization is enabled on the L0 host:

  # cat /sys/module/kvm_intel/parameters/nested

  • If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.

  • If the command returns 0 or N but your system supports nested virtualization, use the following steps to enable the feature.

    1. Unload the kvm_intel module:

       # modprobe -r kvm_intel

    2. Activate the nesting feature:

       # modprobe kvm_intel nested=1

    3. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

       options kvm_intel nested=1

Prerequisites

• An L0 RHEL 8 host running an L1 virtual machine (VM).
• The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the svm and npt flags, creating L2 VMs is possible. This is generally the case on AMD EPYC cores and later.

• Ensure that nested virtualization is enabled on the L0 host:

  # cat /sys/module/kvm_amd/parameters/nested

  • If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.

  • If the command returns 0 or N, use the following steps to enable the feature.

    1. Stop all running VMs on the L0 host.

    2. Unload the kvm_amd module:

       # modprobe -r kvm_amd

    3. Activate the nesting feature:

       # modprobe kvm_amd nested=1

    4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following to the /etc/modprobe.d/kvm.conf file:

       options kvm_amd nested=1

Prerequisites

• An L0 RHEL 8 host running an L1 virtual machine (VM).
• The hypervisor CPU must support nested virtualization. To verify this is the case, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the sie flag, creating L2 VMs is possible.

• Ensure that nested virtualization is enabled on the L0 host:

  # cat /sys/module/kvm/parameters/nested

  • If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.

  • If the command returns 0 or N, use the following steps to enable the feature.

    1. Stop all running VMs on the L0 host.

    2. Unload the kvm module:

       # modprobe -r kvm

    3. Activate the nesting feature:

       # modprobe kvm nested=1

    4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

       options kvm nested=1

Procedure

• Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.

Prerequisites

• An L0 RHEL 8 host is running an L1 VM. The L1 VM is using RHEL 8 as the guest operating system.

• Nested virtualization is enabled on the L0 host:

  # cat /sys/module/kvm_hv/parameters/nested

  • If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.

  • If the command returns 0 or N, use the following steps to enable the feature:

    1. Stop all running VMs on the L0 host.

    2. Unload the kvm module:

       # modprobe -r kvm_hv

    3. Activate the nesting feature:

       # modprobe kvm_hv nested=1

    4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

       options kvm_hv nested=1

Additional information

• Note that IBM POWER8 as the architecture for the L2 VM currently does not supported.