Red Hat SummitChapter 4. Setting up real-time virtual machines
To set up a virtual machine (VM) with a RHEL 9 for Real Time guest operating system, you must create a VM, configure its guest, and optimize and test the VM’s performance.
4.1. Optimizing vCPU pinning for real-time virtual machines
To correctly set up a RHEL real-time (RT) virtual machine (VM), you must first have a plan for optimal pinning of the VM’s virtual CPUs (vCPUs) to the physical CPUs of the host.
Prerequisites
- You have set up the host system for real-time virtualization. For instructions, see Configuring the host environment for real-time virtual machines.
The
hwlocpackage has been installed.dnf install hwloc
# dnf install hwlocCopy to Clipboard Copied! Toggle word wrap Toggle overflow - You have evaluated the performance of the VM to obtain the baseline for later verification. For various methods to do this, see Virtual machine performance monitoring tools.
Procedure
View the CPU topology of your host system:
lstopo-no-graphics
# lstopo-no-graphicsCopy to Clipboard Copied! Toggle word wrap Toggle overflow The following example output shows a system with 32 physical cores with enabled hyperthreading, divided into 2 sockets ("packages"), each with 4 CPU dies. The system also has 250 GB of RAM split across 2 NUMA nodes.
Note that the following examples in this procedure are based on this topology.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Based on the output of
lstopo-no-graphicsand your required real-time VM setup, determine how to pin your vCPUs to physical CPUs. The following items show XML configurations effective for the example host output above and a real-time VM with 4 vCPUs:The following pinning placement uses an exclusive core for each vCPU. For such pinning configuration to be effective, the assigned physical CPUs must be isolated on the host and must not have any processes running on them.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The following pinning placement uses an exclusive L3 core for each vCPU:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
- Monitor the performance of the VM again and compare the values with the previously obtained baseline. For various methods to do this, see Virtual machine performance monitoring tools.
4.2. Installing a RHEL real-time guest operating system
To prepare a virtual machine (VM) environment for real-time workloads, create a new VM and adjust its configuration for low-latency performance.
Prerequisites
- You have set up your system to host real-time VMs. For instructions see Configuring the host environment for real-time virtual machines.
- You have considered what the optimal vCPU pinning configuration for your real-time VM is. For more information, see Optimizing vCPU pinning for real-time virtual machines.
Procedure
Use the
virt-installutility to create a RHEL 9 VM with the following properties:- The VM has 2 or more assigned vCPUs
- The VM uses huge pages for memory backing.
The following example command creates a VM named
RHEL9-RTthat fits the mentioned requirements:Copy to Clipboard Copied! Toggle word wrap Toggle overflow After the installation finishes, shut down the VM.
virsh shutdown <RHEL9-RT>
# virsh shutdown <RHEL9-RT>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Open the XML configuration of the VM.
virsh edit <RHEL9-RT>
# virsh edit <RHEL9-RT>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Adjust the CPU configuration as follows:
<cpu mode='host-model' check='partial'> <feature policy='require' name='tsc-deadline'/> </cpu><cpu mode='host-model' check='partial'> <feature policy='require' name='tsc-deadline'/> </cpu>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove non-essential virtual hardware from the VM to improve its performance.
Delete the section for the
virtioRNG device.<rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </rng><rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </rng>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove USB devices, such as the following:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove serial devices, such as the following:
<serial type='dev'> <source path='/dev/ttyS0'/> <target port='0'/> </serial>
<serial type='dev'> <source path='/dev/ttyS0'/> <target port='0'/> </serial>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the QXL device.
<video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> </video>
<video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> </video>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Disable the graphical display.
<graphics type='vnc' ports='-1' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'> </graphics>
<graphics type='vnc' ports='-1' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'> </graphics>Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the USB controller setting, change the model to
noneto disable it.<controller type='usb' index='0' model='none'/>
<controller type='usb' index='0' model='none'/>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the Trusted Platform Module (TPM) configuration, so that it does not interfere with RT operations.
<tpm model='tpm-crb'> <backend type='emulator' version='2.0'/> </tpm><tpm model='tpm-crb'> <backend type='emulator' version='2.0'/> </tpm>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Disable the
memballoonfunction.<memballoon model='none'>
<memballoon model='none'>Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the
<features>section of the configuration, ensure that thePMUandvmportfeatures are disabled, to avoid the latency they might cause.<features> [...] <pmu state='off'/> <vmport state='off'/> </features><features> [...] <pmu state='off'/> <vmport state='off'/> </features>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Edit the
<numatune>section to set up the NUMA nodes.<numatune> <memory mode='strict' nodeset='1'/> </numatune><numatune> <memory mode='strict' nodeset='1'/> </numatune>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the
<cputune>section of the configuration to set up vCPU NUMA pinning as planned out in Optimizing vCPU pinning for real-time virtual machines.The following example configures a VM with 4 vCPUs and these parameters:
- The isolated core 15 from NUMA node 0 is the non-realtime vCPU
- Cores 16, 47, and 48, from NUMA nodes 1 - 3, are the real-time vCPU
- The configuration pins all the QEMU I/O threads to the host housekeeping cores (0 and 32).
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf your host uses hardware with enabled hyperthreading, also ensure that your
<cputune>configuration meets the following requirements:- Assign the siblings of a physical core to perform either real-time or housekeeping tasks.
- Use both the siblings of a physical core in the same VM.
- For vCPUs that are pinned to the siblings of the same physical core, assign the vCPU to the same task (real-time processes or housekeeping) as the sibling.
Note that the example configuration above meet these requirements.
- Save and exit the XML configuration.
Troubleshooting
Verification
On the host, view the configuration of the VM and verify that it has the required parameters:
virsh dumpxml <RHEL9-RT>
# virsh dumpxml <RHEL9-RT>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.3. Configuring the RHEL guest operating system for real time
To optimize a RHEL 9 virtual machine (VM) environment for real-time workloads, configure the guest operating system for low-latency performance.
Prerequisites
- You have installed a RHEL 9 virtual machine and adjusted its configuration for real-time workloads. For instructions, see Installing a RHEL real-time guest operating system.
Procedure
- Start the VM.
Install real-time packages in the guest operating system.
dnf install -y kernel-rt tuned tuned-profiles-realtime tuned-profiles-nfv realtime-tests
# dnf install -y kernel-rt tuned tuned-profiles-realtime tuned-profiles-nfv realtime-testsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Adjust the virtual guest profile for
tuned. To do so, edit the/etc/tuned/realtime-virtual-guest-variables.conffile and add the following lines:isolated_cores=<isolated-core-nrs> isolate_managed_irq=Y
isolated_cores=<isolated-core-nrs> isolate_managed_irq=YCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<isolated-core-nrs>with the numbers of host cores that you want to isolate for real-time workloads.Ensure that irqbalance is disabled in the guest operating system.
rpm -q irqbalance && systemctl stop irqbalance && systemctl disable irqbalance
# rpm -q irqbalance && systemctl stop irqbalance && systemctl disable irqbalanceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Activate the realtime-virtual-guest profile for tuned.
tuned-adm profile realtime-virtual-guest
# tuned-adm profile realtime-virtual-guestCopy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure that the real-time kernel is used by the guest operating system by default.
grubby --set-default vmlinuz-5.14.0-XXX.el9.x86_64+rt
# grubby --set-default vmlinuz-5.14.0-XXX.el9.x86_64+rtCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Configure huge pages for the guest operating system in the same way as in the host. For instructions, see Configuring huge pages for real-time virtualization hosts.
Verification
- Perform the same stress test on the guest operating system as on the host. For instructions, see Stress testing the real-time virtualization system.
Troubleshooting
If the results of the stress test exceed the required latency, do the following:
- Perform the stress tests on the host again. If the latency results are suboptimal, adjust the host configuration of TuneD and huge pages, and re-test. For instructions, see Configuring TuneD for the real-time virtualization host and Configuring huge pages for real-time virtualization hosts.
-
If the stress test results on the host show sufficiently low latency but on the guest they do not, use the
trace-cmdutility to generate a detailed test report. For instructions, see Collecting data to troubleshoot latency issues for RHEL real-time guests (Red Hat Knowledgebase).
4.4. Setting up cache protection for real-time virtual machines
Eviction of cache lines might cause performance issues in real-time virtual machines (VMs). Optionally, to avoid this problem, use the User Interface for Resource Control (resctrlfs`) feature to manage your caches and cache partitions:
- Divide the main memory cache of the host system into partitions
- Assign separate tasks to each partition
- Assign vCPUs that run real-time applications to one cache partition
- Assign vCPUs and host CPUs that run housekeeping workloads to a different cache partition
Prerequisites
- You have created a real-time virtual machine on your host. For instructions, see Installing a RHEL real-time guest operating system.
Your host is using an Intel processor that supports L2 or L3 cache partitioning. To ensure this is the case:
Install the
intel-cmt-catutility.dnf install intel-cmt-cat
# dnf install intel-cmt-catCopy to Clipboard Copied! Toggle word wrap Toggle overflow Use the
pqosutility to display your core cache details.Copy to Clipboard Copied! Toggle word wrap Toggle overflow This output indicates your CPU supports L3 cache partitioning.
Procedure
The following steps assume that you have the following NUMA pinning assignment of vCPUs to CPUs.
<cputune>
<vcpupin vcpu='0' cpuset='16'/>
<vcpupin vcpu='1' cpuset='17'/>
<vcpupin vcpu='2' cpuset='18'/>
<vcpupin vcpu='3' cpuset='19'/>
<cputune>
<vcpupin vcpu='0' cpuset='16'/>
<vcpupin vcpu='1' cpuset='17'/>
<vcpupin vcpu='2' cpuset='18'/>
<vcpupin vcpu='3' cpuset='19'/>Mount the
resctrlfile system. This makes it possible to use the resource control capabilities of the processor.mount -t resctrl resctrl /sys/fs/resctrl
# mount -t resctrl resctrl /sys/fs/resctrlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Depending on whether your system supports L2 or L3 cache partitioning, a subdirectory named
L2orL3is mounted in the/sys/fs/resctrl/infodirectory. The following steps assume your system supports L3 cache partitioning.Move into the cache directory and list its content.
cd /sys/fs/resctrl/info/L3/; ls
# cd /sys/fs/resctrl/info/L3/; ls bit_usage cbm_mask min_cbm_bits num_closids shareable_bits sparse_masksCopy to Clipboard Copied! Toggle word wrap Toggle overflow View the value of the
cbm_maskfile.cat cbm_mask ffff
# cat cbm_mask ffffCopy to Clipboard Copied! Toggle word wrap Toggle overflow This value represents the cache bitmask in a hexadecimal code.
ffffmeans that all 16 bits of the cache can be used by the workload.View the value of the "shareable_bits" file.
cat shareable_bits 0
# cat shareable_bits 0Copy to Clipboard Copied! Toggle word wrap Toggle overflow This value represents partitions of the L3 cache that are shared by other executing processes, such as I/O, and therefore should not be used in exclusive cache partitions.
0means that you can use all the L3 cache partitions.View the
schematafile to see the global cache allocation.cat /sys/fs/resctrl/schemata L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ffff;10=ffff;12=ffff;14=ffff
# cat /sys/fs/resctrl/schemata L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ffff;10=ffff;12=ffff;14=ffffCopy to Clipboard Copied! Toggle word wrap Toggle overflow This output indicates that for the L3 cache partition, CPU sockets 0, 2, 4, 6, 8, 10, 12, and 14 are fully allocated to the default control group. In this example, CPU sockets 16 - 19 are pinned to vCPUs 0-3.
Determine what cache distribution you want to set for real-time applications. For example, for an even distribution of 8 MB for both real-time applications and housekeeping applications:
-
The cache bitmask for real-time applications is
ff00 -
The cache bitmask for housekeeping applications is
00ff
-
The cache bitmask for real-time applications is
Adjust the default
schematafile with your required cache allocation for housekeeping processes. For example, to assign 8MB to CPU socket 8, do the following:echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=00ff;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/schemata
# echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=00ff;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/schemataCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a specific control group for real-time processes, for example
part1.mkdir /sys/fs/resctrl/part1
# mkdir /sys/fs/resctrl/part1Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
schematafile for thepart1control group, and set it to use cache allocation that does not conflict with the housekeeping cache allocation.echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ff00;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/part1/schemata
# echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ff00;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/part1/schemataCopy to Clipboard Copied! Toggle word wrap Toggle overflow With this setting, the L3 cache 8 uses half of its cache allocation for real-time processes and the other half for the housekeeping processes. All the other L3 caches can be used freely by both real-time processes and housekeeping.
Assign the CPUs pinned to real-time vCPUs (in this case 17, 18, and 19) to this control group.
echo 17,18,19 > part1/cpus_list
# echo 17,18,19 > part1/cpus_listCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
If you previously tested the latency of the VM, run the
cyclictestutility again. For instructions, see Stress testing the real-time virtualization system.If the maximum latency is lower than previously, you have set up cache protection correctly.
4.5. Troubleshooting RHEL real-time guest installation
While installing a RHEL 9 virtual machine (VM) on your real-time host, you might encounter one of the following errors. Use the following recommendations to fix or work around these issues.
Error: Host doesn’t support any virtualization options
- Ensure that virtualization is enabled in the host BIOS
Check that cpu flags on your host contain
vmxfor Intel, orsvmfor AMD.cat /proc/cpuinfo | grep vmx
$ cat /proc/cpuinfo | grep vmxCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the
lsmodcommand detects thekvmandkvm_intelorkvm_amdmodules.lsmod | grep kvm
$ lsmod | grep kvmCopy to Clipboard Copied! Toggle word wrap Toggle overflow Make sure the
kernel-rt-kvmpackage is installed.dnf info kernel-rt-kvm
$ dnf info kernel-rt-kvmCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Check if the
/dev/kvmdevice exists. -
Run the
virt-host-validateutility to detect any further issues. - Run kvm-unit-tests.
Permission-related issues while accessing the disc image
-
In the
/etc/libvirt/qemu.conffile, uncomment thegroup =anduser =lines. Restart the
virtqemudservice.service virtqemud restart
$ service virtqemud restartCopy to Clipboard Copied! Toggle word wrap Toggle overflow
-
In the
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}