Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Configuring virtualization on RHEL 9 for Real Time

Red Hat Enterprise Linux for Real Time 9

Installing and testing virtual machines on RHEL for Real Time hosts.

Red Hat Customer Content Services

Abstract

Set up, optimize, and test a server to develop and deploy applications in an environment that uses KVM as a real-time hypervisor.

Providing feedback on Red Hat documentation
Copy link

We appreciate your feedback on our documentation. Let us know how we can improve it.

Submitting feedback through Jira (account required)

  1. Log in to the Jira website.
  2. Click Create in the top navigation bar
  3. Enter a descriptive title in the Summary field.
  4. Enter your suggestion for improvement in the Description field. Include links to the relevant parts of the documentation.
  5. Click Create at the bottom of the dialogue.

To use virtual machines (VMs) in real-time workloads, Red Hat provides virtualization capabilities in RHEL for Real Time.

In RHEL for Real Time, you can configure the host and guest operating systems to achieve low-latency and deterministic behavior for VMs. This makes real-time VMs more suitable for applications that require real-time performance, such as industrial automation, telecommunications, and automotive systems.

For details on the benefits of using a real-time RHEL system, see RHEL for Real Time for optimizing latency.

To host real-time VMs, RHEL 9 systems must meet the following requirements:

  • Real-time kernel

    To configure your RHEL 9 host as a real-time system, you must install the real-time kernel. For instructions, see Installing RHEL for Real Time.

    In addition, you must do the following on the host:

    1. Enable the NFV repository: 

      # sudo subscription-manager repos --enable=rhel-9-server-nfv-rpms

    2. Install the tuned-profiles-nfv package: 

      # dnf install tuned-profiles-nfv

    3. If your host uses RHEL 9.6 or earlier, also install the kernel-rt-kvm package: 

      # dnf install kernel-rt-kvm

  • Virtualization

    The RHEL 9 virtualization packages must be installed on your host. For instructions, see Enabling virtualization.

  • BIOS

    To successfully deploy and run real-time virtual machines, the system BIOS of your host must be configured for minimum latency. To do so, follow the instructions specific to the hardware model of your host machine. For example:

    Specific low-latency settings vary across different vendors and models, but in general, steps for minimizing latency include the following:

    • Disable advanced Hardware Power Management options. These provide more control over various specific power management aspects, but might also cause latency spikes.

    • Disable lower CPU power states, such as C-states or C1E.

      Important

      Some low-latency BIOS guides might recommend disabling virtualization. However, this step must be skipped for real-time VMs to work correctly on your host.

To ensure that your RHEL 9 can work as a host for real-time virtual machines, you must optimize the host’s performance and test its latency between input and system response.

To optimize your RHEL 9 system as a host for real-time virtual machines (VMs), configure and enable the realtime-virtual-host profile for TuneD.

Prerequisites

  • Your host meets the system requirements for real-time virtualization.

  • The irqbalance service is disabled. If irqbalance is enabled, its handling of Interrupt requests (IRQs) might conflict with TuneD. To disable irqbalance: 

    # systemctl stop irqbalance && systemctl disable irqbalance

Procedure

  1. Start editing the configuration of the realtime-virtual-host profile for TuneD. To do so, open the /etc/tuned/realtime-virtual-host-variables.conf file in a text editor.

  2. Adjust the configuration in /etc/tuned/realtime-virtual-host-variables.conf to suit your requirements. Consider especially the following factors in the setup:

    • The number of cores and NUMA nodes your machine has
    • The number of RT guests that you plan to run
    • The number of vCPUs that each RT guest will have

    The most important modifications to /etc/tuned/realtime-virtual-host-variables.conf include the following:

    • Update the isolated_cores parameter to adjust which host cores per socket will be dedicated to RT virtualization tasks and which cores will remain for system maintenance on the host (also known as housekeeping).

      For example, the following setting uses core 3, core 6, and cores 8 to 15 for RT tasks, and all the other cores as housekeeping: 

      isolated_cores=3,6,8-15

      Note that by default, one core per socket (core 0) is used for housekeeping and all other cores for RT tasks.

      Important

      Core 0 must always be set as a housekeeping core. Using core 0 for RT tasks disrupts the RT functionality.

    • Enable IRQ isolation for kernel-managed IRQs. To do so, ensure the following line is not commented out in the configuration: 

      isolate_managed_irq=Y

      If IRQ isolation is disabled, host kernel-managed IRQs can interrupt isolated cores, which might cause unexpected latency.

    • Uncomment the netdev_queue_count parameter and set its value to the number of housekeeping cores.
  3. Save the changes to /etc/tuned/realtime-virtual-host-variables.conf.

  4. Activate the real-time virtual host profile. 

    # tuned-adm profile realtime-virtual-host
  5. Restart the host.

Verification

Next steps

To further decrease latency in virtual machines (VMs) on RHEL 9, set the host to use huge memory pages. Huge pages can significantly enhance the performance of applications that use large amounts of memory, which is generally the case for RT applications.

For more information on huge pages, see Configuring huge pages.

Prerequisites

Procedure

  1. Set up the default huge page size to be 1 gibibyte. 

    $ grubby --args "default_hugepagesz=1G" --update-kernel ALL

  2. Reserve huge pages on the host. 

    $ echo <X> > /sys/devices/system/node/node_<Y>_/hugepages/<hugepages-size_dir>/nr-hugepages

    In this command, replace the variables as follows:

    • <X> with the number of huge pages to reserve. This value depends on the number of VMs and how much memory they will have. If you are running a single VM, start with two 1GB pages.
    • <Y> with the number of the NUMA node where real-time vCPUs are pinned.
    • <hugepage-size_dir> with the huge-page size in kB. For instance, for 2MB hugepages, this would be hugepages-2048kB.
    Important

    This command sets up huge pages transiently. As a result, you must use the command after every host reboot before you start any real-time VMs. To avoid this, perform the following optional step, which makes huge pages persistent.

  3. Optional: If you want to make the huge-page configuration persistent, also do the following:

    1. Create a file named /usr/lib/systemd/system/hugetlb-gigantic-pages.service with the following contents: 

      [Unit]
Description=HugeTLB Gigantic Pages Reservation
DefaultDependencies=no
Before=dev-hugepages.mount
ConditionPathExists=/sys/devices/system/node
ConditionKernelCommandLine=default_hugepagesz=1G

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/systemd/hugetlb-reserve-pages

[Install]
WantedBy=sysinit.target

    2. Create a file named /usr/lib/systemd/hugetlb-reserve-pages` with the following contents: 

      #!/bin/bash
nodes_path=/sys/devices/system/node/
if [ ! -d $nodes_path ]; then
	echo "ERROR: $nodes_path does not exist"
	exit 1
fi

reserve_pages()
{
	echo $1 > $nodes_path/$2/hugepages/hugepages-1048576kB/nr_hugepages
}

# This example reserves 2 1G pages on node0 and 1 1G page on node1. You
# can modify it to your needs or add more lines to reserve memory in
# other nodes. Don't forget to uncomment the lines, otherwise then won't
# be executed.
# reserve_pages 2 node0
# reserve_pages 1 node1

    3. Enable early boot reservation by using the following commands: 

      $ chmod +x /usr/lib/systemd/hugetlb-reserve-pages
$ sudo systemctl enable hugetlb-gigantic-pages
$ sudo systemctl status hugetlb-gigantic-pages
    4. Uncomment the bottom two lines of /usr/lib/systemd/hugetlb-reserve-pages and update them based on your huge-page reservation requirements.
  4. Reboot to apply all the configuration changes.

To verify that the BIOS of your real-time host has been successfully set up for low-latency workloads, use the hwlatdetect program.

Prerequisites

Procedure

  1. Run the hwladetect utility for at least an hour, and ensure that the measured latency does not exceed 1 microsecond (μs). 

    # hwlatdetect --threshold=1μs --duration=60m

  hwlatdetect:  test duration 60 minutes
	parameters:
		Latency threshold: 1μs
		Sample window:     1000000μs
		Sample width:      500000μs
		Non-sampling period:  500000μs
		Output File:       None

Starting test
test finished
Max Latency: 0us
Samples recorded: 0
Samples exceeding threshold: 0

  2. Optional: For improved validation, run the same test for 24 hours. 

    # hwlatdetect --threshold=1μs --duration=24h

Next steps

After you have configured the host for real-time virtual machines (VMs), you must verify that it is set up correctly. To do so, check the settings for the kernel, huge pages, and isolated CPUs, and ensure that the TuneD profile is active.

Prerequisites

Procedure

  1. View the content of the /proc/cmdline file, and check that the values for the following parameters correspond with how you configured them:

    • Real-time kernel
    • Huge pages

    • Isolated CPUs

      For example: 

      cat /proc/cmdline

BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-70.13.1.rt21.83.el9_0.x86_64 root=/dev/mapper/rhel_virtlab505-root ro crashkernel=auto resume=/dev/mapper/rhel_virtlab505-swap rd.lvm.lv=rhel_virtlab505/root rd.lvm.lv=rhel_virtlab505/swap console=ttyS1,115200 default_hugepages=1G skew_tick=1 isolcpus=1,3,5,7,9,11,13,14,15 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,3,5,7,9,11,13,14,15 rcu_nocbs=1,3,5,7,9,11,13,14,15

  2. Ensure that the realtime-virtual-host tuned profile is active. 

    $ tuned-adm active
Current active profile: realtime-virtual-host

  3. Check the number of huge memory pages. For example: 

    $ cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages

2

Next steps

To ensure that the RHEL for Real Time host or guest that you set up maintains low latency when under heavy load, perform real-time latency stress tests.

Prerequisites

Procedure

  1. Add stress to the housekeeping cores. To do so, start compiling the linux kernel on twice the number of housekeeping cores that you have set up in the previous sections.

    1. Clone the Linux kernel repository and move to its directory. 

      # git clone https://github.com/torvalds/linux.git ; cd linux

    2. Create a default configuration for the kernel compilation. 

      # make defconfig

    3. Start compiling the Linux kernel. 

      # while true; do make -j <double-number-of-housekeeping-cpus> && make clean; done

  2. Perform a cyclictest procedure on the host for 12 hours. In the following example, replace <list_isolated_cores> with a list of cores isolated for real-time tasks, such as 1,3,5,7,9,11,13,14,15. 

    # cyclictest -m -q -p95 --policy=fifo -D 12h -h60 -t <number_of_isolated_cpus> -a <list_isolated_cores> -mainaffinity <list_housekeeping_cpus> -i 200

    When using a modern high-end AMD64 or Intel 64 processor (also known as x86_64), the optimal value of Max Latencies in the output is under 40 microsecond (μs). To terminate the test if the measured latency exceeds 40μs, add the -b 40 option to the command.

  3. Perform an OS-level latency test (OSLAT) on the host for 12 hours. 

    # ./oslat --cpu-list <list_isolated_cores> --rtprio 1 --D 12h -w memmove -m 4K

    When using a modern high-end x86_64 processor, the optimal value of Maximum in the output is under 20 μs. To terminate the test if the measured latency exceeds 20 μs, add the -T 20 option to the command.

Chapter 4. Setting up real-time virtual machines
Copy link

To set up a virtual machine (VM) with a RHEL 9 for Real Time guest operating system, you must create a VM, configure its guest, and optimize and test the VM’s performance.

To correctly set up a RHEL real-time (RT) virtual machine (VM), you must first have a plan for optimal pinning of the VM’s virtual CPUs (vCPUs) to the physical CPUs of the host.

Prerequisites

Procedure

  1. View the CPU topology of your host system: 

    # lstopo-no-graphics

    The following example output shows a system with 32 physical cores with enabled hyperthreading, divided into 2 sockets ("packages"), each with 4 CPU dies. The system also has 250 GB of RAM split across 2 NUMA nodes.

    Note that the following examples in this procedure are based on this topology. 

    Machine (250GB total)
  Package L#0
    NUMANode L#0 (P#0 124GB)
    Die L#0 + L3 L#0 (16MB)
      L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#32)
      L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
        PU L#2 (P#1)
        PU L#3 (P#33)
      L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
        PU L#4 (P#2)
        PU L#5 (P#34)
      L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
        PU L#6 (P#3)
        PU L#7 (P#35)
    Die L#1 + L3 L#1 (16MB)
      L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
        PU L#8 (P#4)
        PU L#9 (P#36)
      L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
        PU L#10 (P#5)
        PU L#11 (P#37)
      L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
        PU L#12 (P#6)
        PU L#13 (P#38)
      L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
        PU L#14 (P#7)
        PU L#15 (P#39)
    Die L#2 + L3 L#2 (16MB)
      L2 L#8 (1024KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
        PU L#16 (P#8)
        PU L#17 (P#40)
      L2 L#9 (1024KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
        PU L#18 (P#9)
        PU L#19 (P#41)
      L2 L#10 (1024KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
        PU L#20 (P#10)
        PU L#21 (P#42)
      L2 L#11 (1024KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
        PU L#22 (P#11)
        PU L#23 (P#43)
    Die L#3 + L3 L#3 (16MB)
      L2 L#12 (1024KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
        PU L#24 (P#12)
        PU L#25 (P#44)
      L2 L#13 (1024KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
        PU L#26 (P#13)
        PU L#27 (P#45)
      L2 L#14 (1024KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
        PU L#28 (P#14)
        PU L#29 (P#46)
      L2 L#15 (1024KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
        PU L#30 (P#15)
        PU L#31 (P#47)
  Package L#1
    NUMANode L#1 (P#1 126GB)
    Die L#4 + L3 L#4 (16MB)
      L2 L#16 (1024KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16
        PU L#32 (P#16)
        PU L#33 (P#48)
      L2 L#17 (1024KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17
        PU L#34 (P#17)
        PU L#35 (P#49)
      L2 L#18 (1024KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18
        PU L#36 (P#18)
        PU L#37 (P#50)
      L2 L#19 (1024KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19
        PU L#38 (P#19)
        PU L#39 (P#51)
    Die L#5 + L3 L#5 (16MB)
      L2 L#20 (1024KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20
        PU L#40 (P#20)
        PU L#41 (P#52)
      L2 L#21 (1024KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21
        PU L#42 (P#21)
        PU L#43 (P#53)
      L2 L#22 (1024KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22
        PU L#44 (P#22)
        PU L#45 (P#54)
      L2 L#23 (1024KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23
        PU L#46 (P#23)
        PU L#47 (P#55)
    Die L#6 + L3 L#6 (16MB)
      L2 L#24 (1024KB) + L1d L#24 (32KB) + L1i L#24 (32KB) + Core L#24
        PU L#48 (P#24)
        PU L#49 (P#56)
      L2 L#25 (1024KB) + L1d L#25 (32KB) + L1i L#25 (32KB) + Core L#25
        PU L#50 (P#25)
        PU L#51 (P#57)
      L2 L#26 (1024KB) + L1d L#26 (32KB) + L1i L#26 (32KB) + Core L#26
        PU L#52 (P#26)
        PU L#53 (P#58)
      L2 L#27 (1024KB) + L1d L#27 (32KB) + L1i L#27 (32KB) + Core L#27
        PU L#54 (P#27)
        PU L#55 (P#59)
    Die L#7 + L3 L#7 (16MB)
      L2 L#28 (1024KB) + L1d L#28 (32KB) + L1i L#28 (32KB) + Core L#28
        PU L#56 (P#28)
        PU L#57 (P#60)
      L2 L#29 (1024KB) + L1d L#29 (32KB) + L1i L#29 (32KB) + Core L#29
        PU L#58 (P#29)
        PU L#59 (P#61)
      L2 L#30 (1024KB) + L1d L#30 (32KB) + L1i L#30 (32KB) + Core L#30
        PU L#60 (P#30)
        PU L#61 (P#62)
      L2 L#31 (1024KB) + L1d L#31 (32KB) + L1i L#31 (32KB) + Core L#31
        PU L#62 (P#31)
        PU L#63 (P#63)

  2. Based on the output of lstopo-no-graphics and your required real-time VM setup, determine how to pin your vCPUs to physical CPUs. The following items show XML configurations effective for the example host output above and a real-time VM with 4 vCPUs:

    • The following pinning placement uses an exclusive core for each vCPU. For such pinning configuration to be effective, the assigned physical CPUs must be isolated on the host and must not have any processes running on them. 

      <cputune>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='5'/>
  <vcpupin vcpu='2' cpuset='6'/>
  <vcpupin vcpu='3' cpuset='7'/>
[...]

    • The following pinning placement uses an exclusive L3 core for each vCPU: 

      <cputune>
  <vcpupin vcpu='0' cpuset='16'/>
  <vcpupin vcpu='1' cpuset='20'/>
  <vcpupin vcpu='2' cpuset='24'/>
  <vcpupin vcpu='3' cpuset='28'/>
[...]

Verification

To prepare a virtual machine (VM) environment for real-time workloads, create a new VM and adjust its configuration for low-latency performance.

Prerequisites

Procedure

  1. Use the virt-install utility to create a RHEL 9 VM with the following properties:

    • The VM has 2 or more assigned vCPUs
    • The VM uses huge pages for memory backing.

    The following example command creates a VM named RHEL9-RT that fits the mentioned requirements: 

    # virt-install -n RHEL9-RT \
    --os-variant=rhel9.6 --memory=3072,hugepages=yes \
    --memorybacking hugepages=yes,size=1,unit=G,locked=yes \
    --vcpus=4 --numatune=1 --disk path=./rhel9-rt.img,bus=virtio,cache=none,format=raw,io=threads,size=30 \
    --graphics none --console pty,target_type=serial \
    -l downloads/rhel9.iso \
    --extra-args 'console=ttyS0,115200n8 serial'

  2. After the installation finishes, shut down the VM. 

    # virsh shutdown <RHEL9-RT>

  3. Open the XML configuration of the VM. 

    # virsh edit <RHEL9-RT>

  4. Adjust the CPU configuration as follows:

    • On AMD64 and Intel 64 hosts: 

      <cpu mode='host-model' check='partial'>
    <feature policy='require' name='tsc-deadline'/>
</cpu>

    • On 64-bit ARM hosts: 

      <cpu mode="host-passthrough" check="none"/>

  5. Remove non-essential virtual hardware from the VM to improve its performance.

    1. Delete the section for the virtio RNG device. 

        <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
  </rng>

    2. Remove USB devices, such as the following: 

      <hostdev mode='subsystem' type='usb' managed='yes'>
  <source>
    <vendor id='0x1234'/>
    <product id='0xabcd'/>
  </source>
</hostdev>

    3. Remove serial devices, such as the following: 

      <serial type='dev'>
  <source path='/dev/ttyS0'/>
  <target port='0'/>
</serial>

    4. Remove the QXL device. 

      <video>
  <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
</video>

    5. Disable the graphical display. 

      <graphics type='vnc' ports='-1' autoport='yes' listen='127.0.0.1'>
  <listen type='address' address='127.0.0.1'>
</graphics>

    6. In the USB controller setting, change the model to none to disable it. 

      <controller type='usb' index='0' model='none'/>

    7. Remove the Trusted Platform Module (TPM) configuration, so that it does not interfere with RT operations. 

        <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
  </tpm>

    8. Disable the memballoon function. 

        <memballoon model='none'>

    9. In the <features> section of the configuration, ensure that the PMU and vmport features are disabled, to avoid the latency they might cause. 

        <features>
     [...]
     <pmu state='off'/>
     <vmport state='off'/>
  </features>

      On 64-bit ARM hosts, it is required to only disable PMU. 

        <features>
     [...]
     <pmu state='off'/>
  </features>

  6. Edit the <numatune> section to set up the NUMA nodes. 

      <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>

  7. Edit the <cputune> section of the configuration to set up vCPU NUMA pinning as planned out in Optimizing vCPU pinning for real-time virtual machines.

    The following example configures a VM with 4 vCPUs and these parameters:

    • The isolated core 15 from NUMA node 0 is the non-realtime vCPU
    • Cores 16, 47, and 48, from NUMA nodes 1 - 3, are the real-time vCPU
    • The configuration pins all the QEMU I/O threads to the host housekeeping cores (0 and 32). 
    <cputune>
  <vcpupin vcpu='0' cpuset='15'/>
  <vcpupin vcpu='1' cpuset='47'/>
  <vcpupin vcpu='2' cpuset='16'/>
  <vcpupin vcpu='3' cpuset='48'/>
  <emulatorpin cpuset='0,32'/>
  <emulatorsched scheduler='fifo' priority='1'/>
  <vcpusched vcpus='0' scheduler='fifo' priority='1'/>
  <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  <vcpusched vcpus='2' scheduler='fifo' priority='1'/>
  <vcpusched vcpus='3' scheduler='fifo' priority='1'/>
</cputune>
    Note

    If your host uses hardware with enabled hyperthreading, also ensure that your <cputune> configuration meets the following requirements:

    • Assign the siblings of a physical core to perform either real-time or housekeeping tasks.
    • Use both the siblings of a physical core in the same VM.
    • For vCPUs that are pinned to the siblings of the same physical core, assign the vCPU to the same task (real-time processes or housekeeping) as the sibling.

    Note that the example configuration above meet these requirements.

  8. Save and exit the XML configuration.

Troubleshooting

Verification

  • On the host, view the configuration of the VM and verify that it has the required parameters: 

    # virsh dumpxml <RHEL9-RT>

To optimize a RHEL 9 virtual machine (VM) environment for real-time workloads, configure the guest operating system for low-latency performance.

Prerequisites

Procedure

  1. Start the VM.

  2. Install real-time packages in the guest operating system. 

    # dnf install -y kernel-rt tuned tuned-profiles-realtime tuned-profiles-nfv realtime-tests

  3. Adjust the virtual guest profile for tuned. To do so, edit the /etc/tuned/realtime-virtual-guest-variables.conf file and add the following lines: 

    isolated_cores=<isolated-core-nrs>
isolate_managed_irq=Y

    Replace <isolated-core-nrs> with the numbers of host cores that you want to isolate for real-time workloads.

  4. Ensure that irqbalance is disabled in the guest operating system. 

    # rpm -q irqbalance && systemctl stop irqbalance && systemctl disable irqbalance

  5. Activate the realtime-virtual-guest profile for tuned. 

    # tuned-adm profile realtime-virtual-guest

  6. Ensure that the real-time kernel is used by the guest operating system by default. 

    # grubby --set-default vmlinuz-5.14.0-XXX.el9.x86_64+rt
  7. Configure huge pages for the guest operating system in the same way as in the host. For instructions, see Configuring huge pages for real-time virtualization hosts.

Verification

Troubleshooting

If the results of the stress test exceed the required latency, do the following:

  1. Perform the stress tests on the host again. If the latency results are suboptimal, adjust the host configuration of TuneD and huge pages, and re-test. For instructions, see Configuring TuneD for the real-time virtualization host and Configuring huge pages for real-time virtualization hosts.
  2. If the stress test results on the host show sufficiently low latency but on the guest they do not, use the trace-cmd utility to generate a detailed test report. For instructions, see Collecting data to troubleshoot latency issues for RHEL real-time guests (Red Hat Knowledgebase).

Eviction of cache lines might cause performance issues in real-time virtual machines (VMs). Optionally, to avoid this problem, use the User Interface for Resource Control (resctrlfs`) feature to manage your caches and cache partitions:

  • Divide the main memory cache of the host system into partitions
  • Assign separate tasks to each partition
  • Assign vCPUs that run real-time applications to one cache partition
  • Assign vCPUs and host CPUs that run housekeeping workloads to a different cache partition

Prerequisites

  • You have created a real-time virtual machine on your host. For instructions, see Installing a RHEL real-time guest operating system.

  • Your host is using an Intel processor that supports L2 or L3 cache partitioning. To ensure this is the case:

    1. Install the intel-cmt-cat utility. 

      # dnf install intel-cmt-cat

    2. Use the pqos utility to display your core cache details. 

      # pqos -d

[...]
   Allocation
      Cache Allocation Technology (CAT)
        L3 CAT

      This output indicates your CPU supports L3 cache partitioning.

Procedure

The following steps assume that you have the following NUMA pinning assignment of vCPUs to CPUs. 

<cputune>
    <vcpupin vcpu='0' cpuset='16'/>
    <vcpupin vcpu='1' cpuset='17'/>
    <vcpupin vcpu='2' cpuset='18'/>
    <vcpupin vcpu='3' cpuset='19'/>

  1. Mount the resctrl file system. This makes it possible to use the resource control capabilities of the processor. 

    # mount -t resctrl resctrl /sys/fs/resctrl

    Depending on whether your system supports L2 or L3 cache partitioning, a subdirectory named L2 or L3 is mounted in the /sys/fs/resctrl/info directory. The following steps assume your system supports L3 cache partitioning.

  2. Move into the cache directory and list its content. 

    # cd /sys/fs/resctrl/info/L3/; ls

bit_usage  cbm_mask  min_cbm_bits  num_closids  shareable_bits  sparse_masks

  3. View the value of the cbm_mask file. 

    # cat cbm_mask

ffff

    This value represents the cache bitmask in a hexadecimal code. ffff means that all 16 bits of the cache can be used by the workload.

  4. View the value of the "shareable_bits" file. 

    # cat shareable_bits

0

    This value represents partitions of the L3 cache that are shared by other executing processes, such as I/O, and therefore should not be used in exclusive cache partitions. 0 means that you can use all the L3 cache partitions.

  5. View the schemata file to see the global cache allocation. 

    # cat /sys/fs/resctrl/schemata

L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ffff;10=ffff;12=ffff;14=ffff

    This output indicates that for the L3 cache partition, CPU sockets 0, 2, 4, 6, 8, 10, 12, and 14 are fully allocated to the default control group. In this example, CPU sockets 16 - 19 are pinned to vCPUs 0-3.

  6. Determine what cache distribution you want to set for real-time applications. For example, for an even distribution of 8 MB for both real-time applications and housekeeping applications:

    • The cache bitmask for real-time applications is ff00
    • The cache bitmask for housekeeping applications is 00ff

  7. Adjust the default schemata file with your required cache allocation for housekeeping processes. For example, to assign 8MB to CPU socket 8, do the following: 

    # echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=00ff;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/schemata

  8. Create a specific control group for real-time processes, for example part1. 

    # mkdir /sys/fs/resctrl/part1

  9. Create a schemata file for the part1 control group, and set it to use cache allocation that does not conflict with the housekeeping cache allocation. 

    # echo "L3:0=ffff;2=ffff;4=ffff;6=ffff;8=ff00;10=ffff;12=ffff;14=ffff" > /sys/fs/resctrl/part1/schemata

    With this setting, the L3 cache 8 uses half of its cache allocation for real-time processes and the other half for the housekeeping processes. All the other L3 caches can be used freely by both real-time processes and housekeeping.

  10. Assign the CPUs pinned to real-time vCPUs (in this case 17, 18, and 19) to this control group. 

    # echo 17,18,19 > part1/cpus_list

Verification

  • If you previously tested the latency of the VM, run the cyclictest utility again. For instructions, see Stress testing the real-time virtualization system.

    If the maximum latency is lower than previously, you have set up cache protection correctly.

While installing a RHEL 9 virtual machine (VM) on your real-time host, you might encounter one of the following errors. Use the following recommendations to fix or work around these issues.

  • Error: Host doesn’t support any virtualization options

    • Ensure that virtualization is enabled in the host BIOS

    • Check that cpu flags on your host contain vmx for Intel, or svm for AMD. 

      $ cat /proc/cpuinfo | grep vmx

    • Check that the lsmod command detects the kvm and kvm_intel or kvm_amd modules. 

      $ lsmod | grep kvm

    • Make sure the kernel-rt-kvm package is installed. 

      $ dnf info kernel-rt-kvm
    • Check if the /dev/kvm device exists.
    • Run the virt-host-validate utility to detect any further issues.
    • Run kvm-unit-tests.

  • Permission-related issues while accessing the disc image

    • In the /etc/libvirt/qemu.conf file, uncomment the group = and user = lines.

    • Restart the virtqemud service. 

      $ service virtqemud restart

Legal Notice
Copy link

Copyright © 2025 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top