Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 9. Using the rteval container for real time task execution

The rteval (real-time evaluation) container in Red Hat Enterprise Linux (RHEL) for Real Time ensures low-latency execution of critical tasks. It measures timer wake-up times under various system loads to maintain real-time responsiveness and ensure timely task execution.

The rteval tool sets the measurement process (using tools like cyclictest or rtla) as a high-priority task. This measurement process has a higher priority than the load generated on the machine. As a result, rteval container measure the wake-up times of the real-time tasks under different loads, ensuring that the system can handle real-time workloads effectively.

9.1. Testing a host for rteval container
Copy link

To run the rteval container on latency-sensitive workloads, you must tune the host machine because the container technology does not require an additional kernel in the virtualization stack. Most tuning strategies applicable to bare metal are also applicable to container environments.

You must apply the realtime profile with tuned-adm with default parameters defined in realtime-variables.conf file.

The realtime profile performs the following tasks:

  • Sets various kernel command-line options.
  • Detects Non-Uniform Memory Access (NUMA) topology.
  • Assigns all CPUs except the first CPU of each node to the isolcpus set when more than one NUMA node is present.

Configure the host machine for rteval container.

Prerequisites

  • The host machine is running on a Red Hat Enterprise Linux version 9.6 and later.
  • The tuned and tuned-profiles-realtime packages are installed.
  • The tuned service is running.
  • The podman application is installed and running.

Procedure

  1. Install the required packages: 

    $ sudo dnf install rteval kernel-rt podman -y
    Copy to Clipboard Toggle word wrap

  2. View the installed kernels: 

    $ sudo grubby --info=ALL
index=0
kernel="/boot/vmlinuz-5.XX.0-XX.X.X.el9_6.x86_64+rt"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=UUID=3e14acf4-a359-4045-b8fc-990ff83743ec rd.lvm.lv=rhel_rt-qe-11/root rd.lvm.lv=rhel_rt-qe-11/swap console=ttyS0,115200n81 $tuned_params"
root="/dev/mapper/rhel_rt--qe--11-root"
initrd="/boot/initramfs-5.XX.0-XX.X.X.el9_6.x86_64+rt.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.XX.0-XX.X.X.el9_6.x86_64+rt) 9.6 (Plow)"
id="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-5.XX.0-XX.X.X.el9_6.x86_64+rt"
index=1
kernel="/boot/vmlinuz-5.XX.0-XX.X.X.el9_6.x86_64"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=UUID=3e14acf4-a359-4045-b8fc-990ff83743ec rd.lvm.lv=rhel_rt-qe-11/root rd.lvm.lv=rhel_rt-qe-11/swap console=ttyS0,115200n81 $tuned_params"
root="/dev/mapper/rhel_rt--qe--11-root"
initrd="/boot/initramfs-5.XX.0-XX.X.X.el9_6.x86_64.img $tuned_initrd"
title="Red Hat Enterprise Linux (5.XX.0-XX.X.X.el9_6.x86_64) 9.6 (Plow)"
id="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-5.XX.0-XX.X.X.el9_6.x86_64"
index=2
kernel="/boot/vmlinuz-0-rescue-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
args="ro crashkernel=2G-64G:256M,64G-:512M resume=UUID=3e14acf4-a359-4045-b8fc-990ff83743ec rd.lvm.lv=rhel_rt-qe-11/root rd.lvm.lv=rhel_rt-qe-11/swap console=ttyS0,115200n81"
root="/dev/mapper/rhel_rt--qe--11-root"
initrd="/boot/initramfs-0-rescue-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.img"
title="Red Hat Enterprise Linux (0-rescue-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX) 9.6 (Plow)"
id="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-0-rescue"
    Copy to Clipboard Toggle word wrap

  3. Set the Real Time kernel as the default kernel: 

    $ select a in /boot/vmlinuz-*rt*; do grubby --set-default=$a; break; done
    Copy to Clipboard Toggle word wrap

  4. Apply the realtime profile with tuned-adm: 

    $ sudo tuned-adm profile realtime
    Copy to Clipboard Toggle word wrap

  5. Reboot the host machine: 

    $ sudo reboot
    Copy to Clipboard Toggle word wrap

Verification

  1. Verify the kernel version and tuning parameter: 

    $ sudo uname -r
5.XX.0-XX.X.X.el9_6.x86_64+rt
    Copy to Clipboard Toggle word wrap 
    $ sudo cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.XX.0-XX.X.X.el9_6.x86_64+rt root=/dev/mapper/rhel_rt--qe--11-root ro crashkernel=2G-64G:256M,64G-:512M resume=UUID=3e14acf4-a359-4045-b8fc-990ff83743ec rd.lvm.lv=rhel_rt-qe-11/root rd.lvm.lv=rhel_rt-qe-11/swap console=ttyS0,115200n81 skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 isolcpus=managed_irq,domain,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 intel_pstate=disable nosoftlockup
    Copy to Clipboard Toggle word wrap

9.2. Testing bare metal for baseline results
Copy link

Run the system on bare metal without any additional software except the essential operating system and the rteval container. This ensures that the system is optimized for low-latency performance, and the results are not affected by other software or processes.

Prerequisites

Procedure

  • There are two scenarios to run the rteval container:

    • On a single NUMA node: 

      $ rteval --duration 2h
      Copy to Clipboard Toggle word wrap

    • Run the rteval container on multiple NUMA nodes: 

      $ rteval --duration 2h --loads-cpulist 0,1 --measurement-cpulist 2-47
      Copy to Clipboard Toggle word wrap

After completing the bare metal test, use the rteval container as a baseline to configure real-world scenarios with your applications or services.

After tuning the host with the real time profile, you can further optimize performance by selectively placing containers on specific CPUs and adjust container runtime behavior. With these strategies, you can explore how CPU isolation and cgroup configurations affect latency in containerized workloads.

9.3.1. Running podman on all CPUs
Copy link

To run podman with the rteval container, tune your system by using the tuned realtime profile or custom system tuning. Determine whether you need CPU isolation for the scenarios you are measuring. Ensure that you set up CPU isolation correctly to avoid issues when running containers in certain scenarios.

Check isolcpus= argument in /proc/cmdline. If isolcpus is not set, your system is not isolating any CPUs, and you can run containers across all CPUs.

Prerequisites

  • isolcpus= argument in /proc/cmdline is not set to run containers across all CPUs.
  • The host machine is running on a Red Hat Enterprise Linux version 9.6 or later.
  • The podman service is running.
  • The rteval container is installed and running.

Procedure

  1. Log in to the podman registry: 

    $ podman login registry.redhat.io
    Copy to Clipboard Toggle word wrap

  2. Run rteval container. Select one of the following ways to run the container:

    • On all CPUs on a single NUMA node box: 

      $ podman run -it --rm --privileged --pids-limit=0 registry.redhat.io/rhel10/rteval \
	/bin/bash -c 'rteval --duration 2h’
      Copy to Clipboard Toggle word wrap

    • On a multi NUMA node machine: 

      $ podman run -it --rm --privileged --pids-limit=0 registry.redhat.io/rhel10/rteval \
	/bin/bash -c 'rteval --duration 2h --loads-cpulist 0,1 --measurement-cpulist 2-47
      Copy to Clipboard Toggle word wrap
      --pids-limit=0

      kcompile can run without hitting the container runtime’s default limit.

      kcompile is a command-line utility used to compile kernel modules for the currently running kernel without requiring to rebuild the entire kernel.

      --privileged
      the container can access all devices on the host system. This is necessary for rteval to run correctly.

These commands run a single container across all available nodes. The tuned service manages host tuning, enabling you to evaluate bare metal performance when using only a single CPU.

Verification

  • In a new terminal, list all containers including the rteval container to ensure it is running correctly: 

    $ podman ps -a
    Copy to Clipboard Toggle word wrap

9.3.2. Running podman with split CPU assignment
Copy link

You can assign different containers to different CPU sets for testing load separation and measurements. For example, you can run two different containers when only one NUMA node is present and you want to separate the loads and measurements into containers. In this case, both containers run on every CPU and no partitioning is used for tuning.

Example commands:

  • Loads container: 

    $ podman run -it --rm --privileged --pids-limit=0 registry.redhat.io/rhel10/rteval \
   	/bin/bash -c 'rteval --duration 2h --onlyload'
    Copy to Clipboard Toggle word wrap

  • Measurements container: 

    $ podman run -it --rm --privileged --pids-limit=0 registry.redhat.io/rhel10/rteval \
   	/bin/bash -c 'rteval --duration 2h --onlymeasure'
    Copy to Clipboard Toggle word wrap

For the scenario with partitioning on boxes that have more than one NUMA node, or a manually partitioned machine, the example commands are:

  • Loads container: 

    $ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 0,1 registry.redhat.io/rhel10/rteval \
/bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 0,1'
    Copy to Clipboard Toggle word wrap

  • Measurements container: 

    $ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 2-47 registry.redhat.io/rhel10/rteval \
/bin/bash -c 'rteval --duration 2h --noload --measurement-cpulist 2-47'
    Copy to Clipboard Toggle word wrap

After running these commands, the load container generates load on the housekeeping cores, while the measurement container operates on the isol_cpu set.

If no partitioning is configured, one container generates load across all CPUs on the system, and another container measures latency across all nodes.

In both scenarios, the loads and measurements are successfully separated between two containers.

You can adjust the housekeeping CPU set per NUMA node in the real time profile. This optimizes the performance of your system by ensuring that the housekeeping tasks are distributed equally across the NUMA nodes.

This is particularly useful for systems with multiple NUMA nodes, as it helps to reduce contention and improve overall performance.

The default realtime tuned profile reserves one housekeeping CPU per NUMA node (hk_per_numa=1). You can modify this behavior if you need more CPUs available for container workloads.

Prerequisites

  • The host machine is running on a Red Hat Enterprise Linux version 9.6 or later.
  • The tuned service is running.
  • The rteval container is installed and running.
  • The podman service is running.
  • The tuned-profiles-realtime package is installed.

Procedure

  1. Modify the realtime-variables.conf file to adjust the housekeeping CPU set per NUMA node.

    • Open the realtime-variables.conf file located at /etc/tuned in a text editor: 

      $ sudo vi /etc/tuned/realtime-variables.conf
      Copy to Clipboard Toggle word wrap

    • Locate the isolated_cores variable. By default, this is set to 1, meaning one core per NUMA node is reserved for isolated or non-housekeeping use. You can increase this value but it must be less than the total number of CPUs per NUMA node.

      The following example sets isolated_cores to 3 on a system with 24 cores per NUMA node: 

      isolated_cores=${f:calc_isolated_cores:3}
      Copy to Clipboard Toggle word wrap
  2. Save your changes and close the file.

  3. Reapply the tuned real time profile: 

    $ sudo tuned-adm profile realtime
    Copy to Clipboard Toggle word wrap

This results in a total of 6 CPUs (3 per NUMA node) generating load during the test, while the system reserves the remaining cores for the isolcpus set. This configuration is used for measurements. In some cases, mixed priority configurations might deploy containers on custom topology instead of isolcpus set.

Alternatively, you can manually specify a custom CPU range instead of relying on an automatic per-node count. This ensures full control over the isolated cores, making it easier to fine-tune systems with non-uniform topologies or specialized CPU layouts.

Verification

  1. Verify the changes in realtime-variables.conf file.
  2. Reboot the system to apply the changes.

  3. View the /proc/cmdline file to confirm the isolcpus setting: 

    $ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.XX.X-XX.X.X.el9_6.x86_64+rt root=/dev/mapper/rhel_rt--qe--11-root ro  crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=00cbf36d-ffaa-4285-a381-5c1d868eb3e3 rd.lvm.lv=rhel_rt-qe-11/root rd.lvm.lv=rhel_rt-qe-11/swap console=ttyS0,115200n81 skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 isolcpus=managed_irq,domain,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 intel_pstate=disable nosoftlockup
    Copy to Clipboard Toggle word wrap

To run multiple containers across isolated CPUs, you can use the --cpuset-cpus option to specify which CPUs each container should use. This divides the load across multiple isolated CPUs, improving performance and reducing contention.

You can divide the isolcpus set across multiple containers simulate the following tasks:

  • Concurrent latency-sensitive tasks.
  • Multiple loads across a partitioned system.

To simulate concurrent latency-sensitive tasks, you can assign specific isolated CPUs to each container. The following example demonstrates how to configure and run containers across different CPU sets.

Run one container on CPUs 0-6, another on CPUs 7-28, and a third on CPUs 29-47. Use the following commands: 

$ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 0-6 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 0-6'
Copy to Clipboard Toggle word wrap 
$ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 7-28 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 7-28'
Copy to Clipboard Toggle word wrap 
$ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 29-47 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 29-47'
Copy to Clipboard Toggle word wrap

Start the rteval load generator on the non-isolated CPU set. Next, simulate a high-throughput application, such as a high-speed database container, on a part of the isolcpus set. For this example, CPUs 7-28 are used to represent the high-speed database container. Run the following commands in separate terminal sessions to start the loads. 

$ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 0-6 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 0-6'
Copy to Clipboard Toggle word wrap

Then, in a separate terminal, generate some load on part of subsets of the isolated CPUs: 

$ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 20-30 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --onlyload --loads-cpulist 20-30'
Copy to Clipboard Toggle word wrap

Now, to run measurement threads on the remaining CPUs, you have two options. You can either deploy the two remaining subsets of the isolated CPUs to separate containers or run a single measurement container that utilizes both remaining CPU subsets.

  • Option 1: Deploy two measurement containers 

    $ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 7-19 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --noload --measurement-cpulist 7-19'
    Copy to Clipboard Toggle word wrap 
    $ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 31-47 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --noload --measurement-cpulist 31-47'
    Copy to Clipboard Toggle word wrap

  • Option 2: Deploy a single measurement container 

    $ podman run -it --rm --privileged --pids-limit=0 --cpuset-cpus 7-19,31-47 registry.redhat.io/rhel10/rteval \
    /bin/bash -c 'rteval --duration 2h --noload --measurement-cpulist 7-19,31-47'
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat