realtime_interrupt

The realtime_interrupt test validates interrupt-driven realtime performance on systems that run Red Hat Enterprise Linux.

The realtime_interrupt test validates interrupt-driven realtime performance on systems running Red Hat Enterprise Linux for Real Time. You can run this test to measure latency caused by hardware interrupts and timer events. This ensures that the hardware meets realtime performance requirements under sustained load.

Note
  • The realtime_interrupt test is required for all realtime certifications.
  • The test is planned only on systems running the Red Hat Realtime Kernel (kernel-rt).
What the test covers
The realtime_interrupt test validates interrupt-driven realtime performance on systems running RHEL. It measures latencies of 150.0 µs and 50.0 µs caused by hardware interrupts and timer events. These measurements verify that the hardware can meet realtime performance requirements under sustained load.
RHEL version supported
  • RHEL 8
  • RHEL 9
  • RHEL 10
What the test does
  • Preliminary tests: Before running the realtime test, the following checks are performed:
    • Isolated CPUs - Detects and identifies isolated CPUs for running realtime workloads.
    • Tuned version - Verifies the installed version of the tuned package on the system under test.
    • Active Tuned profile - Confirms that the active tuned profile is set to realtime. If not, the test fails.
Realtime tests
  • Hardware latency detection (hwlatdetect Subtest): The hwlatdetect subtest measures system latency caused by hardware interrupts and detects delays from BIOS, firmware, or other hardware components. It uses the hwlatdetect tool, which runs a high-priority task with the SCHED_FIFO scheduling policy and rt_priority of 95. This setup disables software-based interrupts, ensuring that all measured latency comes only from hardware or firmware. By minimizing interference from the operating system, hwlatdetect helps differentiate between hardware-induced and operating system-induced latency. This makes it easier to identify potential causes of performance problems in realtime or latency-sensitive systems.
  • Timer event latency (rteval subtest): The rteval subtest measures timer event latency by running the cyclictest tool. The cyclictest tool creates one measurement thread per CPU, each running at a high realtime priority. Every thread performs a recurring measurement cycle with a 100 µs period as follows:
    • Capture a timestamp (t1).
    • Sleep for the defined period.
    • Capture a second timestamp (t2).
    • Calculate latency as: latency = t2 − (t1 + period).
    • Repeat continuously. Note

      Latency represents the difference between the theoretical wakeup time (t1+period) and the actual wakeup time (t2). Each measurement thread records minimum, maximum, and average latency values.

      While [application]cyclictest runs, rteval simultaneously applies system loads, including:

      • A parallel Linux kernel compilation.
      • A scheduler benchmark (hackbench). After completing the run, rteval performs a statistical analysis of the collected data. It calculates mean, median, mode, variance, and standard deviation to assess system stability and latency consistency.
Configuration Verification

For RHEL 8 and RHEL 9, the test verifies the following:

  • Isolated CPUs are correctly configured in /sys/devices/system/cpu/isolated.
  • The tuned package version is ≥ 2.19.0, which supports automatic setup of isolated_cores.
  • The realtime tuned profile is active.
  • Cores are reserved for housekeeping tasks instead of fully utilizing all CPUs. If any of these checks fail, the test issues a warning and continues with execution.
Preparing for the test
  • Install and boot the realtime kernel before adding the system to the certification. The certification tool automatically detects the realtime kernel and schedules the realtime_interrupt test.
  • For RHEL 8 and RHEL 9, with tuned version ≥ 2.19.0, set the tuned profile to realtime and reboot.
  • Ensure that BIOS and firmware settings are optimized for realtime performance. Note
    • If you require realtime tuning assistance, then you must provide Red Hat access to your system so that the necessary configuration or BIOS changes can be made.
    • Newly installed kernels inherit the kernel command-line parameters from your previously configured kernels. For more information, see the Changing kernel command-line parameters for all boot entries section.
Running the test

The realtime test is non-interactive.

  • Run the following command and then select the appropriate realtime_interrupt test name from the list that displays:
    # rhcert-run

    The test only appears when the system is running kernel-rt.

  • Run the 50.0 µs realtime interrupt test by entering the following command:
    # rhcert-cli run --test hardware/realtime_interrupt_50
Evaluation criteria

The realtime_interrupt test assesses realtime performance based on two key metrics reported by rteval.

  • rteval or cyclictest thresholds for 150.0 µs:
    rteval_maxlatency = 150.0 µs max latency
    rteval_maxstddev  = 3.0 µs max standard deviation
    Note

    These values apply only to 1-socket and 2-socket systems.

  • rteval or cyclictest thresholds for 50.0 µs:
    rteval_maxlatency = 50.0 µs max latency
    rteval_maxstddev  = 3.0 µs max standard deviation
  • 4-socket systems thresholds:
    rteval_maxlatency = 250.0 µs max latency
    rteval_maxstddev  = 5.0 µs max standard deviation
Success criteria
All measured values remain within thresholds.
Run time
The system management mode, hwlatdetect, runs for two hours on all machines. The timer event analysis, rteval, runs for twelve hours on all machines. The required supportable test adds about a minute to the overall run time.