Chapter 1. Before You Start Tuning Your Red Hat Enterprise Linux for Real Time System

Procedure 1.1. To successfully test your system and interpret the results:

1. Check the vendor documentation for any tuning steps required for low latency operation.
   This step aims to reduce or remove any System Management Interrupts (SMIs) that would transition the system into System Management Mode (SMM). While a system is in SMM it is running firmware and not running operating system code, meaning any timers that expire while in SMM will have to wait until the system transitions back into normal operation. This can cause unexplained latencies since SMIs cannot be blocked by Linux and the only indication that we actually took an SMI may be found in vendor-specific performance counter registers.

   Warning

   Red Hat strongly recommends that you do not completely disable SMIs, as it can result in catastrophic hardware failure.
2. Ensure that RHEL-RT and rt-tests package is installed.
   This step verifies that you have tuned the system properly.
3. Run the hwlatdetect program.
   hwlatdetect looks for hardware-firmware induced latencies by polling the clock-source and looking for unexplained gaps.
   Generally, you do not need to run any sort of load on the system while running hwlatdetect, since the program is looking for latencies introduced by hardware architecture or BIOS/EFI firmware.
   A typical output of hwlatdetect looks like this:

   # hwlatdetect --duration=60s
   hwlatdetect:  test duration 60 seconds
   	detector: tracer
   	parameters:
   		Latency threshold: 10us
   		Sample window:     1000000us
   		Sample width:      500000us
   		Non-sampling period:  500000us
   		Output File:       None
   
   Starting test
   test finished
   Max Latency: Below threshold
   Samples recorded: 0
   Samples exceeding threshold: 0

   The above result represents a system that was tuned to minimize system interruptions from firmware.
   However, not all systems can be tuned to minimize system interruptions as shown below:

   # hwlatdetect --duration=10s
   hwlatdetect:  test duration 10 seconds
   	detector: tracer
   	parameters:
   		Latency threshold: 10us
   		Sample window:     1000000us
   		Sample width:      500000us
   		Non-sampling period:  500000us
   		Output File:       None
   
   Starting test
   test finished
   Max Latency: 18us
   Samples recorded: 10
   Samples exceeding threshold: 10
   SMIs during run: 0
   ts: 1519674281.220664736, inner:17, outer:15
   ts: 1519674282.721666674, inner:18, outer:17
   ts: 1519674283.722667966, inner:16, outer:17
   ts: 1519674284.723669259, inner:17, outer:18
   ts: 1519674285.724670551, inner:16, outer:17
   ts: 1519674286.725671843, inner:17, outer:17
   ts: 1519674287.726673136, inner:17, outer:16
   ts: 1519674288.727674428, inner:16, outer:18
   ts: 1519674289.728675721, inner:17, outer:17
   ts: 1519674290.729677013, inner:18, outer:17

   The above result shows that while doing consecutive reads of the system clocksource, there were 10 delays that showed up in the 15-18 us range.
   hwlatdetect was using the tracer mechanism as the detector for unexplained latencies. Previous versions used a kernel module rather than ftrace tracer.
   parameters report a latency and how the detection was run. The default latency threshold was 10 microseconds (10 us), the sample window was 1 second, the sampling window was 0.5 seconds.
   As a result, tracer ran a detector thread that ran for one half of each second of the specified duration.
   The detector thread runs a loop which does the following pseudocode:

   t1 = timestamp()
   	loop:
   		t0 = timestamp()
   		if (t0 - t1) > threshold
   		   outer = (t0 - t1)
   		t1 = timestamp
   		if (t1 - t0) > threshold
   		   inner = (t1 - t0)
   		if inner or outer:
   		   print
   		if t1 > duration:
   		   goto out
   		goto loop
   	out:

   The inner loop comparison checks that t0 - t1 does not exceed the specified threshold (10 us default). The outer loop comparison checks the time between the bottom of the loop and the top t1 - t0. The time between consecutive reads of the timestamp register should be dozens of nanoseconds (essentially a register read, a comparison and a conditional jump) so any other delay between consecutive reads is introduced by firmware or by the way the system components were connected.

   Note

   The values printed out by the hwlatdetector for inner and outer are the best case maximum latency. The latency values are the deltas between consecutive reads of the current system clocksource (usually the Time Stamp Counter or TSC register, but potentially the HPET or ACPI power management clock) and any delays between consecutive reads, introduced by the hardware-firmware combination.