Optimizing RHEL 9 for Real Time for low latency operation

Linux uses three main thread scheduling policies.

The syslog server forwards log messages from programs over a network. The less often this occurs, the larger the pending transaction is likely to be. If the transaction is very large, it can cause an I/O spike. To prevent this, keep the interval reasonably small.

The system logging daemon, syslogd, is used to collect messages from different programs. It also collects information reported by the kernel from the kernel logging daemon, klogd. Typically, syslogd logs to a local file, but it can also be configured to log over a network to a remote logging server.

Every running application uses system resources. Ensuring that there are no unnecessary applications running on your system can significantly improve performance.

The taskset utility only works on CPU affinity and has no knowledge of other NUMA resources such as memory nodes. If you want to perform process binding in conjunction with NUMA, use the numactl command instead of taskset.

For more information about the NUMA API, see Andi Kleen’s whitepaper An NUMA API for Linux.

The debugfs file system is specially designed for debugging and making information available to users. It is mounted automatically in RHEL 8 in the /sys/kernel/debug/ directory.

InfiniBand is a type of communications architecture often used to increase bandwidth, improve quality of service (QOS), and provide for failover. It can also be used to improve latency by using the Remote Direct Memory Access (RDMA) mechanism.

The support for InfiniBand on RHEL for Real Time is the same as the support available on Red Hat Enterprise Linux 9. For more information, see Configuring InfiniBand and RDMA networks.

RoCEE (RDMA over Converged Enhanced Ethernet) is a protocol that implements Remote Direct Memory Access (RDMA) over Ethernet networks. It allows you to maintain a consistent, high-speed environment in your data centers, while providing deterministic, low latency data transport for critical transactions.

High Performance Networking (HPN) is a set of shared libraries that provides RoCEE interfaces into the kernel. Instead of going through an independent network infrastructure, HPN places data directly into remote system memory using standard Ethernet infrastructure, resulting in less CPU overhead and reduced infrastructure costs.

Support for RoCEE and HPN under RHEL for Real Time does not differ from the support offered under RHEL 8.

When testing the real-time workload in a container running on the main RHEL kernel, add the following options to the podman run command as necessary:

The real-time threads have higher priority than the standard threads. The policies have scheduling priority values that range from the minimum value of 1 to the maximum value of 99.

The following policies are critical to real-time:

Each SCHED_DEADLINE task is characterized by period, runtime, and deadline parameters. The values for these parameters are integers of nanoseconds.

You can make persistent changes to kernel tuning parameters by adding the parameter to the /etc/sysctl.conf file.

Traditional UNIX and POSIX signals have their uses, especially for error handling, but they are not suitable as an event delivery mechanism in real-time applications. This is because the current Linux kernel signal handling code is quite complex, mainly due to legacy behavior and the many APIs that need to be supported. This complexity means that the code paths that are taken when delivering a signal are not always optimal, and long latencies can be experienced by applications.

The original motivation behind UNIX signals was to multiplex one thread of control (the process) between different "threads" of execution. Signals behave somewhat like operating system interrupts. That is, when a signal is delivered to an application, the application’s context is saved and it starts executing a previously registered signal handler. Once the signal handler completes, the application returns to executing where it was when the signal was delivered. This can get complicated in practice.

Signals are too non-deterministic to trust in a real-time application. A better option is to use POSIX Threads (pthreads) to distribute your workload and communicate between various components. You can coordinate groups of threads using the pthreads mechanisms of mutexes, condition variables, and barriers. The code paths through these relatively new constructs are much cleaner than the legacy handling code for signals.

The sched_yield command is a synchronization mechanism that can allow lower priority threads a chance to run. This type of request is prone to failure when issued from within a poorly-written application.

A higher priority thread can call sched_yield() to allow other threads a chance to run. The calling process gets moved to the tail of the queue of processes running at that priority. When this occurs in a situation where there are no other processes running at the same priority, the calling process continues running. If the priority of that process is high, it can potentially create a busy loop, rendering the machine unusable.

When a SCHED_DEADLINE task calls sched_yield(), it gives up the configured CPU, and the remaining runtime is immediately throttled until the next period. The sched_yield() behavior allows the task to wake up at the start of the next period.

The scheduler is better able to determine when, and if, there actually are other threads waiting to run. Avoid using sched_yield() on any real-time task.

The systemd command can be used to set real-time priority for services launched during the boot process. Some kernel threads can be given a very high priority. This allows the default priorities to integrate well with the requirements of the Real Time Specification for Java (RTSJ). RTSJ requires a range of priorities from 10 to 89.

For deployments where RTSJ is not in use, there is a wide range of scheduling priorities below 90 that can be used by applications. Use extreme caution when scheduling any application thread above priority 49 because it can prevent essential system services from running, because it can prevent essential system services from running. This can result in unpredictable behavior, including blocked network traffic, blocked virtual memory paging, and data corruption due to blocked filesystem journaling.

If any application threads are scheduled above priority 89, ensure that the threads run only a very short code path. Failure to do so would undermine the low latency capabilities of the RHEL for Real Time kernel.

When developing real-time application, consider resolving symbols at startup to avoid non-deterministic latencies during program execution. Resolving symbols at startup can slow down program initialization. You can instruct Dynamic Libraries to load at application startup by setting the LD_BIND_NOW variable with ld.so, the dynamic linker/loader.

For example, the following shell script exports the LD_BIND_NOW variable with a value of 1, then runs a program with a scheduler policy of FIFO and a priority of 1.

#!/bin/sh

LD_BIND_NOW=1
export LD_BIND_NOW

chrt --fifo 1 _/opt/myapp/myapp-server &_

BIOS power management options help save power by changing the system clock frequency or by putting the CPU into one of various sleep states. These actions are likely to affect how quickly the system responds to external events.

To improve response times, disable all power management options in the BIOS.

Error Detection and Correction (EDAC) units are devices for detecting and correcting errors signaled from Error Correcting Code (ECC) memory. Usually EDAC options range from no ECC checking to a periodic scan of all memory nodes for errors. The higher the EDAC level, the more time the BIOS uses. This may result in missing crucial event deadlines.

To improve response times, turn off EDAC. If this is not possible, configure EDAC to the lowest functional level.

System Management Interrupts (SMIs) are a hardware vendors facility to ensure that the system is operating correctly. The BIOS code usually services the SMI interrupt. SMIs are typically used for thermal management, remote console management (IPMI), EDAC checks, and various other housekeeping tasks.

If the BIOS contains SMI options, check with the vendor and any relevant documentation to determine the extent to which it is safe to disable them.

The runtime verification (RV) monitors are encapsulated inside the RV monitor abstraction and coordinate between the defined specifications and the kernel trace to capture runtime events in trace files. The RV monitor includes:

In addition to verifying and monitoring a system at runtime, you can enable a response to an unexpected system event. The forms of reaction can vary from capturing an event in the trace file to initiating an extreme reaction, such as a shut-down to avoid a system failure on safety critical systems.

Reactors are reaction methods available for RV monitors to define reactions to system events as required. By default, monitors provide a trace output of the actions.

Runtime verification (RV) monitors are classified into following types:

The user interface is located at /sys/kernel/tracing/rv and resembles the tracing interface. The user interface includes the mentioned files and folders.

It is not required to run any load on the system while running the hwlatdetect program, because the test looks for latencies introduced by the hardware architecture or BIOS or EFI firmware. The default values for hwlatdetect are to poll for 0.5 seconds each second, and report any gaps greater than 10 microseconds between consecutive calls to fetch the time. hwlatdetect returns the best maximum latency possible on the system. Therefore, if you have an application that requires maximum latency values of less than 10us and hwlatdetect reports one of the gaps as 20us, then the system can only guarantee latency of 20us.

The hardware latency detector (hwlatdetect) uses the tracer mechanism to detect latencies introduced by the hardware architecture or BIOS/EFI firmware. By checking the latencies measured by hwlatdetect, you can determine if a potential hardware is suitable to support the RHEL for Real Time kernel.

The table for Testing method, parameters, and results, lists the parameters and the latency values detected by the hwlatdetect utility.

After finding the suitable hardware-firmware combination, the next step is to test the real-time performance of the system while under a load.

With the rteval utility, you can test a system’s real-time performance under load.

On real-time, the taskset command helps to set or retrieve the CPU affinity of a running process. The taskset command takes -p and -c options. The -p or --pid option work an existing process and does not start a new task. The -c or --cpu-list specify a numerical list of processors instead of a bitmask. The list can contain more than one items, separated by comma, and a range of processors. For example, 0,5,7,9-11.

You can also set processor affinity using the real-time sched_setaffinity() system call.

With the cpusets mechanism, you can assign a set of CPUs and memory nodes for SCHED_DEADLINE tasks. In a task set that has high and low CPU utilizing tasks, isolating a CPU to run the high utilization task and scheduling small utilization tasks on different sets of CPU, enables all tasks to meet the assigned runtime.

A common source of latency spikes is when multiple CPUs contend on common locks in the kernel timer tick handler. The usual lock responsible for the contention is xtime_lock, which is used by the timekeeping system and the Read-Copy-Update (RCU) structure locks. By using skew_tick=1, you can offset the timer tick per CPU to start at a different time and avoid potential lock conflicts.

The skew_tick kernel command line parameter might prevent latency fluctuations on moderate to large systems with large core-counts and have latency-sensitive workloads.

The pcscd daemon manages connections to parallel communication (PC or PCMCIA) and smart card (SC) readers. Although pcscd is usually a low priority task, it can often use more CPU than any other daemon. Therefore, the additional background noise can lead to higher preemption costs to real-time tasks and other undesirable impacts on determinism.

The mlock() and mlockall() system calls lock a specified memory range and do not page this memory. The following are the mlock() system call groups:

The mlock() system calls, lock pages in the address range starting at addr and continuing for len bytes. When the call returns successfully, all pages that contain a part of the specified address range stay in the memory until unlocked later.

With mlockall() system calls, you can lock all mapped pages into the specified address range. Memory locks do not stack. Any page locked by several calls will unlock the specified address range or the entire region with a single munlock() system call. With munlockall() system calls, you can unlock the entire program space.

The status of the pages contained in a specific range depends on the value in the flags argument. The flags argument can be 0 or MLOCK_ONFAULT.

Memory locks are not inherited by a child process through fork and automatically removed when a process terminates.

The real-time mlock() system calls use the addr parameter to specify the start of an address range and len to define the length of the address space in bytes. The alloc_workbuf() function dynamically allocates a memory buffer and locks it. Memory allocation is done by the posix_memalig() function to align the memory area to a page. The function free_workbuf() unlocks the memory area.

To lock and unlock real-time memory with mlockall() and munlockall() system calls, set the flags argument to 0 or one of the constants: MCL_CURRENT or MCL_FUTURE. With MCL_FUTURE, a future system call, such as mmap2(), sbrk2(), or malloc3(), might fail, because it causes the number of locked bytes to exceed the permitted maximum.

For large memory allocations on real-time systems, the memory allocation (malloc) method uses the mmap() system call to find memory space. You can assign and lock memory areas by setting MAP_LOCKED in the flags parameter. As mmap() assigns memory on a page basis, it avoids two locks on the same page, which prevents the double-lock or single-unlock problems.

The parameters for memory lock system call and the functions they perform are listed and described in the mlock parameters table.

You can configure the timerlat tracer by adding timerlat in the curret_tracer file of the tracing system. The current_tracer file is generally mounted in the /sys/kernel/tracing directory. The timerlat tracer measures the interrupt requests (IRQs) and saves the trace output for analysis when a thread latency is more than 100 microseconds.

The timerlat tracer is built on top of osnoise tracer. Therefore, you can set the options in the /osnoise/config directory to trace and capture information for thread scheduling latencies.

The rtla-timerlat-top tracer displays a summary of the periodic output from the timerlat tracer. The tracer output also provides information about each operating system noise and events, such as osnoise, and tracepoints. You can view this information by using the -t option.

By using the rtla timerlat top --help command, you can view the help usage on options for the rtla-timerlat-top tracer.

The Linux kernel includes the real-time analysis (rtla) tool, which provides an interface for the operating system noise (osnoise) tracer. The rtla-osnoise tracer creates a thread that runs periodically for a specified given period. At the start of a period, the thread disables interrupts, starts sampling, and captures the time in a loop.

The rtla-osnoise tracer provides the following capabilities:

The rtla-osnoise tracer prints a run report with the following information about the noise sources at the conclusion of the period:

You can configure the rtla-osnoise tracer by adding osnoise in the curret_tracer file of the tracing system. The current_tracer file is generally mounted in the /sys/kernel/tracing/ directory. The rtla-osnoise tracer measures the interrupt requests (IRQs) and saves the trace output for analysis when a thread latency is more than 20 microseconds for a single noise occurrence.

The configuration options for the rtla-osnoise tracer is available in the /sys/kernel/tracing/ directory.

The rtla-osnoise includes a set of tracepoints to identify the source of the operating system noise (osnoise).

The osnoise/options file includes a set of on and off configuration options for the rtla-osnoise tracer.

The rtla osnoise-top tracer measures and prints a periodic summary from the osnoise tracer along with the information about the occurrence counters of the interference source.

By using the rtla osnoise top --help command, you can view the help usage on the available options for the rtla-osnoise-top tracer.

Disabling the atime attribute increases performance and decreases power usage by limiting the number of writes to the file-system journal.

The teletype (tty) default kernel console enables your interaction with the system by passing input data to the system and displaying the output information about the graphics console.

Not configuring the graphics console, prevents it from logging on the graphics adapter. This makes tty0 unavailable to the system and helps disable printing messages on the graphics console.

You can control the amount of output messages that are sent to the graphics console by configuring the required log levels in the /proc/sys/kernel/printk file.

Multiple instances of clock sources found in multiprocessor systems, such as non-uniform memory access (NUMA) and Symmetric multiprocessing (SMP), interact among themselves and the way they react to system events, such as CPU frequency scaling or entering energy economy modes, determine whether they are suitable clock sources for the real-time kernel.

The preferred clock source is the Time Stamp Counter (TSC). If the TSC is not available, the High Precision Event Timer (HPET) is the second best option. However, not all systems have HPET clocks, and some HPET clocks can be unreliable.

In the absence of TSC and HPET, other options include the ACPI Power Management Timer (ACPI_PM), the Programmable Interval Timer (PIT), and the Real Time Clock (RTC). The last two options are either costly to read or have a low resolution (time granularity), therefore they are sub-optimal for use with the real-time kernel.

The list of available clock sources in your system is in the /sys/devices/system/clocksource/clocksource0/available_clocksource file.

The currently used clock source in your system is stored in the /sys/devices/system/clocksource/clocksource0/current_clocksource file.

Sometimes the best-performing clock for a system’s main application is not used due to known problems on the clock. After ruling out all problematic clocks, the system can be left with a hardware clock that is unable to satisfy the minimum requirements of a real-time system.

Requirements for crucial applications vary on each system. Therefore, the best clock for each application, and consequently each system, also varies. Some applications depend on clock resolution, and a clock that delivers reliable nanoseconds readings can be more suitable. Applications that read the clock too often can benefit from a clock with a smaller reading cost (the time between a read request and the result).

In these cases it is possible to override the clock selected by the kernel, provided that you understand the side effects of the override and can create an environment which will not trigger the known shortcomings of the given hardware clock.

You can compare the speed of the clocks in your system. Reading from the TSC involves reading a register from the processor. Reading from the HPET clock involves reading a memory area. Reading from the TSC is faster, which provides a significant performance advantage when timestamping hundreds of thousands of messages per second.

The current generation of AMD64 Opteron processors can be susceptible to a large gettimeofday skew. This skew occurs when both cpufreq and the Time Stamp Counter (TSC) are in use. RHEL for Real Time provides a method to prevent this skew by forcing all processors to simultaneously change to the same frequency. As a result, the TSC on a single processor never increments at a different rate than the TSC on another processor.

The clock_timing program reads the current clock source 10 million times. In conjunction with the time utility it measures the amount of time needed to do this.

Modern processors actively transition to higher power saving states (C-states) from lower states. Unfortunately, transitioning from a high power saving state back to a running state can consume more time than is optimal for a real-time application. To prevent these transitions, an application can use the Power Management Quality of Service (PM QoS) interface.

With the PM QoS interface, the system can emulate the behavior of the idle=poll and processor.max_cstate=1 parameters, but with a more fine-grained control of power saving states. idle=poll prevents the processor from entering the idle state. processor.max_cstate=1 prevents the processor from entering deeper C-states (energy-saving modes).

When an application holds the /dev/cpu_dma_latency file open, the PM QoS interface prevents the processor from entering deep sleep states, which cause unexpected latencies when they are being exited. When the file is closed, the system returns to a power-saving state.

You can control power management transitions by configuring power management states with one of the following ways:

Isolating interrupts (IRQs) from user processes on different dedicated CPUs can minimize or eliminate latency in real-time environments.

Interrupts are generally shared evenly between CPUs. This can delay interrupt processing when the CPU has to write new data and instruction caches. These interrupt delays can cause conflicts with other processing being performed on the same CPU.

It is possible to allocate time-critical interrupts and processes to a specific CPU (or a range of CPUs). In this way, the code and data structures for processing this interrupt will most likely be in the processor and instruction caches. As a result, the dedicated process can run as quickly as possible, while all other non-time-critical processes run on the other CPUs. This can be particularly important where the speeds involved are near or at the limits of memory and available peripheral bus bandwidth. Any wait for memory to be fetched into processor caches will have a noticeable impact in overall processing time and determinism.

In practice, optimal performance is entirely application-specific. For example, tuning applications with similar functions for different companies, required completely different optimal performance tunings.

The irqbalance daemon is enabled by default and periodically forces interrupts to be handled by CPUs in an even manner. However in real-time deployments, irqbalance is not needed, because applications are typically bound to specific CPUs.

You can use the IRQ balancing service to specify which CPUs you want to exclude from consideration for interrupt (IRQ) balancing. The IRQBALANCE_BANNED_CPUS parameter in the /etc/sysconfig/irqbalance configuration file controls these settings. The value of the parameter is a 64-bit hexadecimal bit mask, where each bit of the mask represents a CPU core.

Assigning CPU affinity enables binding and unbinding processes and threads to a specified CPU or range of CPUs. This can reduce caching problems.

The taskset utility uses the process ID (PID) of a task to view or set its CPU affinity. You can use the utility to run a command with a chosen CPU affinity.

To set the affinity, you need to get the CPU mask to be as a decimal or hexadecimal number. The mask argument is a bitmask that specifies which CPU cores are legal for the command or PID being modified.

The /proc/sys/vm/panic_on_oom file contains a value which is the switch that controls Out of Memory (OOM) behavior. When the file contains 1, the kernel panics on OOM and stops functioning as expected.

The default value is 0, which instructs the kernel to call the oom_killer() function when the system is in an OOM state. Usually, oom_killer() terminates unnecessary processes, which allows the system to survive.

You can change the value of /proc/sys/vm/panic_on_oom.

You can prioritize the processes that get terminated by the oom_killer() function. This can ensure that high-priority processes keep running during an OOM state. Each process has a directory, /proc/PID. Each directory includes the following files:

In an Out of Memory state, the oom_killer() function terminates processes with the highest oom_score.

You can prioritize the processes to terminate by editing the oom_adj file for the process.

You can disable the oom_killer() function for a process by setting oom_adj to the reserved value of -17. This will keep the process alive, even in an OOM state.

The tuna command-line interface (CLI) is a tool to help you make tuning changes to your system.

The tuna tool is designed to be used on a running system, and changes take place immediately. This allows any application-specific measurement tools to see and analyze system performance immediately after changes have been made.

The tuna CLI now has a set of commands, which formerly were the action options. These commands are:

You can view the commands with the tuna -h command. For each command, there are optional arguments, which you can view with the tuna <command> -h command. For example, with the tuna isolate -h command, you can view the options for isolate.

You can use the tuna CLI to isolate interrupts (IRQs) from user processes on different dedicated CPUs to minimize latency in real-time environments. For more information about isolating CPUs, see Interrupt and process binding.

You can use the tuna CLI to move interrupts (IRQs) to dedicated CPUs to minimize or eliminate latency in real-time environments. For more information about moving IRQs, see Interrupt and process binding.

You can use the tuna CLI to change process scheduling policy and priority.

# tuna priority FIFO:1 --threads=42369,42416,43859

# tuna show_threads --threads=42369,42416,43859

Thread priorities are set using a series of levels, ranging from 0 (lowest priority) to 99 (highest priority). The systemd service manager can be used to change the default priorities of threads after the kernel boots.

Using systemd, you can set up real-time priority for services launched during the boot process.

Unit configuration directives are used to change the priority of a service during boot process. The boot process priority change is done by using the following directives in the service section of /etc/systemd/system/service.service.d/priority.conf:

Using systemd, you can specify the CPUs on which services can run.

Scheduler priorities are defined in groups, with some groups dedicated to particular kernel functions.

The goal of coalesce tuning is to minimize the number of interrupts required for a given workload. In high-throughput situations, the goal is to have as few interrupts as possible while maintaining a high data rate. In low-latency situations, more interrupts can be used to handle traffic quickly.

You can adjust the settings on your network card to increase or decrease the number of packets that are combined into a single interrupt. As a result, you can achieve improved throughput or latency for your traffic.

On an Ethernet link, continuous data transmission between a network interface and a switch port can lead to full buffer capacity. Full buffer capacity results in network congestion. In this case, when the sender transmits data at a higher rate than the processing capacity of the receiver, packet loss can occur due to the lower data processing capacity of a network interface on the other end of the link which is a switch port.

The flow control mechanism manages data transmission across the Ethernet link where each sender and receiver has different sending and receiving capacities. To avoid packet loss, the Ethernet flow control mechanism temporarily suspends the packet transmission to manage a higher transmission rate from a switch port. Note that routers do not forward pause frames beyond a switch port.

When receive (RX) buffers become full, a receiver sends pause frames to the transmitter. The transmitter then stops data transmission for a short sub-second time frame, while continuing to buffer incoming data during this pause period. This duration provides enough time for the receiver to empty its interface buffers and prevent buffer overflow.

By default, most of the network drivers in Red Hat Enterprise Linux have pause frame support enabled. To display the current settings of a network interface, enter:

# ethtool --show-pause enp1s0
Pause parameters for enp1s0:
...
RX:     on
TX:     on
...

Verify with your switch vendor to confirm if your switch supports pause frames.

The trace-cmd utility provides a front-end to the ftrace utility.

You can use the trace-cmd utility to access all ftrace functionalities.

The command examples show how to trace kernel functions by using the trace-cmd utility.

Choosing the CPUs to isolate requires careful consideration of the CPU topology of the system. Different use cases require different configuration:

The hwloc package provides utilities that are useful for getting information about CPUs, including lstopo-no-graphics and numactl.

The initial mechanism for isolating CPUs is specifying the boot parameter isolcpus=cpulist on the kernel boot command line. The recommended way to do this for RHEL for Real Time is to use the TuneD daemon and its tuned-profiles-realtime package.

The nohz and nohz_full parameters modify activity on specified CPUs. To enable these kernel boot parameters, you need to use one of the following TuneD profiles: realtime-virtual-host, realtime-virtual-guest, or cpu-partitioning.

If a SCHED_OTHER task spawns a large number of other tasks, they will all run on the same CPU. The migration task or softirq will try to balance these tasks so they can run on idle CPUs.

The sched_nr_migrate option can be adjusted to specify the number of tasks that will move at a time. Because real-time tasks have a different way to migrate, they are not directly affected by this. However, when softirq moves the tasks, it locks the run queue spinlock, thus disabling interrupts.

If there are a large number of tasks that need to be moved, it occurs while interrupts are disabled, so no timer events or wakeups will be allowed to happen simultaneously. This can cause severe latencies for real-time tasks when sched_nr_migrate is set to a large value.

Increasing the sched_nr_migrate variable provides high performance from SCHED_OTHER threads that spawn many tasks at the expense of real-time latency.

For low real-time task latency at the expense of SCHED_OTHER task performance, the value must be lowered. The default value is 8.

Turning off TCP timestamps can reduce TCP performance spikes.

Generating timestamps can cause TCP performance spikes. You can reduce TCP performance spikes by disabling TCP timestamps. If you find that generating TCP timestamps is not causing TCP performance spikes, you can enable them.

You can view the status of TCP timestamp generation.

You can offload RCU callbacks using the rcu_nocbs and rcu_nocb_poll kernel parameters.

You can assign a housekeeping CPU to handle all RCU callback threads. To do this, use the tuna command and move all RCU callbacks to the housekeeping CPU.

This action relieves all CPUs other than CPU X from handling RCU callback threads.

Although the RCU offload threads can perform the RCU callbacks on another CPU, each CPU is responsible for awakening the corresponding RCU offload thread. You can relieve a CPU from this responsibility,

You can trace latencies using the ftrace utility.

The following are the main files in the /sys/kernel/debug/tracing/ directory.

Depending on how the kernel is configured, not all tracers may be available for a given kernel. For the RHEL for Real Time kernels, the trace and debug kernels have different tracers than the production kernel does. This is because some of the tracers have a noticeable overhead when the tracer is configured into the kernel, but not active. Those tracers are only enabled for the trace and debug kernels.

The following provides a number of examples for changing the filtering of functions being traced. You can use the * wildcard at both the beginning and end of a word. For example: *irq\* will select all functions that contain irq in the name. The wildcard cannot, however, be used inside a word.

Encasing the search term and the wildcard character in double quotation marks ensures that the shell will not attempt to expand the search to the present working directory.

POSIX is a standard for implementing and representing time sources. You can assign a POSIX clock to an application without affecting other applications in the system. This is in contrast to hardware clocks which are selected by the kernel and implemented across the system.

The function used to read a given POSIX clock is clock_gettime(), which is defined at <time.h>. The kernel counterpart to clock_gettime() is a system call. When a user process calls clock_gettime():

However, the context switch from the user application to the kernel has a CPU cost. Even though this cost is very low, if the operation is repeated thousands of times, the accumulated cost can have an impact on the overall performance of the application. To avoid context switching to the kernel, thus making it faster to read the clock, support for the CLOCK_MONOTONIC_COARSE and CLOCK_REALTIME_COARSE POSIX clocks was added, in the form of a virtual dynamic shared object (VDSO) library function.

Time readings performed by clock_gettime(), using one of the _COARSE clock variants, do not require kernel intervention and are executed entirely in user space. This yields a significant performance gain. Time readings for _COARSE clocks have a millisecond (ms) resolution, meaning that time intervals smaller than 1 ms are not recorded. The _COARSE variants of the POSIX clocks are suitable for any application that can accommodate millisecond clock resolution.

The example code output shows using the clock_gettime function with the CLOCK_MONOTONIC_COARSE POSIX clock.

#include <time.h>

main()
{
	int rc;
	long i;
	struct timespec ts;

	for(i=0; i<10000000; i++) {
		rc = clock_gettime(CLOCK_MONOTONIC_COARSE, &ts);
	}
}

You can improve upon the example above by adding checks to verify the return code of clock_gettime(), to verify the value of the rc variable, or to ensure the content of the ts structure is to be trusted.

Applications that require low latency on every packet sent must be run on sockets with the TCP_NODELAY option enabled. This sends buffer writes to the kernel as soon as an event occurs.

If applications have several buffers that are logically related and must be sent as one packet, apply one of the following workarounds to avoid poor performance:

When a logical packet has been built in the kernel by the various components in the application, the socket should be uncorked, allowing TCP to send the accumulated logical packet immediately.

The TCP_NODELAY option sends buffer writes to the kernel when events occur, with no delays. Enable TCP_NODELAY using the setsockopt() function.

The TCP_CORK option prevents TCP from sending any packets until the socket is "uncorked".

Mutual exclusion (mutex) algorithms are used to prevent processes simultaneously using a common resource. A fast user-space mutex (futex) is a tool that allows a user-space thread to claim a mutex without requiring a context switch to kernel space, provided the mutex is not already held by another thread.

When you initialize a pthread_mutex_t object with the standard attributes, a private, non-recursive, non-robust, and non-priority inheritance-capable mutex is created. This object does not provide any of the benfits provided by the pthreads API and the RHEL for Real Time kernel.

To benefit from the pthreads API and the RHEL for Real Time kernel, create a pthread_mutexattr_t object. This object stores the attributes defined for the futex.

To define any additional capabilities for the mutex, create a pthread_mutexattr_t object. This object stores the defined attributes for the futex. This is a basic safety procedure that you must always perform.

For more information about advanced mutex attributes, see Advanced mutex attributes.

When you initialize a pthread_mutex_t object with the standard attributes, a private, non-recursive, non-robust, and non-priority inheritance-capable mutex is created.

The following advanced mutex attributes can be stored in a mutex attribute object:

After the mutex has been created using the mutex attribute object, you can keep the attribute object to initialize more mutexes of the same type, or you can clean it up. The mutex is not affected in either case.

The perf record command is used for collecting system-wide statistics. It can be used in all processors.

You can analyze the results of the perf on other systems using the perf archive command. This may not be necessary, if:

The data from the perf record feature can now be investigated directly using the perf report command.

There are a range of available options to get the hardware tracepoint activity.

You can view specific events using the perf stat command.

A floating point unit is the functional part of the processor that performs floating point arithmetic operations. Floating point units handle mathematical operations and make floating numbers or decimal calculations simpler.

Using the --matrix-method option, you can stress test the CPU floating point operations and processor data cache.

The stress-ng tool runs multiple stress tests. In the default mode, it runs the specified stressor mechanisms in parallel.

To measure the CPU heat generation, the specified stressors generate high temperatures for a short time duration to test the system’s cooling reliability and stability under maximum heat generation. Using the --matrix-size option, you can measure CPU temperatures in degrees Celsius over a short time duration.

The stress-ng tool can measure a stress test throughput by measuring the bogo operations per second. The size of a bogo operation depends on the stressor being run. The test outcomes are not precise, but they provide a rough estimate of the performance.

You must not use this measurement as an accurate benchmark metric. These estimates help to understand the system performance changes on different kernel versions or different compiler versions used to build stress-ng. Use the --metrics-brief option to display the total available bogo operations and the matrix stressor performance on your machine.

When under memory pressure, the kernel starts writing pages out to swap. You can stress the virtual memory by using the --page-in option to force non-resident pages to swap back into the virtual memory. This causes the virtual machine to be heavily exercised. Using the --page-in option, you can enable this mode for the bigheap, mmap and virtual machine (vm) stressors. The --page-in option, touch allocated pages that are not in core, forcing them to page in.

Running timers at high frequency can generate a large interrupt load. The –timer stressor with an appropriately selected timer frequency can force many interrupts per second.

With stress-ng, you can test and analyze the page fault rate by generating major page faults in a page that are not loaded in the memory. On new kernel versions, the userfaultfd mechanism notifies the fault finding threads about the page faults in the virtual memory layout of a process.

The CPU stress test contains methods to exercise a CPU. You can print an output to view all methods using the which option.

If you do not specify the test method, by default, the stressor checks all the stressors in a round-robin fashion to test the CPU with each stressor.

The verify mode validates the results when a test is active. It sanity checks the memory contents from a test run and reports any unexpected failures.

All stressors do not have the verify mode and enabling one will reduce the bogo operation statistics because of the extra verification step being run in this mode.

You can use all the following options with both the real time kernel and the main RHEL kernel. The kernel-rt package brings potential determinism improvements and allows the usual troubleshooting.

You can run a container built with a Dockerfile.

This example shows the podman run command with the required, real time-specific options. For example:

The chrt utility checks and adjusts scheduler policies and priorities. It can start new processes with the desired properties or change the properties of a running process.

You can display the current scheduling policy and scheduling priority for a specified process.

Real-time processes use a set of functions to control policy and priority. You can use the sched_getscheduler() function to display the scheduler policy for a specified process.

You can use the sched_get_priority_min() and sched_get_priority_max() functions to check the valid priority range for a given scheduler policy.

The sched_get program is now ready and can be run from the directory in which it is saved.

The SCHED_RR (round-robin) policy differs slightly from the SCHED_FIFO (first-in, first-out) policy. SCHED_RR allocates concurrent processes that have the same priority in a round-robin rotation. In this way, each process is assigned a timeslice. The sched_rr_get_interval() function reports the timeslice allocated to each process.

Timeslice information is returned as a timespec. This is the number of seconds and nanoseconds since the base time of 00:00:00 GMT, 1 January 1970:

struct timespec {
  time_t tv_sec;  /* seconds / long tv_nsec; / nanoseconds */
};

The sched_getattr() function queries the scheduling policy currently applied to the specified process, identified by PID. If PID equals to zero, the policy of the calling process is retrieved.

The size argument should reflect the size of the sched_attr structure as known to userspace. The kernel fills out sched_attr::size to the size of its sched_attr structure.

If the input structure is smaller, the kernel returns values outside the provided space. As a result, the system call fails with an E2BIG error. The other sched_attr fields are filled out as described in The sched_attr structure.

The sched_attr structure contains or defines a scheduling policy and its associated attributes for a specified thread. The sched_attr structure has the following form:

    struct sched_attr {
     u32 size;
     u32 sched_policy
     u64 sched_flags
     s32 sched_nice
     u32 sched_priority

     /* SCHED_DEADLINE fields */
               u64 sched_runtime
               u64 sched_deadline
               u64 sched_period
           };

SCHED_DEADLINE fields must be specified only for deadline scheduling:

A process can voluntarily yield the CPU either because it has completed, or because it is waiting for an event, such as data from a disk, a key press, or for a network packet.

A process can also involuntarily yield the CPU. This is called preemption and occurs when a higher priority process wants to use the CPU.

Preemption can have a particularly negative impact on system performance, and constant preemption can lead to a state known as thrashing. This problem occurs when processes are constantly preempted, and no process ever runs to completion.

Changing the priority of a task can help reduce involuntary preemption.

You can check the voluntary and involuntary preemption status for a specified process. The statuses are stored in /proc/PID/status.

The chrt utility checks and adjusts scheduler policies and priorities. It can start new processes with the desired properties, or change the properties of a running process.

The chrt utility options include command options and parameters specifying the process and priority for the command.

Real-time processes use a different set of library calls to control policy and priority. The following library calls are used to set the priority of non-real-time processes.

These functions adjust the nice value of a non-real-time process. The nice value serves as a suggestion to the scheduler on how to order the list of ready-to-run, non-real-time processes to be run on a processor. The processes at the head of the list run before the ones further down the list.

The scheduler policy and other parameters can be set using the sched_setscheduler() function. Currently, real-time policies have one parameter, sched_priority. This parameter is used to adjust the priority of the process.

The sched_setscheduler() function requires three parameters, in the form: sched_setscheduler(pid_t pid, int policy, const struct sched_param *sp);.

If the process ID is zero, the sched_setscheduler() function acts on the calling process.

The following code excerpt sets the scheduler policy of the current process to the SCHED_FIFO scheduler policy and the priority to 50:

struct sched_param sp = { .sched_priority = 50 };
int ret;

ret = sched_setscheduler(0, SCHED_FIFO, &sp);
if (ret == -1) {
  perror("sched_setscheduler");
  return 1;
}

The sched_setparam() function is used to set the scheduling parameters of a particular process. This can then be verified using the sched_getparam() function.

Unlike the sched_getscheduler() function, which only returns the scheduling policy, the sched_getparam() function returns all scheduling parameters for the given process.

struct sched_param sp;
int ret;

ret = sched_getparam(0, &sp);
sp.sched_priority += 2;
ret = sched_setparam(0, &sp);

If this code were used in a real application, it would need to check the return values from the function and handle any errors appropriately.

The sched_setattr() function sets the scheduling policy and its associated attributes for an instance ID specified in PID. When pid=0, sched_setattr() acts on the process and attributes of the calling thread.

The real-time scheduling policies share one main characteristic: they run until a higher priority thread interrupts the thread or the threads wait, either by sleeping or performing I/O.

In the case of SCHED_RR, the operating system interrupts a running thread so that another thread of equal SCHED_RR priority can run. In either of these cases, no provision is made by the POSIX specifications that define the policies for allowing lower priority threads to get any CPU time. This characteristic of real-time threads means that it is easy to write an application, which monopolizes 100% of a given CPU. However, this causes problems for the operating system. For example, the operating system is responsible for managing both system-wide and per-CPU resources and must periodically examine data structures describing these resources and perform housekeeping activities with them. But if a core is monopolized by a SCHED_FIFO thread, it cannot perform its housekeeping tasks. Eventually the entire system becomes unstable and can potentially crash.

On the RHEL for Real Time kernel, interrupt handlers run as threads with a SCHED_FIFO priority. The default priority is 50. A cpu-hog thread with a SCHED_FIFO or SCHED_RR policy higher than the interrupt handler threads can prevent interrupt handlers from running. This causes the programs waiting for data signaled by those interrupts to starve and fail.

The real-time kernel includes a safeguard mechanism to enable allocating bandwidth for use by the real-time tasks. The safeguard mechanism is known as real-time scheduler throttling.

The default values for the real-time throttling mechanism define that the real-time tasks can use 95% of the CPU time. The remaining 5% will be devoted to non real-time tasks, such as tasks running under SCHED_OTHER and similar scheduling policies. It is important to note that if a single real-time task occupies the 95% CPU time slot, the remaining real-time tasks on that CPU will not run. Only the non real-time tasks use the remaining 5% of CPU time. The default values can have the following performance impacts:

The real-time scheduler throttling is controlled by the following parameters in the /proc file system:

Thread starvation occurs when a thread is on a CPU run queue for longer than the starvation threshold and does not make progress. A common cause of thread starvation is to run a fixed-priority polling application, such as SCHED_FIFO or SCHED_RR bound to a CPU. Since the polling application does not block for I/O, this can prevent other threads, such as kworkers, from running on that CPU.

An early attempt to reduce thread starvation is called as real-time throttling. In real-time throttling, each CPU has a portion of the execution time dedicated to non real-time tasks. The default setting for throttling is on with 95% of the CPU for real-time tasks and 5% reserved for non real-time tasks. This works if you have a single real-time task causing starvation but does not work if there are multiple real-time tasks assigned to a CPU. You can work around the problem by using: