SupportConsoleDevelopersStart a trialContact

Chapter 12. Measuring scheduling latency using rtla-osnoise in RHEL for Real Time

An ultra-low latency is an environment that is optimized to process high volumes of data packets with low tolerance for delay. Providing exclusive resources to applications, including the CPU, is a prevalent practice in ultra-low-latency environments. For example, for high performance network processing in network functions virtualization (NFV) applications, a single application has the CPU power limit set to run tasks continuously.

The Linux kernel includes the real-time analysis (rtla) tool, which provides an interface for the operating system noise (osnoise) tracer. The operating system noise is the interference that occurs in an application as a result of activities inside the operating system. Linux systems can experience noise due to:

  • Non maskable interrupts (NMIs)
  • Interrupt requests (IRQs)
  • Soft interrupt requests (SoftIRQs)
  • Other system threads activity
  • Hardware-related jobs, such as non maskable high priority system management interrupts (SMIs)

12.1. The rtla-osnoise tracer

The Linux kernel includes the real-time analysis (rtla) tool, which provides an interface for the operating system noise (osnoise) tracer. The rtla-osnoise tracer creates a thread that runs periodically for a specified given period. At the start of a period, the thread disables interrupts, starts sampling, and captures the time in a loop.

The rtla-osnoise tracer provides the following capabilities:

  • Measure how much operating noise a CPU receives.
  • Characterize the type of operating system noise occurring in the CPU.
  • Print optimized trace reports that help to define the root cause of unexpected results.
  • Saves an interference counter for each interference source. The interference counter for non maskable interrupts (NMIs), interrupt requests (IRQs), software interrupt requests (SoftIRQs), and threads increase when the tool detects the entry events for these interferences.

The rtla-osnoise tracer prints a run report with the following information about the noise sources at the conclusion of the period:

  • Total amount of noise.
  • The maximum amount of noise.
  • The percentage of CPU that is allocated to the thread.
  • The counters for the noise sources.

12.2. Configuring the rtla-osnoise tracer to measure scheduling latency

You can configure the rtla-osnoise tracer by adding osnoise in the curret_tracer file of the tracing system. The current_tracer file is generally mounted in the /sys/kernel/tracing/ directory. The rtla-osnoise tracer measures the interrupt requests (IRQs) and saves the trace output for analysis when a thread latency is more than 20 microseconds for a single noise occurrence.

Procedure

  1. List the current tracer: 

    # cat /sys/kernel/tracing/current_tracer
nop

    The no operations (nop) is the default tracer.

  2. Add the timerlat tracer in the current_tracer file of the tracing system: 

    # cd /sys/kernel/tracing/
# echo osnoise > current_tracer

  3. Generate the tracing output: 

    # cat trace
# tracer: osnoise

12.3. The rtla-osnoise options for configuration

The configuration options for the rtla-osnoise tracer is available in the /sys/kernel/tracing/ directory.

Configuration options for rtla-osnoise

osnoise/cpus
Configures the CPUs for the osnoise thread to run on.
osnoise/period_us
Configures the period for a osnoise thread run.
osnoise/runtime_us
Configures the run duration for a osnoise thread.
osnoise/stop_tracing_us
Stops the system tracing if a single noise is more than the configured value. Setting 0 disables this option.
osnoise/stop_tracing_total_us
Stops the system tracing if the total noise is more than the configured value. Setting 0 disables this option.
tracing_thresh
Sets the minimum delta between two time() call reads to be considered as noise, in microseconds. When set to 0,tracing_thresh uses the default value, which is 5 microseconds.

12.4. The rtla-osnoise tracepoints

The rtla-osnoise includes a set of tracepoints to identify the source of the operating system noise (osnoise).

Trace points for rtla-osnoise

osnoise:sample_threshold
Displays a noise when the noise is more than the configured threshold (tolerance_ns).
osnoise:nmi_noise
Displays noise and the noise duration from non maskable interrupts (NMIs).
osnoise:irq_noise
Displays noise and the noise duration from interrupt requests (IRQs).
osnoise:softirq_noise
Displays noise and the noise duration from soft interrupt requests (SoftIRQs),
osnoise:thread_noise
Displays noise and the noise duration from a thread.

12.5. The rtla-osnoise tracer options

The osnoise/options file includes a set of on and off configuration options for the rtla-osnoise tracer.

Options for rtla-osnoise

DEFAULTS
Resets the options to the default value.
OSNOISE_WORKLOAD
Stops the osnoise workload dispatch.
PANIC_ON_STOP
Sets the panic() call if the tracer stops. This option captures a vmcore dump file.
OSNOISE_PREEMPT_DISABLE
Disables preemption for osnoise workloads, which allows only interrupt requests (IRQs) and hardware-related noise.
OSNOISE_IRQ_DISABLE
Disables interrupt requests (IRQs) for osnoise workloads, which allows only non maskable interrupts (NMIs) and hardware-related noise.

12.6. Measuring operating system noise with the rtla-osnoise-top tracer

The rtla osnoise-top tracer measures and prints a periodic summary from the osnoise tracer along with the information about the occurrence counters of the interference source.

Procedure

  1. Measure the system noise: 

    # rtla osnoise top -P F:1 -c 0-3 -r 900000 -d 1M -q

    The command output displays a periodic summary with information about the real-time priority, the assigned CPUs to run the thread, and the period of the run in microseconds.

12.7. The rtla-osnoise-top tracer options

By using the rtla osnoise top --help command, you can view the help usage on the available options for the rtla-osnoise-top tracer.

Options for rtla-osnoise-top

-a, --auto us
Sets the automatic trace mode. This mode sets some commonly used options while debugging the system. It is equivalent to use -s us -T 1 and -t.
-p, --period us
Sets the osnoise tracer duration period in microseconds.
-r, --runtime us
Sets the osnoise tracer runtime in microseconds.
-s, --stop us
Stops the trace if a single sample is more than the argument in microseconds. With -t, the command saves the trace to the output.
-S, --stop-total us
Stops the trace if the total sample is more than the argument in microseconds. With -T, the command saves a trace to the output.
-T, --threshold us
Specifies the minimum delta between two time reads to be considered noise. The default threshold is 5 us.
-q, --quiet
Prints only a summary at the end of a run.
-c, --cpus cpu-list
Sets the osnoise tracer to run the sample threads on the assigned cpu-list.
-d, --duration time[s|m|h|d]
Sets the duration of a run.
-D, --debug
Prints debug information.
-t, --trace[=file]
Saves the stopped trace to [file|osnoise_trace.txt] file.
-e, --event sys:event
Enables an event in the trace (-t) session. The argument can be a specific event, for example -e sched:sched_switch, or all events of a system group, such as -e sched system group.
--filter <filter>
Filters the previous -e sys:event system event with a filter expression.
--trigger <trigger>
Enables a trace event trigger to the previous -e sys:event system event.
-P, --priority o:prio|r:prio|f:prio|d:runtime:period
Sets the scheduling parameters to the osnoise tracer threads.
-h, --help
Prints the help menu.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.