红帽全球峰会此内容没有您所选择的语言版本。
Chapter 15. Measuring scheduling latency using rtla-osnoise in RHEL for Real Time
An ultra-low latency is an environment that is optimized to process high volumes of data packets with low tolerance for delay. Providing exclusive resources to applications, including the CPU, is a prevalent practice in ultra-low-latency environments. For example, for high performance network processing in network functions virtualization (NFV) applications, a single application has the CPU power limit set to run tasks continuously.
The Linux kernel includes the real-time analysis (rtla) tool, which provides an interface for the operating system noise (osnoise) tracer. The operating system noise is the interference that occurs in an application as a result of activities inside the operating system. Linux systems can experience noise due to:
- Non maskable interrupts (NMIs)
- Interrupt requests (IRQs)
- Soft interrupt requests (SoftIRQs)
- Other system threads activity
- Hardware-related jobs, such as non maskable high priority system management interrupts (SMIs)
15.1. The rtla-osnoise tracer
The Linux kernel includes the real-time analysis (rtla) tool, which provides an interface for the operating system noise (osnoise) tracer. The rtla-osnoise tracer creates a thread that runs periodically for a specified given period. At the start of a period, the thread disables interrupts, starts sampling, and captures the time in a loop.
The rtla-osnoise tracer provides the following capabilities:
- Measure how much operating noise a CPU receives.
- Characterize the type of operating system noise occurring in the CPU.
- Print optimized trace reports that help to define the root cause of unexpected results.
- Saves an interference counter for each interference source. The interference counter for non maskable interrupts (NMIs), interrupt requests (IRQs), software interrupt requests (SoftIRQs), and threads increase when the tool detects the entry events for these interferences.
The rtla-osnoise tracer prints a run report with the following information about the noise sources at the conclusion of the period:
- Total amount of noise.
- The maximum amount of noise.
- The percentage of CPU that is allocated to the thread.
- The counters for the noise sources.
You can configure the rtla-osnoise tracer by adding osnoise in the curret_tracer file of the tracing system. The current_tracer file is generally mounted in the /sys/kernel/tracing/ directory. The rtla-osnoise tracer measures the interrupt requests (IRQs) and saves the trace output for analysis when a thread latency is more than 20 microseconds for a single noise occurrence.
Procedure
List the current tracer:
# cat /sys/kernel/tracing/current_tracer nopThe
no operations(nop) is the default tracer.Add the
timerlattracer in thecurrent_tracerfile of the tracing system:# cd /sys/kernel/tracing/ # echo osnoise > current_tracerGenerate the tracing output:
# cat trace # tracer: osnoise
15.3. The rtla-osnoise options for configuration
The configuration options for the rtla-osnoise tracer is available in the /sys/kernel/tracing/ directory.
Configuration options for rtla-osnoise
- osnoise/cpus
-
Configures the CPUs for the
osnoisethread to run on. - osnoise/period_us
-
Configures the
periodfor aosnoisethread run. - osnoise/runtime_us
-
Configures the run duration for a
osnoisethread. - osnoise/stop_tracing_us
-
Stops the system tracing if a single noise is more than the configured value. Setting
0disables this option. - osnoise/stop_tracing_total_us
-
Stops the system tracing if the total noise is more than the configured value. Setting
0disables this option. - tracing_thresh
-
Sets the minimum delta between two
time()call reads to be considered as noise, in microseconds. When set to0,tracing_threshuses the default value, which is 5 microseconds.
15.4. The rtla-osnoise tracepoints
The rtla-osnoise includes a set of tracepoints to identify the source of the operating system noise (osnoise).
Trace points for rtla-osnoise
- osnoise:sample_threshold
-
Displays a noise when the noise is more than the configured threshold (
tolerance_ns). - osnoise:nmi_noise
- Displays noise and the noise duration from non maskable interrupts (NMIs).
- osnoise:irq_noise
- Displays noise and the noise duration from interrupt requests (IRQs).
- osnoise:softirq_noise
- Displays noise and the noise duration from soft interrupt requests (SoftIRQs),
- osnoise:thread_noise
- Displays noise and the noise duration from a thread.
15.5. The rtla-osnoise tracer options
The osnoise/options file includes a set of on and off configuration options for the rtla-osnoise tracer.
Options for rtla-osnoise
- DEFAULTS
- Resets the options to the default value.
- OSNOISE_WORKLOAD
-
Stops the
osnoiseworkload dispatch. - PANIC_ON_STOP
-
Sets the
panic()call if the tracer stops. This option captures avmcoredump file. - OSNOISE_PREEMPT_DISABLE
-
Disables preemption for
osnoiseworkloads, which allows only interrupt requests (IRQs) and hardware-related noise. - OSNOISE_IRQ_DISABLE
-
Disables interrupt requests (IRQs) for
osnoiseworkloads, which allows only non maskable interrupts (NMIs) and hardware-related noise.
The rtla osnoise-top tracer measures and prints a periodic summary from the osnoise tracer along with the information about the occurrence counters of the interference source.
Procedure
Measure the system noise:
# rtla osnoise top -P F:1 -c 0-3 -r 900000 -d 1M -qThe command output displays a periodic summary with information about the real-time priority, the assigned CPUs to run the thread, and the period of the run in microseconds.
15.7. The rtla-osnoise-top tracer options
By using the rtla osnoise top --help command, you can view the help usage on the available options for the rtla-osnoise-top tracer.
Options for rtla-osnoise-top
- -a, --auto us
-
Sets the automatic trace mode. This mode sets some commonly used options while debugging the system. It is equivalent to use
-sus-T1and-t. - -p, --period us
-
Sets the
osnoisetracer duration period in microseconds. - -r, --runtime us
-
Sets the
osnoisetracer runtime in microseconds. - -s, --stop us
-
Stops the trace if a single sample is more than the argument in microseconds. With
-t, the command saves the trace to the output. - -S, --stop-total us
-
Stops the trace if the total sample is more than the argument in microseconds. With
-T, the command saves a trace to the output. - -T, --threshold us
- Specifies the minimum delta between two time reads to be considered noise. The default threshold is 5 us.
- -q, --quiet
- Prints only a summary at the end of a run.
- -c, --cpus cpu-list
-
Sets the
osnoisetracer to run the sample threads on the assignedcpu-list. - -d, --duration time[s|m|h|d]
- Sets the duration of a run.
- -D, --debug
- Prints debug information.
- -t, --trace[=file]
-
Saves the stopped trace to
[file|osnoise_trace.txt]file. - -e, --event sys:event
-
Enables an event in the trace (
-t) session. The argument can be a specific event, for example-e sched:sched_switch, or all events of a system group, such as-e schedsystem group. - --filter <filter>
-
Filters the previous
-e sys:eventsystem event with a filter expression. - --trigger <trigger>
-
Enables a trace event trigger to the previous
-e sys:eventsystem event. - -P, --priority o:prio|r:prio|f:prio|d:runtime:period
-
Sets the scheduling parameters to the
osnoisetracer threads. - -h, --help
- Prints the help menu.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}