Chapter 30. Tracing latencies using ftrace

Procedure

1. View the available tracers on the system.

   # cat /sys/kernel/debug/tracing/available_tracers
   function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop

   Copy to Clipboard Toggle word wrap

   The user interface for ftrace is a series of files within debugfs.

   The ftrace files are also located in the /sys/kernel/debug/tracing/ directory.

2. Move to the /sys/kernel/debug/tracing/ directory.

   # cd /sys/kernel/debug/tracing

   Copy to Clipboard Toggle word wrap

   The files in this directory can only be modified by the root user, because enabling tracing can have an impact on the performance of the system.

3. To start a tracing session:

   1. Select a tracer you want to use from the list of available tracers in /sys/kernel/debug/tracing/available_tracers.

   2. Insert the name of the selector into the /sys/kernel/debug/tracing/current_tracer.

      # echo preemptoff > /sys/kernel/debug/tracing/current_tracer

      Copy to Clipboard Toggle word wrap
      Note

      If you use a single '>' with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use '>>' instead.

4. The function-trace option is useful because tracing latencies with wakeup_rt, preemptirqsoff, and so on automatically enables function tracing, which may exaggerate the overhead.

   Check if function and function_graph tracing are enabled:

   # cat /sys/kernel/debug/tracing/options/function-trace
   1

   Copy to Clipboard Toggle word wrap
   • A value of 1 indicates that function and function_graph tracing are enabled.
   • A value of 0 indicates that function and function_graph tracing are disabled.

5. By default, function and function_graph tracing are enabled. To turn function and function_graph tracing on or off, echo the appropriate value to the /sys/kernel/debug/tracing/options/function-trace file.

   # echo 0 > /sys/kernel/debug/tracing/options/function-trace
   # echo 1 > /sys/kernel/debug/tracing/options/function-trace

   Copy to Clipboard Toggle word wrap
   Important

   When using the echo command, ensure you place a space character in between the value and the > character. At the shell prompt, using 0>, 1>, and 2> (without a space character) refers to standard input, standard output, and standard error. Using them by mistake could result in an unexpected trace output.

6. Adjust the details and parameters of the tracers by changing the values for the various files in the /debugfs/tracing/ directory.

   For example:

   The irqsoff, preemptoff, preempirqsoff, and wakeup tracers continuously monitor latencies. When they record a latency greater than the one recorded in tracing_max_latency the trace of that latency is recorded, and tracing_max_latency is updated to the new maximum time. In this way, tracing_max_latency always shows the highest recorded latency since it was last reset.

   • To reset the maximum latency, echo 0 into the tracing_max_latency file:

     # echo 0 > /sys/kernel/debug/tracing/tracing_max_latency

     Copy to Clipboard Toggle word wrap

   • To see only latencies greater than a set amount, echo the amount in microseconds:

     # echo 200 > /sys/kernel/debug/tracing/tracing_max_latency

     Copy to Clipboard Toggle word wrap

     When the tracing threshold is set, it overrides the maximum latency setting. When a latency is recorded that is greater than the threshold, it will be recorded regardless of the maximum latency. When reviewing the trace file, only the last recorded latency is shown.

   • To set the threshold, echo the number of microseconds above which latencies must be recorded:

     # echo 200 > /sys/kernel/debug/tracing/tracing_thresh

     Copy to Clipboard Toggle word wrap

7. View the trace logs:

   # cat /sys/kernel/debug/tracing/trace

   Copy to Clipboard Toggle word wrap

8. To store the trace logs, copy them to another file:

   # cat /sys/kernel/debug/tracing/trace > /tmp/lat_trace_log

   Copy to Clipboard Toggle word wrap

9. View the functions being traced:

   # cat /sys/kernel/debug/tracing/set_ftrace_filter

   Copy to Clipboard Toggle word wrap
10. Filter the functions being traced by editing the settings in /sys/kernel/debug/tracing/set_ftrace_filter. If no filters are specified in the file, all functions are traced.

11. To change filter settings, echo the name of the function to be traced. The filter allows the use of a '*' wildcard at the beginning or end of a search term.

    For examples, see ftrace examples.