Chapter 13. Recording and analyzing performance profiles with perf


The perf tool allows you to record performance data and analyze it at a later time.

13.1. The purpose of perf record

The perf record command samples performance data and stores it in a perf.data file. You can then read and visualize data with other perf commands. perf.data is generated in the current directory and can be accessed at a later time, possibly on a different machine. You can profile the entire system by running perf record -a. This records performance data until you press Ctrl+C or for the duration of a specific command. Similar to that, you can profile a single application/process, either by running perf record ./your_app or by attaching to an already running one by using perf record -p PID.

13.2. Recording a performance profile

You can use perf record to sample and record performance data. The level of data captured depends on the access level:

  • Without root access: perf record collects data only in the user space.
  • With root access: perf record collects data in both the user and kernel space.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.
  • You have root access if you want to collect data for user and kernel space.

Procedure

  • Do one of the following:

    • To sample and record the performance data for the user space:

      $ perf record command
      Copy to Clipboard Toggle word wrap
    • To sample and record the performance data for kernel space (use root or sudo access):

      # perf record command
      Copy to Clipboard Toggle word wrap

      Replace command with the actual command you want to profile. If you do not specify a command, then perf record samples data until you manually stop it by pressing Ctrl+C.

You can use perf record in per-CPU mode to sample and record performance data in both user-space and the kernel-space simultaneously across all threads on a monitored CPU. By default, per-CPU mode monitors all online CPUs.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  • Sample and record the performance data:

    # perf record -a command
    Copy to Clipboard Toggle word wrap

    Replace command with the command you want to sample data during. If you do not specify a command, then perf record samples data until you manually stop it by pressing Ctrl+C.

13.4. Capturing call graph data with perf record

You can configure the perf record tool so that it records which function calls other functions in the performance profile. This helps to identify a bottleneck if several processes are calling the same function.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  • Sample and record performance data with the --call-graph option:

    $ perf record --call-graph method command
    Copy to Clipboard Toggle word wrap
    • Replace command with the command you want to sample data during. If you do not specify a command, then perf record samples data until you manually stop it by pressing Ctrl+C.
    • Replace method with one of the following unwinding methods:

      fp
      Uses the frame pointer method. Depending on compiler optimization, such as with binaries built with the GCC option --fomit-frame-pointer, this may not be able to unwind the stack.
      dwarf
      Uses DWARF Call Frame Information to unwind the stack.
      lbr
      Uses the last branch record hardware on Intel processors.

13.5. Analyzing perf.data with perf report

You can use perf report to display and analyze data from the perf.data file.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.
  • There is a perf.data file in the current directory.
  • If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

  • Display the contents of the perf.data file for further analysis:

    # perf report
    This command displays output similar to the following:
    Samples: 2K of event 'cycles', Event count (approx.): 235462960
    Overhead  Command          Shared Object                     Symbol
       2.36%  kswapd0          [kernel.kallsyms]                 [k] page_vma_mapped_walk
       2.13%  sssd_kcm         libc-2.28.so                      [.] memset_avx2_erms 2.13% perf [kernel.kallsyms] [k] smp_call_function_single 1.53% gnome-shell libc-2.28.so [.] strcmp_avx2
       1.17%  gnome-shell      libglib-2.0.so.0.5600.4           [.] g_hash_table_lookup
       0.93%  Xorg             libc-2.28.so                      [.] memmove_avx_unaligned_erms 0.89% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_object_unref 0.87% kswapd0 [kernel.kallsyms] [k] page_referenced_one 0.86% gnome-shell libc-2.28.so [.] memmove_avx_unaligned_erms
       0.83%  Xorg             [kernel.kallsyms]                 [k] alloc_vmap_area
       0.63%  gnome-shell      libglib-2.0.so.0.5600.4           [.] g_slice_alloc
       0.53%  gnome-shell      libgirepository-1.0.so.1.0.0      [.] g_base_info_unref
       0.53%  gnome-shell      ld-2.28.so                        [.] _dl_find_dso_for_object
       0.49%  kswapd0          [kernel.kallsyms]                 [k] vma_interval_tree_iter_next
    0.48%  gnome-shell      libpthread-2.28.so                [.] pthread_getspecific 0.47% gnome-shell libgirepository-1.0.so.1.0.0 [.] 0x0000000000013b1d 0.45% gnome-shell libglib-2.0.so.0.5600.4 [.] g_slice_free1 0.45% gnome-shell libgobject-2.0.so.0.5600.4 [.] g_type_check_instance_is_fundamentally_a 0.44% gnome-shell libc-2.28.so [.] malloc 0.41% swapper [kernel.kallsyms] [k] apic_timer_interrupt 0.40% gnome-shell ld-2.28.so [.] _dl_lookup_symbol_x 0.39% kswapd0 [kernel.kallsyms] [k] raw_callee_save___pv_queued_spin_unlock
    Copy to Clipboard Toggle word wrap

    For more information, see the perf-record(1) man page on your system.

13.6. Attaching perf record to a running process

You can attach perf record to a running process. This instructs perf record to only sample and record performance data in the specified processes.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  • Attach perf record to a running process:

    $ perf record -p ID1,ID2 sleep seconds
    Copy to Clipboard Toggle word wrap

    This command samples and records performance data of the processes with the process ID’s ID1 and ID2 for a time period of seconds seconds as dictated by using the sleep command. You can also configure perf to record events in specific threads:

    $ perf record -t ID1,ID2 sleep seconds
    Copy to Clipboard Toggle word wrap
    Note

    When using the -t flag and stipulating thread ID’s, perf disables inheritance by default. You can enable inheritance by adding the --inherit option.

13.7. Interpretation of perf report output

The table displayed by running the perf report command sorts the data into the following several columns:

Overhead
Indicates what percentage of overall samples were collected in that particular function.
Command
Tells you which process the samples were collected from.
Shared Object
Displays the name of the ELF image where the samples come from (the name [kernel.kallsyms] is used when the samples come from the kernel).
Symbol
Displays the function name or symbol. In default mode, the functions are sorted in descending order with those with the highest overhead displayed first.

You can use the perf tool to record performance data into a perf.data file to be analyzed on a different device.

Prerequisites

Procedure

  1. Capture performance data you are interested in investigating further:

    # perf record -a --call-graph fp sleep <seconds>
    Copy to Clipboard Toggle word wrap

    This example generates a perf.data over the entire system for a period of seconds seconds as dictated by the use of the sleep command. It would also capture call graph data using the frame pointer method.

  2. Generate an archive file containing debug symbols of the recorded data:

    # perf archive
    Copy to Clipboard Toggle word wrap

Verification

  • Verify that the archive file has been generated in your current active directory:

    # ls perf.data*
    Copy to Clipboard Toggle word wrap

    The output will display every file in your current directory that begins with perf.data. The archive file will be named either: perf.data.tar.gz or perf.data.tar.bz2.

You can use the perf tool to analyze a perf.data file that was generated on a different device.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.
  • A perf.data file and associated archive file generated on a different device are present on the current device being used.

Procedure

  1. Copy both the perf.data file and the archive file into your current active directory.
  2. Extract the archive file into ~/.debug:

    # mkdir -p ~/.debug
    # tar xf perf.data.tar.bz2 -C ~/.debug
    Copy to Clipboard Toggle word wrap
    Note

    The archive file might also be named perf.data.tar.gz.

  3. Open the perf.data file for further analysis:

    # perf report
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat