Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 13. Investigating busy CPUs with perf

When investigating performance issues on a system, you can use the perf tool to identify and monitor the busiest CPUs in order to focus your efforts.

13.1. Displaying CPU events counted with perf stat

You can use perf stat to display which CPU events were counted on by disabling CPU count aggregation. You must count events in the system-wide mode by using the -a flag in order to use this functionality.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  • Count the events with CPU count aggregation disabled: 

    # perf stat -a -A sleep seconds
    Copy to Clipboard

    This command displays the count of a default set of common hardware and software events recorded for a time-period of seconds seconds, as dictated by using the sleep command, over each CPU in ascending order, starting with CPU0. As such, it may be useful to specify an event such as cycles: 

    # perf stat -a -A -e cycles sleep seconds
    Copy to Clipboard

13.2. Displaying CPU samples taken with perf report

The perf record command samples performance data and stores it in a perf.data file. You can read this file by using the perf report command. The perf record command also records the CPU on which each sample was taken. To view this information, configure the perf report command accordingly.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.
  • There is a perf.data file created with perf record in the current directory. If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

  • Display the contents of the perf.data file for further analysis while sorting by CPU: 

    # perf report --sort cpu
    Copy to Clipboard

    • You can sort by CPU and command to display more detailed information about where CPU time is being spent: 

      # perf report --sort cpu,comm
      Copy to Clipboard

      This example will list commands from all monitored CPUs by total overhead in descending order of overhead usage and identify the CPU the command was executed.

13.3. Displaying specific CPUs during profiling with perf top

You can configure perf top to display specific CPUs and their relative usage while profiling your system in real time.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  • Start the perf top interface while sorting by CPU: 

    # perf top --sort cpu
    Copy to Clipboard

    This command lists CPUs and their respective overhead in descending order of overhead usage in real time. You can sort by CPU and command for more detailed information of where CPU time is being spent: 

    # perf top --sort cpu,comm
    Copy to Clipboard

    This command lists commands by total overhead in descending order of overhead usage and identifies the CPU the command was executed on in real time.

13.4. Monitoring specific CPUs by using perf record and perf report

You can use the perf tool to sample performance data from specific CPUs and analyze the results to identify CPU-specific behavior or bottlenecks.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.

Procedure

  1. Record performance data from specific CPUs:

    Use the -C option with perf record to target specific CPUs. The following examples demonstrate how to specify individual CPUs or a range.

    • To sample data from selected CPUs (comma-separated): 

      # perf record -C 0,1 sleep <seconds>
      Copy to Clipboard

      This command samples and records performance data from CPUs 0 and 1 for the specified number of seconds.

    • To sample data from a range of CPUs: 

      # perf record -C 0-2 sleep <seconds>
      Copy to Clipboard

      This command samples and records performance data from CPUs 0, 1, and 2 over the specified duration.

  2. Analyze the recorded performance data:

    Use the perf report command to read and analyze the perf.data file. 

    # perf report
    Copy to Clipboard

    This command displays the contents of the perf.data file. NOTE: If you want to see which CPU each sample was recorded on, see Displaying which CPU samples were taken on with perf report.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat