Chapter 22. Investigating busy CPUs with perf
When investigating performance issues on a system, you can use the perf
tool to identify and monitor the busiest CPUs in order to focus your efforts.
22.1. Displaying which CPU events were counted on with perf stat
You can use perf stat
to display which CPU events were counted on by disabling CPU count aggregation. You must count events in system-wide mode by using the -a
flag in order to use this functionality.
Prerequisites
-
You have the
perf
user space tool installed as described in Installing perf.
Procedure
Count the events with CPU count aggregation disabled:
# perf stat -a -A sleep seconds
The previous example displays counts of a default set of common hardware and software events recorded over a time period of
seconds
seconds, as dictated by using thesleep
command, over each individual CPU in ascending order, starting withCPU0
. As such, it may be useful to specify an event such as cycles:# perf stat -a -A -e cycles sleep seconds
22.2. Displaying which CPU samples were taken on with perf report
The perf record
command samples performance data and stores this data in a perf.data
file which can be read with the perf report
command. The perf record
command always records which CPU samples were taken on. You can configure perf report
to display this information.
Prerequisites
-
You have the
perf
user space tool installed as described in Installing perf. -
There is a
perf.data
file created withperf record
in the current directory. If theperf.data
file was created with root access, you need to runperf report
with root access too.
Procedure
Display the contents of the
perf.data
file for further analysis while sorting by CPU:# perf report --sort cpu
You can sort by CPU and command to display more detailed information about where CPU time is being spent:
# perf report --sort cpu,comm
This example will list commands from all monitored CPUs by total overhead in descending order of overhead usage and identify the CPU the command was executed on.
Additional resources
22.3. Displaying specific CPUs during profiling with perf top
You can configure perf top
to display specific CPUs and their relative usage while profiling your system in real time.
Prerequisites
-
You have the
perf
user space tool installed as described in Installing perf.
Procedure
Start the
perf top
interface while sorting by CPU:# perf top --sort cpu
This example will list CPUs and their respective overhead in descending order of overhead usage in real time.
You can sort by CPU and command for more detailed information of where CPU time is being spent:
# perf top --sort cpu,comm
This example will list commands by total overhead in descending order of overhead usage and identify the CPU the command was executed on in real time.
22.4. Monitoring specific CPUs with perf record and perf report
You can configure perf record
to only sample specific CPUs of interest and analyze the generated perf.data
file with perf report
for further analysis.
Prerequisites
-
You have the
perf
user space tool installed as described in Installing perf.
Procedure
Sample and record the performance data in the specific CPU’s, generating a
perf.data
file:Using a comma separated list of CPUs:
# perf record -C 0,1 sleep seconds
The previous example samples and records data in CPUs 0 and 1 for a period of
seconds
seconds as dictated by the use of thesleep
command.Using a range of CPUs:
# perf record -C 0-2 sleep seconds
The previous example samples and records data in all CPUs from CPU 0 to 2 for a period of
seconds
seconds as dictated by the use of thesleep
command.
Display the contents of the
perf.data
file for further analysis:# perf report
This example will display the contents of
perf.data
. If you are monitoring several CPUs and want to know which CPU data was sampled on, see Displaying which CPU samples were taken on with perf report.