Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 11. Profiling CPU usage in real time with perf top

You can use the perf top command to measure CPU usage of different functions in real time.

11.1. The purpose of perf top
Copy link

The perf top command is used for real-time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top informs you about functions being used across all CPUs in both the user-space and the kernel-space. To use perf top you need root access.

11.2. Profiling CPU usage with perf top
Copy link

You can use perf top to monitor real-time CPU usage and identify functions consuming the most processing time.

Prerequisites

  • You have the perf user space tool installed. For more information, see Installing perf.
  • You have root access.

Procedure

  • Start the perf top monitoring interface: 

    # perf top
The monitoring interface looks similar to the following:
Samples: 8K of event 'cycles', 2000 Hz, Event count (approx.): 4579432780 lost: 0/0 drop: 0/0
Overhead  Shared Object       Symbol
   2.20%  [kernel]            [k] do_syscall_64
   2.17%  [kernel]            [k] module_get_kallsym
   1.49%  [kernel]            [k] copy_user_enhanced_fast_string
   1.37%  libpthread-2.29.so  [.] pthread_mutex_lock 1.31% [unknown] [.] 0000000000000000 1.07% [kernel] [k] psi_task_change 1.04% [kernel] [k] switch_mm_irqs_off 0.94% [kernel] [k] fget
   0.74%  [kernel]            [k] entry_SYSCALL_64
   0.69%  [kernel]            [k] syscall_return_via_sysret
   0.69%  libxul.so           [.] 0x000000000113f9b0
   0.67%  [kernel]            [k] kallsyms_expand_symbol.constprop.0
   0.65%  firefox             [.] moz_xmalloc
   0.65%  libpthread-2.29.so  [.] __pthread_mutex_unlock_usercnt
   0.60%  firefox             [.] free
   0.60%  libxul.so           [.] 0x000000000241d1cd
   0.60%  [kernel]            [k] do_sys_poll
   0.58%  [kernel]            [k] menu_select
   0.56%  [kernel]            [k] _raw_spin_lock_irqsave
   0.55%  perf                [.] 0x00000000002ae0f3
    Copy to Clipboard Toggle word wrap

    In this example, the kernel function do_syscall_64 is using the most CPU time.

The perf top monitoring interface provides a real-time view of CPU usage and function activity. Understanding its output helps identify performance bottlenecks and optimizes system behavior.

Key columns in perf top output
The interface displays several following columns:
Overhead
Shows the percentage of CPU time consumed by a given function. This helps pinpoint the most resource-intensive operations.
Shared Object
Indicates the name of the program or library where the function resides.
Symbol

Displays the name of the function or symbol.

  • Functions running in kernel space are marked with [k].
  • Functions running in user space are marked with [.].
Causes of unresolved symbols in perf output

For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped.

This information can be included by installing the corresponding debuginfo package or by compiling the application with debugging enabled, such as using the -g option in gcc. Once the necessary debug information is available, perf can accurately map sampled addresses to human-readable function names during reporting.

Note

After making debug information available, it is not necessary to re-run the perf record command. Running the perf report command again will reflect the resolved symbols.

11.4. Enabling debug and source repositories
Copy link

A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance.

Procedure

  • Enable the source and debug information package channels: 

    # subscription-manager repos --enable rhel-10-for-$(uname -m)-baseos-debug-rpms
# subscription-manager repos --enable rhel-10-for-$(uname -m)-baseos-source-rpms
# subscription-manager repos --enable rhel-10-for-$(uname -m)-appstream-debug-rpms
# subscription-manager repos --enable rhel-10-for-$(uname -m)-appstream-source-rpms
    Copy to Clipboard Toggle word wrap

    The $(uname -m) part is automatically replaced with a matching value for architecture of your system:

Expand

Architecture name

Value

64-bit Intel and AMD

x86_64

64-bit ARM

aarch64

IBM POWER

ppc64le

64-bit IBM Z

s390x

Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package.

Prerequisites

  • The application or library you want to debug must be installed on the system.
  • GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications.
  • Repositories providing debuginfo and debugsource packages must be configured and enabled on the system. For details, see Enabling debug and source repositories.

Procedure

  1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. 

    $ gdb -q /bin/ls
Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: dnf debuginfo-install coreutils-9.5-6.el10.x86_64
(gdb)
    Copy to Clipboard Toggle word wrap

  2. To exit GDB, type q and confirm with Enter. 

    (gdb) q
    Copy to Clipboard Toggle word wrap

  3. Run the command suggested by GDB to install the required debuginfo packages: 

    # dnf debuginfo-install coreutils-9.5-6.el10.x86_64
    Copy to Clipboard Toggle word wrap

    The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat