Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 26. Getting started with SystemTap

As a system administrator, you can use SystemTap to identify underlying causes of a bug or performance problem on a running RHEL system. As an application developer, you can use SystemTap to closely monitor and analyze your application’s behavior within the RHEL environment.

26.1. The purpose of SystemTap

SystemTap is a tracing and probing tool that you can use to study and monitor the activities of your operating system (particularly, the kernel) in fine detail. SystemTap provides information similar to the output of tools such as netstat, ps, top, and iostat. However, SystemTap provides more filtering and analysis options for collected information. In SystemTap scripts, you specify the information that SystemTap gathers.

SystemTap aims to supplement the existing suite of Linux monitoring tools by providing users with an infrastructure to track kernel activity and combining this capability with the following two attributes:

Flexibility
With the SystemTap framework, you can develop simple scripts for investigating and monitoring a wide variety of kernel functions, system calls, and other events that occur in kernel space. With this, SystemTap is not so much a tool as it is a system that you can use to develop your own kernel-specific forensic and monitoring tools.
Ease-of-Use
With SystemTap, you can monitor kernel activity without having to recompile the kernel or reboot the system.

26.2. Installing SystemTap

To begin using SystemTap, install the required packages. To use SystemTap on more than one kernel on a system with multiple kernels, install the corresponding kernel packages for each kernel version.

Prerequisites

Procedure

  1. Install the required SystemTap packages: 

    # dnf install systemtap
    Copy to Clipboard

  2. Install the required kernel packages:

    1. Using stap-prep: 

      # stap-prep
      Copy to Clipboard

    2. If stap-prep does not work, install the required kernel packages manually: 

      # dnf install kernel-debuginfo-$(uname -r) kernel-debuginfo-common-$(uname -m)-$(uname -r) kernel-devel-$(uname -r)
      Copy to Clipboard

      $(uname -m) is automatically replaced with the hardware platform of your system and $(uname -r) is automatically replaced with the version of your running kernel.

Verification

  • If the kernel to be probed with SystemTap is currently in use, test if your installation was successful: 

    # stap -v -e 'probe kernel.function("vfs_read") {printf("read performed\n"); exit()}'
    Copy to Clipboard

    For a successful SystemTap deployment, you see an output similar to the following: 

    Pass 1: parsed user script and 45 library script(s) in 340usr/0sys/358real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 290usr/260sys/568real ms.
Pass 3: translated to C into "/tmp/stapiArgLX/stap_e5886fa50499994e6a87aacdc43cd392_399.c" in 490usr/430sys/938real ms.
Pass 4: compiled C into "stap_e5886fa50499994e6a87aacdc43cd392_399.ko" in 3310usr/430sys/3714real ms.
Pass 5: starting run.
    1 
    
read performed
    2 
    
Pass 5: run completed in 10usr/40sys/73real ms.
    3
    Copy to Clipboard

    The last three lines of output (beginning with Pass 5) indicate the following:

    1
    SystemTap successfully created the instrumentation to probe the kernel and ran the instrumentation.
    2
    SystemTap detected the specified event (in this case, a VFS read).
    3
    SystemTap executed a valid handler (it printed text and then closed it with no errors).

26.3. Privileges to run SystemTap

Running SystemTap scripts requires elevated system privileges, but in some instances non-privileged users might need to run SystemTap instrumentation on their machine.

To allow users to build and run the SystemTap scripts without root access, add users to both of these user groups:

stapdev
Members of this group can use stap to run SystemTap scripts, or staprun to run SystemTap instrumentation modules. Running stap involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This requires elevated privileges to the system, which are granted to stapdev members. These privileges also grant effective root access to stapdev members. Grant stapdev group membership only to users who can be trusted with root access.
stapusr
Members of this group can only use staprun to run SystemTap instrumentation modules. In addition, these users can run those modules only from the /lib/modules/<kernel_version>/systemtap/ directory. This directory must be owned and writable only by the root user.

26.4. Running SystemTap scripts

You can run SystemTap scripts from standard input or from a file. Find sample scripts that are distributed with the installation of SystemTap in Sample SystemTap scripts or in the /usr/share/systemtap/examples directory.

Prerequisites

  • SystemTap and the required kernel packages are installed as described in Installing Systemtap.

  • To run SystemTap scripts as a normal user, add the user to the SystemTap groups: 

    # usermod --append --groups
stapdev,stapusr <user_name>
    Copy to Clipboard

Procedure

  • Run the SystemTap script:

    • From standard input: 

      # stap -e "probe timer.s(1) {exit()}"
      Copy to Clipboard

    • From a file: 

      # stap <file_name>.stp
      Copy to Clipboard

26.5. Sample SystemTap scripts

You can find sample scripts that are distributed with the installation of SystemTap in the /usr/share/systemtap/examples directory. Use the stap command to execute different SystemTap scripts:

Tracing function calls

You can use the para-callgraph.stp SystemTap script to trace function calls and function returns. 

# stap para-callgraph.stp <argument1 argument2>
Copy to Clipboard

The script takes two command-line arguments:

  • The name of the function(s) whose entry/exit you are tracing.

  • An optional trigger function, which enables or disables tracing on a per-thread basis.

    Tracing in each thread will continue as long as the trigger function has not exited yet.

Monitoring polling applications

You can use the timeout.stp SystemTap script to identify and monitor which applications are polling. Knowing this, you can track unnecessary or excessive polling, which helps you to pinpoint areas for improvement in terms of CPU usage and power savings. 

# stap timeout.stp
Copy to Clipboard

This script tracks how many times each application uses poll, select, epoll, itimer, futex, nanosleep, and Signal system calls over time.

Tracking system call volume per process

You can use the syscalls_by_proc.stp SystemTap script to see what processes are performing the highest volume of system calls. It displays the top 20 processes. 

# stap syscalls_by_proc.stp
Copy to Clipboard
Tracing functions called in network socket code

You can use the socket-trace.stp SystemTap script to trace functions called from the kernel’s net/socket.c file. This helps you to identify how each process interacts with the network at the kernel level in fine detail. 

# stap socket-trace.stp
Copy to Clipboard
Tracking I/O time for each file read or write

You can use the iotime.stp SystemTap script to monitor the amount of time it takes for each process to read from or write to any file. This helps you to determine what files are slow to load on a system. 

# stap iotime.stp
Copy to Clipboard
Tracking IRQ’s and other processes stealing cycles from a task

You can use the cycle_thief.stp SystemTap script to track the amount of time a task is running and the amount of time it is not running. This helps you to identify which processes are stealing cycles from a task. 

# stap cycle_thief.stp -x pid
Copy to Clipboard
Note

You can find more examples and information about SystemTap scripts in the /usr/share/systemtap/examples/index.html file. Open it in a web browser to see a list of all the available scripts and their descriptions.

26.6. SystemTap cross-instrumentation

Cross-instrumentation of SystemTap is creating SystemTap instrumentation modules from a SystemTap script on one system to be used on another system that does not have SystemTap fully deployed. When you run a SystemTap script, a kernel module is built out of that script. SystemTap then loads the module into the kernel.

Normally, SystemTap scripts can run only on systems where SystemTap is deployed. To run SystemTap on ten systems, SystemTap needs to be deployed on all those systems. In some cases, this might be neither feasible nor desired. For example, corporate policy might prohibit you from installing packages that provide compilers or debug information about specific machines, which will prevent the deployment of SystemTap.

To work around this, use cross-instrumentation. Cross-instrumentation is the process of generating SystemTap instrumentation modules from a SystemTap script on one system to be used on another system. This process offers the following benefits:

  • The kernel information packages for various machines can be installed on a single host machine. Kernel packaging bugs may prevent the installation. In such cases, the kernel-debuginfo and kernel-devel packages for the host system and target system must match. If a bug occurs, report the bug at Red Hat JIRA.
  • Each target machine needs only one package to be installed to use the generated SystemTap instrumentation module: systemtap-runtime. The host system must be the same architecture and running the same distribution of Linux as the target system in order for the built instrumentation module to work.
Terminology

  • instrumentation module

    The kernel module is built from a SystemTap script; the SystemTap module is built on the host system, and will be loaded on the target kernel of the target system.

  • host system

    The system on which the instrumentation modules (from SystemTap scripts) are compiled, to be loaded on target systems.

  • target system

    The system in which the instrumentation module is being built (from SystemTap scripts).

  • target kernel

    The kernel of the target system. This is the kernel that loads and runs the instrumentation module.

26.7. Initializing cross-instrumentation of SystemTap

Initialize cross-instrumentation of SystemTap to build SystemTap instrumentation modules from a SystemTap script on one system and use these modules on another system that does not have SystemTap fully deployed.

Prerequisites

  • SystemTap is installed on the host system as described in Installing Systemtap.

  • The systemtap-runtime package is installed on each target system: 

    # dnf install systemtap-runtime
    Copy to Clipboard
  • Both the host and the target systems are the same architecture.

  • Both the host and the target systems are running the same major version of Red Hat Enterprise Linux (such as Red Hat Enterprise Linux 10).

    Important

    Kernel packaging bugs might prevent multiple kernel-debuginfo and kernel-devel packages from being installed on one system. In such cases, the minor version for the host and the target systems must match. If a bug occurs, report the bug at Red Hat JIRA.

Procedure

  1. Determine the kernel running on each target system: 

    $ uname -r
    Copy to Clipboard
  2. Repeat this step for each target system.
  3. On the host system, install the target kernel and related packages for each target system by the method described in Installing Systemtap.

  4. Build an instrumentation module on the host system, copy this module to and run this module on on the target system either:

    1. If an SSH connection can be made to the target system from the host system, use remote implementation: 

      # stap --remote <target_system> script
      Copy to Clipboard

      You must ensure an SSH connection can be made to the target system from the host system for this to be successful.

    2. Manually:

      1. Build the instrumentation module on the host system: 

        # stap -r <kernel_version> script -m <module_name> -p 4
        Copy to Clipboard

        Here, <kernel_version> is the version of the target kernel determined in step 1, script is the script to be converted into an instrumentation module, and <module_name> is the desired name of the instrumentation module. The -p4 option tells SystemTap to not load and run the compiled module.

      2. Once the instrumentation module is compiled, copy it to the target system and load it by using the following command: 

        # staprun <module_name>.ko
        Copy to Clipboard
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat