Chapter 30. Analyzing system performance with eBPF

Procedure

• Install bcc-tools:

  # dnf install bcc-tools

  The BCC tools are installed in the /usr/share/bcc/tools/ directory.

Verification

• Inspect the installed tools:

  # ls -l /usr/share/bcc/tools/

  ...
  -rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
  -rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
  -rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
  -rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
  drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
  -rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
  -rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
  -rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
  ...

  The doc directory in the listing above contains documentation for each tool.

Prerequisites

• You have root permissions on the system.
• You have installed the bcc-tools package.

Procedure

• Use execsnoop to examine the new system processes.

  1. Run the execsnoop program in one command line session:

     # /usr/share/bcc/tools/execsnoop

  2. To create a short-lived process of the ls command, in another command line session, enter:

     $ ls /usr/share/bcc/tools/doc/

  3. The command line session that runs execsnoop shows the output similar to the following:

     PCOMM	PID    PPID   RET ARGS
     ls   	8382   8287     0 /usr/bin/ls --color=auto /usr/share/bcc/tools/doc/
     ...

     The execsnoop program prints a line of output for each new process that consumes system resources. It even detects processes of programs that run very shortly, such as ls, and most monitoring tools would not register them.

     The execsnoop output displays the following fields:

     PCOMM

     The process name. (ls)

     PID

     The process ID. (8382)

     PPID

     The parent process ID. (8287)

     RET

     The return value of the exec() system call (0), which loads program code into new processes.

     ARGS

     The location of the started program with arguments.

     To see more details, examples, and options for execsnoop, see /usr/share/bcc/tools/doc/execsnoop_example.txt file. For more information about exec(), see exec(3) manual pages.

• Use opensnoop to track what files a command opens.

  1. In one command line session, run the opensnoop program to print the output for files opened only by the process of the uname command:

     # /usr/share/bcc/tools/opensnoop -n uname

  2. In another command line session, enter the command to open certain files:

     $ uname

  3. The command line session that runs opensnoop shows the output similar to the following:

     PID    COMM 	FD ERR PATH
     8596   uname 	3  0   /etc/ld.so.cache
     8596   uname 	3  0   /lib64/libc.so.6
     8596   uname 	3  0   /usr/lib/locale/locale-archive
     ...

     The opensnoop program watches the open() system call across the whole system, and prints a line of output for each file that uname tried to open along the way.

     The opensnoop output displays the following fields:

     PID

     The process ID. (8596)

     COMM

     The process name. (uname)

     FD

     The file descriptor - a value that open() returns to refer to the open file. (3)

     ERR

     Any errors.

     PATH

     The location of files that open() tried to open.

     If a command tries to read a non-existent file, then the FD column returns -1 and the ERR column prints a value corresponding to the relevant error. As a result, opensnoop can help you identify an application that does not behave properly.

     To see more details, examples, and options for opensnoop, see /usr/share/bcc/tools/doc/opensnoop_example.txt file. For more information about open(), see open(2) manual pages.

• Use the biotop to monitor the top processes performing I/O operations on the disk.

  1. Run the biotop program in one command line session with argument 30 to produce a 30-second summary:

     # /usr/share/bcc/tools/biotop 30

     Note

     When no argument provided, the output screen by default refreshes every 1 second.

  2. In another command line session, enter the command to read the content from the local hard disk device and write the output to the /dev/zero file:

     # dd if=/dev/vda of=/dev/zero

     This step generates certain I/O traffic to illustrate biotop.

  3. The command line session that runs biotop shows the output similar to the following:

     PID    COMM             D MAJ MIN DISK       I/O  Kbytes     AVGms
     9568   dd               R 252 0   vda      16294 14440636.0  3.69
     48     kswapd0          W 252 0   vda       1763 120696.0    1.65
     7571   gnome-shell      R 252 0   vda        834 83612.0     0.33
     1891   gnome-shell      R 252 0   vda       1379 19792.0     0.15
     7515   Xorg             R 252 0   vda        280  9940.0     0.28
     7579   llvmpipe-1       R 252 0   vda        228  6928.0     0.19
     9515   gnome-control-c  R 252 0   vda         62  6444.0     0.43
     8112   gnome-terminal-  R 252 0   vda         67  2572.0     1.54
     7807   gnome-software   R 252 0   vda         31  2336.0     0.73
     9578   awk              R 252 0   vda         17  2228.0     0.66
     7578   llvmpipe-0       R 252 0   vda        156  2204.0     0.07
     9581   pgrep            R 252 0   vda         58  1748.0     0.42
     7531   InputThread      R 252 0   vda         30  1200.0     0.48
     7504   gdbus            R 252 0   vda          3  1164.0     0.30
     1983   llvmpipe-1       R 252 0   vda         39   724.0     0.08
     1982   llvmpipe-0       R 252 0   vda         36   652.0     0.06
     ...

     The biotop output displays the following fields:

     PID

     The process ID. (9568)

     COMM

     The process name. (dd)

     DISK

     The disk performing the read operations. (vda)

     I/O

     The number of read operations performed. (16294)

     Kbytes

     The amount of Kbytes reached by the read operations. (14,440,636)

     AVGms

     The average I/O time of read operations. (3.69)

     For more details, examples, and options for biotop, see the /usr/share/bcc/tools/doc/biotop_example.txt file. For more information about dd, see dd(1) manual pages.

• Use xfsslower to expose unexpectedly slow file system operations.

  The xfsslower measures the time spent by XFS file system in performing read, write, open or sync (fsync) operations. The 1 argument ensures that the program shows only the operations that are slower than 1 ms.

  1. Run the xfsslower program in one command line session:

     # /usr/share/bcc/tools/xfsslower 1

     Note

     When no arguments provided, xfsslower by default displays operations slower than 10 ms.

  2. In another command line session, enter the command to create a text file in the vim editor to start interaction with the XFS file system:

     $ vim text

  3. The command line session that runs xfsslower shows something similar upon saving the file from the previous step:

     TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
     13:07:14 b'bash'        4754   R 256     0           7.11 b'vim'
     13:07:14 b'vim'         4754   R 832     0           4.03 b'libgpm.so.2.1.0'
     13:07:14 b'vim'         4754   R 32      20          1.04 b'libgpm.so.2.1.0'
     13:07:14 b'vim'         4754   R 1982    0           2.30 b'vimrc'
     13:07:14 b'vim'         4754   R 1393    0           2.52 b'getscriptPlugin.vim'
     13:07:45 b'vim'         4754   S 0       0           6.71 b'text'
     13:07:45 b'pool'        2588   R 16      0           5.58 b'text'
     ...

     Each line represents an operation in the file system, which took more time than a certain threshold. xfsslower detects possible file system problems, which can take form of unexpectedly slow operations.

     The xfsslower output displays the following fields:

     COMM

     The process name. (b’bash')

     T

     The operation type. (R)

     • Read
     • Write
     • Sync

     OFF_KB

     The file offset in KB. (0)

     FILENAME

     The file that is read, written, or synced.

     To see more details, examples, and options for xfsslower, see /usr/share/bcc/tools/doc/xfsslower_example.txt file. For more information about fsync, see fsync(2) manual pages.