Chapter 22. Analyzing system performance with BPF Compiler Collection

Procedure

1. Run the execsnoop program in one command line:

   # /usr/share/bcc/tools/execsnoop

2. To create a short-lived process of the ls command, in another command line, enter:

   $ ls /usr/share/bcc/tools/doc/

   The command line running execsnoop shows the following output:

   PCOMM	PID    PPID   RET ARGS
   ls   	8382   8287     0 /usr/bin/ls --color=auto /usr/share/bcc/tools/doc/

   The execsnoop program prints a line of output for each new process that consumes system resources. It even detects processes of programs that run very shortly, such as ls, and most monitoring tools would not register them. The execsnoop output displays the following fields:

   PCOMM

   The parent process name. (ls)

   PID

   The process ID. (8382)

   PPID

   The parent process ID. (8287)

   RET

   The return value of the exec() system call (0), which loads program code into new processes.

   ARGS

   The location of the started program with arguments.

   For more information, see the /usr/share/bcc/tools/doc/execsnoop_example.txt file and the exec(3) man page on your system.

Procedure

1. In one command line, run the opensnoop program to print the output for files opened only by the process of the uname command:

   # /usr/share/bcc/tools/opensnoop -n uname

2. In another command line, enter the command to open certain files:

   $ uname

   The command line running opensnoop shows the output similar to the following:
   PID    COMM 	FD ERR PATH
   8596   uname 	3  0   /etc/ld.so.cache
   8596   uname 	3  0   /lib64/libc.so.6
   8596   uname 	3  0   /usr/lib/locale/locale-archive
   ...

   The opensnoop program watches the open() system call across the whole system, and prints a line of output for each file that uname tries to open along the way. The opensnoop output displays the following fields:

   PID

   The process ID. (8596)

   COMM

   The process name. (uname)

   FD

   The file descriptor - a value that open() returns to refer to the open file. (3)

   ERR

   Any errors.

   PATH

   The location of files that open() tries to open.

   If a command tries to read a non-existent file, the FD column returns -1. Also, the ERR column prints a value corresponding to the relevant error. Use opensnoop to identify an application that does not behave properly. For more information, see the /usr/share/bcc/tools/doc/opensnoop_example.txt file and open(2) man page on your system.

Procedure

1. Run the biotop program in one command line with 30 as an argument to produce 30 second summary:

   # /usr/share/bcc/tools/biotop 30

   When you do not provide any argument, the output screen refreshes every 1 second by default.

2. In another command line, enter command to read the content from the local hard disk device and write the output to the /dev/zero file:

   # dd if=/dev/vda of=/dev/zero

   This step generates certain I/O traffic to illustrate biotop. The command line running biotop shows an output similar to the following:

   PID    COMM             D MAJ MIN DISK       I/O  Kbytes     AVGms
   9568   dd               R 252 0   vda      16294 14440636.0  3.69
   48     kswapd0          W 252 0   vda       1763 120696.0    1.65
   7571   gnome-shell      R 252 0   vda        834 83612.0     0.33
   1891   gnome-shell      R 252 0   vda       1379 19792.0     0.15
   7515   Xorg             R 252 0   vda        280  9940.0     0.28
   7579   llvmpipe-1       R 252 0   vda        228  6928.0     0.19
   9515   gnome-control-c  R 252 0   vda         62  6444.0     0.43
   8112   gnome-terminal-  R 252 0   vda         67  2572.0     1.54
   7807   gnome-software   R 252 0   vda         31  2336.0     0.73
   9578   awk              R 252 0   vda         17  2228.0     0.66
   7578   llvmpipe-0       R 252 0   vda        156  2204.0     0.07
   9581   pgrep            R 252 0   vda         58  1748.0     0.42
   7531   InputThread      R 252 0   vda         30  1200.0     0.48
   7504   gdbus            R 252 0   vda          3  1164.0     0.30
   1983   llvmpipe-1       R 252 0   vda         39   724.0     0.08
   1982   llvmpipe-0       R 252 0   vda         36   652.0     0.06
   ...

   The biotop output displays the following fields:

   PID

   The process ID. (9568)

   COMM

   The process name. (dd)

   DISK

   The disk performs the read operations. (vda)

   I/O

   The number of read operations performed. (16294)

   Kbytes

   The amount of Kbytes reached by the read operations. (14,440,636)

   AVGms

   The average I/O time of read operations. (3.69)

   For more information, see the /usr/share/bcc/tools/doc/biotop_example.txt file and the dd(1) man page on your system.

Procedure

1. Run the xfsslower program in one command line:

   # /usr/share/bcc/tools/xfsslower 1

   When you do not provide any arguments, xfsslower displays operations slower than 10 ms by default.

2. Use vim in another command line to create a text file to start interaction with the XFS file system:

   $ vim text

   The command line running xfsslower shows something similar upon saving the file from the previous step:
   TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
   13:07:14 b'bash'        4754   R 256     0           7.11 b'vim'
   13:07:14 b'vim'         4754   R 832     0           4.03 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 32      20          1.04 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 1982    0           2.30 b'vimrc'
   13:07:14 b'vim'         4754   R 1393    0           2.52 b'getscriptPlugin.vim'
   13:07:45 b'vim'         4754   S 0       0           6.71 b'text'
   13:07:45 b'pool'        2588   R 16      0           5.58 b'text’
   ...

   Each line represents an operation in the file system, which took more time than a certain threshold. xfsslower detects possible file system problems, which can take the form of unexpectedly slow operations. The xfsslower output displays the following fields:

   COMM

   The process name. (b’bash')

   T

   The operation type. (R)

   • Read
   • Write
   • Open
   • Sync

   OFF_KB

   The file offset in KB. (0)

   FILENAME

   The file that is read, written, or synced.

   For more information, see the /usr/share/bcc/tools/doc/xfsslower_example.txt file and the fsync(2) man page on your system.