SystemTap Beginners Guide
Introduction to SystemTap
Abstract
Chapter 1. Introduction
netstat
, ps
, top
, and iostat
; however, SystemTap is designed to provide more filtering and analysis options for collected information.
1.1. Documentation Goals
- To introduce users to SystemTap, familiarize them with its architecture, and provide setup instructions for all kernel types.
- To provide pre-written SystemTap scripts for monitoring detailed activity in different components of the system, along with instructions on how to run them and analyze their output.
1.2. SystemTap Capabilities
- Flexibility: SystemTap's framework allows users to develop simple scripts for investigating and monitoring a wide variety of kernel functions, system calls, and other events that occur in kernel space. With this, SystemTap is not so much a tool as it is a system that allows you to develop your own kernel-specific forensic and monitoring tools.
- Ease-of-Use: as mentioned earlier, SystemTap allows users to probe kernel-space events without having to resort to the lengthy instrument, recompile, install, and reboot the kernel process.
top
, OProfile, or ps
). These scripts are provided to give readers extensive examples of the application of SystemTap, which in turn will educate them further on the capabilities they can employ when writing their own SystemTap scripts.
Chapter 2. Using SystemTap
2.1. Installation and Setup
Important
2.1.1. Installing SystemTap
root
:
~]#
yum install -y systemtap systemtap-runtime
2.1.2. Installing Required Kernel Information Packages
uname -m
command).
debug
repository.
debug
repository for your system:
~]#
subscription-manager repos --enable=rhel-7-variant-debug-rpms
server
, workstation
, or client
, depending on the variant of the Red Hat Enterprise Linux system you are using. To determine the variant, you can use the following command:
~]#
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
uname -r
3.10.0-327.el7.x86_64
3.10.0-327.4.4.el7
on an AMD64 or Intel 64 machine, then you need to install the following packages:
- kernel-debuginfo-3.10.0-327.4.4.el7.x86_64.rpm
- kernel-debuginfo-common-x86_64-3.10.0-327.4.4.el7.x86_64.rpm
- kernel-devel-3.10.0-327.4.4.el7.x86_64.rpm
yum
package manager to install the packages required for running SystemTap on the current kernel, execute the following command as root
:
~]#
yum install -y kernel-devel-$(uname -r) \
kernel-debuginfo-$(uname -r) \
kernel-debuginfo-common-$(uname -m)-$(uname -r)
2.1.3. Initial Testing
stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
read performed
and then exit properly once a virtual file system read is detected. If the SystemTap deployment was successful, you should get output similar to the following:
Pass 1: parsed user script and 45 library script(s) in 340usr/0sys/358real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) in 290usr/260sys/568real ms. Pass 3: translated to C into "/tmp/stapiArgLX/stap_e5886fa50499994e6a87aacdc43cd392_399.c" in 490usr/430sys/938real ms. Pass 4: compiled C into "stap_e5886fa50499994e6a87aacdc43cd392_399.ko" in 3310usr/430sys/3714real ms. Pass 5: starting run. read performed Pass 5: run completed in 10usr/40sys/73real ms.
Pass 5
) indicate that SystemTap was able to successfully create the instrumentation to probe the kernel, run the instrumentation, detect the event being probed (in this case, a virtual file system read), and execute a valid handler (print text and then close it with no errors).
2.2. Generating Instrumentation for Other Computers
- The kernel information packages for various machines can be installed on a single host machine.
- Each target machine only needs one package to be installed to use the generated SystemTap instrumentation module: systemtap-runtime.
Important
Note
- instrumentation module
- The kernel module built from a SystemTap script; the SystemTap module is built on the host system, and will be loaded on the target kernel of the target system.
- host system
- The system on which the instrumentation modules (from SystemTap scripts) are compiled, to be loaded on target systems.
- target system
- The system in which the instrumentation module is being built (from SystemTap scripts).
- target kernel
- The kernel of the target system. This is the kernel which loads/runs the instrumentation module.
Procedure 2.1. Configuring a Host System and Target Systems
- Install the
systemtap-runtime
package on each target system. - Determine the kernel running on each target system by running
uname -r
on each target system. - Install SystemTap on the host system. The instrumentation module will be built for the target systems on the host system. For instructions on how to install SystemTap, see Section 2.1.1, “Installing SystemTap”.
- Using the target kernel version determined earlier, install the target kernel and related packages on the host system by the method described in Section 2.1.2, “Installing Required Kernel Information Packages”. If multiple target systems use different target kernels, repeat this step for each different kernel used on the target systems.
stap -r kernel_version script -m module_name -p4
uname -r
on the target machine), script refers to the script to be converted into an instrumentation module, and module_name is the desired name of the instrumentation module.
staprun module_name.ko
simple.ko
instrumentation module from a SystemTap script named simple.stp
for the 3.10.0-327.4.4.el7
target kernel, use the following command:
stap -r 2.6.32-53.el6 -e 'probe vfs.read {exit()}' -m simple -p4
simple.ko
. To use the simple.ko
instrumentation module, copy it to the target system and run the following command (on the target system):
staprun simple.ko
2.3. Running SystemTap Scripts
stap
. stap
can run SystemTap scripts from the standard input or from a file.
stap
and staprun
requires elevated privileges to the system. However, not all users can be granted root
access just to run SystemTap. In some cases, for instance, a non-privileged user may need to run SystemTap instrumentation on their machine.
root
access, add them to both of these user groups:
- stapdev
- Members of this group can use
stap
to run SystemTap scripts, orstaprun
to run SystemTap instrumentation modules.Runningstap
involves compiling SystemTap scripts into kernel modules and loading them into the kernel. This requires elevated privileges to the system, which are granted tostapdev
members. Unfortunately, such privileges also grant effectiveroot
access tostapdev
members. As such, only grantstapdev
group membership to users who can be trusted withroot
access. - stapusr
- Members of this group can only use
staprun
to run SystemTap instrumentation modules. In addition, they can only run those modules from/lib/modules/kernel_version/systemtap/
. Note that this directory must be owned only by theroot
user, and it must only be writable by theroot
user.
Note
stap
options:
- -v
- Makes the output of the SystemTap session more verbose. This option (for example,
stap -vvv script.stp
) can be repeated to provide more details on the script's execution. It is particularly useful if errors are encountered when running the script.For more information about common SystemTap script errors, see Chapter 5, Understanding SystemTap Errors. -o filename
- Sends the standard output to filename.
-S size,count
- Limit files to size megabytes and limit the number of files kept around to count. The file names will have a sequence number suffix. This option implements logrotate operations for SystemTap.When used with
-o
, the-S
will limit the size of log files. -x process ID
- Sets the SystemTap handler function
target()
to the specified process ID. For more information abouttarget()
, see SystemTap Functions. -c command
- Sets the SystemTap handler function
target()
to the specified command. The full path to the specified command must be used; for example, instead of specifyingcp
, use/bin/cp
(as instap script -c /bin/cp
). For more information abouttarget()
, see SystemTap Functions. -e 'script'
- Use
script
string rather than a file as input for systemtap translator. -F
- Use SystemTap's Flight recorder mode and make the script a background process. For more information about flight recorder mode, see Section 2.3.1, “SystemTap Flight Recorder Mode”.
stap
can also be instructed to run scripts from the standard input using the -
switch. To illustrate:
Example 2.1. Running Scripts From Standard Input
echo "probe timer.s(1) {exit()}" | stap -
stap
to run the script passed by echo
to standard input. Any stap
options to be used should be inserted before the -
switch; for instance, to make the example in Example 2.1, “Running Scripts From Standard Input” more verbose, the command would be:
echo "probe timer.s(1) {exit()}" | stap -v -
stap
, see man stap
.
staprun
instead. For more information about staprun
and cross-instrumentation, see Section 2.2, “Generating Instrumentation for Other Computers”.
Note
stap
options -v
and -o
also work for staprun
. For more information about staprun
, see the staprun(1) manual page.
2.3.1. SystemTap Flight Recorder Mode
-F
option) limits the amount of output generated. There are two variations of the flight recorder mode: in-memory and file mode. In both cases, the SystemTap script runs as a background process.
2.3.1.1. In-memory Flight Recorder
-F
option) is used without a file name, SystemTap uses a buffer in kernel memory to store the output of the script. Next, SystemTap instrumentation module loads and the probes start running, then instrumentation will detatch and be put in the background. When the interesting event occurs, the instrumentation can be reattached and the recent output in the memory buffer and any continuing output can be seen. The following command starts a script using the flight recorder in-memory mode:
stap -F /usr/share/doc/systemtap-version/examples/io/iotime.stp
Disconnecting from systemtap module. To reconnect, type "staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556"
staprun -A stap_5dd0073edcb1f13f7565d8c343063e68_19556
-s
option specifying the size in megabytes (rounded up to the next power of 2) for the buffer. For example -s2
on the SystemTap command line would specify 2MB for the buffer.
2.3.1.2. File Flight Recorder
-S
option followed by two numerical arguments separated by a comma. The first argument is the maximum size in megabytes for the each output file. The second argument is the number of recent files to keep. The file name is specified by the -o
option followed by the name. SystemTap adds a number suffix to the file name to indicate the order of the files. The following will start SystemTap in file flight recorder mode with the output going to files named /tmp/pfaults.log.[0-9]+
with each file 1MB or smaller and keeping latest two files:
stap -F -o /tmp/pfaults.log -S 1,2 pfaults.stp
SIGTERM
to the process will shutdown the SystemTap script and stop the data collection. For example, if the previous command listed 7590
as the process ID, the following command would shutdown the SystemTap script:
kill -s SIGTERM 7590
ls -sh /tmp/pfaults.log.*
shows the only two files:
1020K /tmp/pfaults.log.5 44K /tmp/pfaults.log.6
/tmp/pfaults.log.6
.
Chapter 3. Understanding How SystemTap Works
3.1. Architecture
Procedure 3.1. SystemTap Session
- First, SystemTap checks the script against the existing tapset library (normally in the
/usr/share/systemtap/tapset/
directory) for any tapsets used. SystemTap will then substitute any located tapsets with their corresponding definitions in the tapset library. - SystemTap then translates the script to C, running the system C compiler to create a kernel module from it. The tools that perform this step are contained in the systemtap package (see Section 2.1.1, “Installing SystemTap” for more information).
- SystemTap loads the module, then enables all the probes (events and handlers) in the script. The
staprun
in the systemtap-runtime package (see Section 2.1.1, “Installing SystemTap” for more information) provides this functionality. - As the events occur, their corresponding handlers are executed.
- Once the SystemTap session is terminated, the probes are disabled, and the kernel module is unloaded.
stap
. This program is SystemTap's main front-end tool. For more information about stap
, see the stap(1) manual page (once SystemTap is properly installed on your machine).
3.2. SystemTap Scripts
Note
SystemTap scripts use the .stp
file extension and contains probes written in the following format:
probe event {statements}
,
). If multiple events are specified in a single probe, SystemTap executes the handler when any of the specified events occurs.
{ }
) and contains the statements to be executed per event. SystemTap executes these statements in sequence; special separators or terminators are generally not necessary between multiple statements.
Note
function function_name(arguments){statements} probe event {function_name(arguments)}
Important
3.2.1. Event
A synchronous event occurs when any process executes an instruction at a particular location in kernel code. This gives other events a reference point from which more contextual data may be available.
- syscall.system_call
- The entry to the system call system_call. If the exit from a syscall is desired, appending a
.return
to the event monitor the exit of the system call instead. For example, to specify the entry and exit of theclose
system call, usesyscall.close
andsyscall.close.return
respectively. - vfs.file_operation
- The entry to the file_operation event for Virtual File System (VFS). Similar to
syscall
event, appending a.return
to the event monitors the exit of thefile_operation
operation. - kernel.function("function")
- The entry to the
function
kernel function. For example,kernel.function("sys_open")
refers to the event that occurs when thesys_open
kernel function is called by any thread in the system. To specify the return of thesys_open
kernel function, append thereturn
string to the event statement; that is,kernel.function("sys_open").return
.When defining probe events, you can use asterisk (*
) for wildcards. You can also trace the entry or exit of a function in a kernel source file. Consider the following example:Example 3.1. wildcards.stp
probe kernel.function("*@net/socket.c") { } probe kernel.function("*@net/socket.c").return { }
In the previous example, the first probe's event specifies the entry of ALL functions in thenet/socket.c
kernel source file. The second probe specifies the exit of all those functions. Note that in this example, there are no statements in the handler; as such, no information will be collected or displayed. - kernel.trace("tracepoint")
- The static probe for tracepoint. Recent kernels (2.6.30 and newer) include instrumentation for specific events in the kernel. These events are statically marked with tracepoints. One example of a tracepoint available in SystemTap is
kernel.trace("kfree_skb")
, which indicates each time a network buffer is freed in the kernel. - module("module").function("function")
- Allows you to probe functions within modules. For example:
Example 3.2. moduleprobe.stp
probe module("ext3").function("*") { } probe module("ext3").function("*").return { }
The first probe in Example 3.2, “moduleprobe.stp” points to the entry of all functions for theext3
module. The second probe points to the exits of all functions for that same module; the use of the.return
suffix is similar tokernel.function()
. Note that the probes in Example 3.2, “moduleprobe.stp” do not contain statements in the probe handlers, and as such will not print any useful data (as in Example 3.1, “wildcards.stp”).A system's kernel modules are typically located in/lib/modules/kernel_version
, where kernel_version refers to the currently loaded kernel version. Modules use the file name extension.ko
.
Asynchronous events are not tied to a particular instruction or location in code. This family of probe points consists mainly of counters, timers, and similar constructs.
- begin
- The startup of a SystemTap session; that is, as soon as the SystemTap script is run.
- end
- The end of a SystemTap session.
- timer events
- An event that specifies a handler to be executed periodically. For example:
Example 3.3. timer-s.stp
probe timer.s(4) { printf("hello world\n") }
Example 3.3, “timer-s.stp” is an example of a probe that printshello world
every four seconds. Note that you can also use the following timer events:timer.ms(milliseconds)
timer.us(microseconds)
timer.ns(nanoseconds)
timer.hz(hertz)
timer.jiffies(jiffies)
When used in conjunction with other probes that collect information, timer events allows you to print periodic updates and see how that information changes over time.
Important
3.2.2. Systemtap Handler/Body
Example 3.4. helloworld.stp
probe begin { printf ("hello world\n") exit () }
begin
event (the start of the session) triggers the handler enclosed in { }
, which simply prints hello world
followed by a new line, then exits.
Note
exit()
function executes. If the users wants to stop the execution of the script, it can interrupted manually with Ctrl+C.
The printf()
statement is one of the simplest functions for printing data. printf()
can also be used to display data using many SystemTap functions in the following format:
printf ("format string\n", arguments)
hello world
and contains no format specifiers.
%s
(for strings) and %d
(for numbers) in format strings, depending on your list of arguments. Format strings can have multiple format specifiers, each matching a corresponding argument; multiple arguments are delimited by a comma (,
).
Note
printf
function is very similar to its C language counterpart. The aforementioned syntax and format for SystemTap's printf
function is identical to that of the C-style printf
.
Example 3.5. variables-in-printf-statements.stp
probe syscall.open { printf ("%s(%d) open\n", execname(), pid()) }
open
; for each event, it prints the current execname()
(a string with the executable name) and pid()
(the current process ID number), followed by the word open
. A snippet of this probe's output would look like:
vmware-guestd(2206) open hald(2360) open hald(2360) open hald(2360) open df(3433) open df(3433) open df(3433) open hald(2360) open
SystemTap supports many functions that can be used as printf()
arguments. Example 3.5, “variables-in-printf-statements.stp” uses the SystemTap functions execname()
(name of the process that called a kernel function/performed a system call) and pid()
(current process ID).
- tid()
- The ID of the current thread.
- uid()
- The ID of the current user.
- cpu()
- The current CPU number.
- gettimeofday_s()
- The number of seconds since UNIX epoch (January 1, 1970).
- ctime()
- Convert number of seconds since UNIX epoch to date.
- pp()
- A string describing the probe point currently being handled.
- thread_indent()
- This particular function is quite useful, providing you with a way to better organize your print results. The function takes one argument, an indentation delta, which indicates how many spaces to add or remove from a thread's "indentation counter". It then returns a string with some generic trace data along with an appropriate number of indentation spaces.The generic data included in the returned string includes a timestamp (number of microseconds since the first call to
thread_indent()
by the thread), a process name, and the thread ID. This allows you to identify what functions were called, who called them, and the duration of each function call.If call entries and exits immediately precede each other, it is easy to match them. However, in most cases, after a first function call entry is made, several other call entries and exits may be made before the first call exits. The indentation counter helps you match an entry with its corresponding exit by indenting the next function call if it is not the exit of the previous one.Consider the following example on the use ofthread_indent()
:Example 3.6. thread_indent.stp
probe kernel.function("*@net/socket.c") { printf ("%s -> %s\n", thread_indent(1), probefunc()) } probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) }
Example 3.6, “thread_indent.stp” prints out thethread_indent()
andprobe
functions at each event in the following format:0 ftp(7223): -> sys_socketcall 1159 ftp(7223): -> sys_socket 2173 ftp(7223): -> __sock_create 2286 ftp(7223): -> sock_alloc_inode 2737 ftp(7223): <- sock_alloc_inode 3349 ftp(7223): -> sock_alloc 3389 ftp(7223): <- sock_alloc 3417 ftp(7223): <- __sock_create 4117 ftp(7223): -> sock_create 4160 ftp(7223): <- sock_create 4301 ftp(7223): -> sock_map_fd 4644 ftp(7223): -> sock_map_file 4699 ftp(7223): <- sock_map_file 4715 ftp(7223): <- sock_map_fd 4732 ftp(7223): <- sys_socket 4775 ftp(7223): <- sys_socketcall
This sample output contains the following information:- The time (in microseconds) since the initial
thread_indent()
call for the thread. - The process name (and its corresponding ID) that made the function call.
- An arrow signifying whether the call was an entry (
<-
) or an exit (->
); the indentations help you match specific function call entries with their corresponding exits. - The name of the function called by the process.
- name
- Identifies the name of a specific system call. This variable can only be used in probes that use the event
syscall.system_call
. - target()
- Used in conjunction with either of the following two commands:
stap script -x process ID
stap script -c command
If you want to specify a script to take an argument of a process ID or command, usetarget()
as the variable in the script to refer to it. For example:Example 3.7. targetexample.stp
probe syscall.* { if (pid() == target()) printf("%s/n", name) }
When Example 3.7, “targetexample.stp” is run with the argument-x process ID
, it watches all system calls (as specified by thesyscall.*
event) and prints out the name of all system calls made by the specified process.This has the same effect as specifyingif (pid() == process ID)
each time you wish to target a specific process. However, usingtarget()
makes it easier to re-use the script, giving you the ability to simply pass a process ID as an argument each time you wish to run the script. For example:stap targetexample.stp -x process ID
3.3. Basic SystemTap Handler Constructs
awk
syntax. This section describes several of the most useful SystemTap handler constructs, which should provide you with enough information to write simple yet useful SystemTap scripts.
3.3.1. Variables
var
to gettimeofday_s()
(as in var = gettimeofday_s()
), then var
is typed as a number and can be printed in a printf()
with the integer format specifier (%d
).
global
outside of the probes. Consider the following example:
Example 3.8. timer-jiffies.stp
global count_jiffies, count_ms probe timer.jiffies(100) { count_jiffies ++ } probe timer.ms(100) { count_ms ++ } probe timer.ms(12345) { hz=(1000*count_jiffies) / count_ms printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n", count_jiffies, count_ms, hz) exit () }
CONFIG_HZ
setting of the kernel using timers that count jiffies and milliseconds, then computing accordingly. The global
statement allows the script to use the variables count_jiffies
and count_ms
(set in their own respective probes) to be shared with probe timer.ms(12345)
.
Note
++
notation in Example 3.8, “timer-jiffies.stp” (count_jiffies ++
and count_ms ++
) is used to increment the value of a variable by 1. In the following probe, count_jiffies
is incremented by 1 every 100 jiffies:
probe timer.jiffies(100) { count_jiffies ++ }
count_jiffies
is an integer. Because no initial value was assigned to count_jiffies
, its initial value is zero by default.
3.3.2. Conditional Statements
- If/Else Statements
- Format:
if (condition) statement1 else statement2
Thestatement1
is executed if thecondition
expression is non-zero. Thestatement2
is executed if thecondition
expression is zero. Theelse
clause (else
statement2) is optional. Bothstatement1
andstatement2
can be statement blocks.Example 3.9. ifelse.stp
global countread, countnonread probe kernel.function("vfs_read"),kernel.function("vfs_write") { if (probefunc()=="vfs_read") countread ++ else countnonread ++ } probe timer.s(5) { exit() } probe end { printf("VFS reads total %d\n VFS writes total %d\n", countread, countnonread) }
Example 3.9, “ifelse.stp” is a script that counts how many virtual file system reads (vfs_read
) and writes (vfs_write
) the system performs within a 5-second span. When run, the script increments the value of the variablecountread
by 1 if the name of the function it probed matchesvfs_read
(as noted by the conditionif (probefunc()=="vfs_read")
); otherwise, it incrementscountnonread
(else {countnonread ++}
). - While Loops
- Format:
while (condition) statement
So long ascondition
is non-zero the block of statements instatement
are executed. Thestatement
is often a statement block and it must change a value socondition
will eventually be zero. - For Loops
- Format:
for (initialization; conditional; increment) statement
Thefor
loop is simply shorthand for a while loop. The following is the equivalentwhile
loop:initialization while (conditional) { statement increment }
Aside from ==
(is equal to), you can also use the following operators in your conditional statements:
- >=
- Greater than or equal to
- <=
- Less than or equal to
- !=
- Is not equal to
3.3.3. Command-Line Arguments
$
or @
immediately followed by the number of the argument on the command line. Use $
if you are expecting the user to enter an integer as a command-line argument, and @
if you are expecting a string.
Example 3.10. commandlineargs.stp
probe kernel.function(@1) { } probe kernel.function(@1).return { }
stap commandlineargs.stp kernel function
). You can also specify the script to accept multiple command-line arguments, noting them as @1
, @2
, and so on, in the order they are entered by the user.
3.4. Associative Arrays
global
variables in the SystemTap script. The syntax for accessing an element in an associative array is similar to that of awk
, and is as follows:
array_name[index_expression]
array_name
is any arbitrary name the array uses. The index_expression
is used to see a specific unique key in the array. To illustrate, let us try to build an array named arr
that specifies the ages of three people (the unique keys): tom
, dick
, and harry
. To assign them the ages (associated values) of 23, 24, and 25 respectively, we'd use the following array statements:
Example 3.11. Basic Array Statements
arr["tom"] = 23 arr["dick"] = 24 arr["harry"] = 25
,
). This is useful if you wish to have a key that contains multiple pieces of information. The following line from Example 4.9, “disktop.stp” uses 5 elements for the key: process ID, executable name, user ID, parent process ID, and string "W". It associates the value of devname
with that key.
device[pid(),execname(),uid(),ppid(),"W"] = devname
Important
global
, regardless of whether the associate array is used in one or multiple probes.
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
=
to set an associated value to indexed unique pairs, as in:
array_name[index_expression] = value
index_expression
and value
. For example, you can use arrays to set a timestamp as the associated value to a process name (which you wish to use as your unique key), as in:
Example 3.12. Associating Timestamps to Process Names
arr[tid()] = gettimeofday_s()
tid()
value (that is, the ID of a thread, which is then used as the unique key). At the same time, SystemTap also uses the function gettimeofday_s()
to set the corresponding timestamp as the associated value to the unique key defined by the function tid()
. This creates an array composed of key pairs containing thread IDs and timestamps.
tid()
returns a value that is already defined in the array arr
, the operator will discard the original associated value to it, and replace it with the current timestamp from gettimeofday_s()
.
3.5.2. Reading Values From Arrays
array_name[index_expression]
statement as an element in a mathematical expression. For example:
Example 3.13. Using Array Values in Simple Computations
delta = gettimeofday_s() - arr[tid()]
arr
was built using the construct in Example 3.12, “Associating Timestamps to Process Names” (from Section 3.5.1, “Assigning an Associated Value”). This sets a timestamp that will serve as a reference point, to be used in computing for delta
.
delta
by subtracting the associated value of the key tid()
from the current gettimeofday_s()
. The construct does this by reading the value of tid()
from the array. This particular construct is useful for determining the time between two events, such as the start and completion of a read operation.
Note
index_expression
cannot find the unique key, it returns a value of 0 (for numerical operations, such as Example 3.13, “Using Array Values in Simple Computations”) or a null (empty) string value (for string operations) by default.
3.5.3. Incrementing Associated Values
++
to increment the associated value of a unique key in an array, as in:
array_name[index_expression] ++
index_expression
. For example, if you wanted to tally how many times a specific process performed a read to the virtual file system (using the vfs.read
event), you can use the following probe:
Example 3.14. vfsreads.stp
probe vfs.read { reads[execname()] ++ }
gnome-terminal
(that is, the first time gnome-terminal
performs a VFS read), that process name is set as the unique key gnome-terminal
with an associated value of 1. The next time that the probe returns the process name gnome-terminal
, SystemTap increments the associated value of gnome-terminal
by 1. SystemTap performs this operation for all process names as the probe returns them.
3.5.4. Processing Multiple Elements in an Array
reads
array.
foreach
statement. Consider the following example:
Example 3.15. cumulative-vfsreads.stp
global reads probe vfs.read { reads[execname()] ++ } probe timer.s(3) { foreach (count in reads) printf("%s : %d \n", count, reads[count]) }
foreach
statement uses the count
variable to reference each iteration of a unique key in the reads
array. The reads[count]
array statement in the same probe retrieves the associated value of each unique key.
foreach
statement in Example 3.15, “cumulative-vfsreads.stp” prints all iterations of process names in the array, and in no particular order. You can instruct the script to process the iterations in a particular order by using +
(ascending) or -
(descending). In addition, you can also limit the number of iterations the script needs to process with the limit value
option.
probe timer.s(3) { foreach (count in reads- limit 10) printf("%s : %d \n", count, reads[count]) }
foreach
statement instructs the script to process the elements in the array reads
in descending order (of associated value). The limit 10
option instructs the foreach
to only process the first ten iterations (that is, print the first 10, starting with the highest value).
3.5.5. Clearing/Deleting Arrays and Array Elements
delete
operator to delete elements in an array, or an entire array. Consider the following example:
Example 3.16. noncumulative-vfsreads.stp
global reads probe vfs.read { reads[execname()] ++ } probe timer.s(3) { foreach (count in reads) printf("%s : %d \n", count, reads[count]) delete reads }
delete reads
statement clears the reads
array within the probe.
Note
global reads, totalreads probe vfs.read { reads[execname()] ++ totalreads[execname()] ++ } probe timer.s(3) { printf("=======\n") foreach (count in reads-) printf("%s : %d \n", count, reads[count]) delete reads } probe end { printf("TOTALS\n") foreach (total in totalreads-) printf("%s : %d \n", total, totalreads[total]) }
reads
and totalreads
track the same information, and are printed out in a similar fashion. The only difference here is that reads
is cleared every 3-second period, whereas totalreads
keeps growing.
3.5.6. Using Arrays in Conditional Statements
if
statements. This is useful if you want to execute a subroutine once a value in the array matches a certain condition. Consider the following example:
Example 3.17. vfsreads-print-if-1kb.stp
global reads probe vfs.read { reads[execname()] ++ } probe timer.s(3) { printf("=======\n") foreach (count in reads-) if (reads[count] >= 1024) printf("%s : %dkB \n", count, reads[count]/1024) else printf("%s : %dB \n", count, reads[count]) }
if
statement in the script converts and prints it out in kB
.
You can also test whether a specific unique key is a member of an array. Further, membership in an array can be used in if
statements, as in:
if([index_expression] in array_name) statement
Example 3.18. vfsreads-stop-on-stapio2.stp
global reads probe vfs.read { reads[execname()] ++ } probe timer.s(3) { printf("=======\n") foreach (count in reads+) printf("%s : %d \n", count, reads[count]) if(["stapio"] in reads) { printf("stapio read detected, exiting\n") exit() } }
if(["stapio"] in reads)
statement instructs the script to print stapio read detected, exiting
once the unique key stapio
is added to the array reads
.
3.5.7. Computing for Statistical Aggregates
<<< value
.
Example 3.19. stat-aggregates.stp
global reads probe vfs.read { reads[execname()] <<< count }
<<< count
stores the amount returned by count
to the associated value of the corresponding execname()
in the reads
array. Remember, these values are stored; they are not added to the associated values of each unique key, nor are they used to replace the current associated values. In a manner of speaking, think of it as having each unique key (execname()
) having multiple associated values, accumulating with each probe handler run.
Note
count
returns the amount of data written by the returned execname()
to the virtual file system.
@extractor(variable/array index expression)
. extractor
can be any of the following integer extractors:
- count
- Returns the number of all values stored into the variable/array index expression. Given the sample probe in Example 3.19, “stat-aggregates.stp”, the expression
@count(writes[execname()])
will return how many values are stored in each unique key in arraywrites
. - sum
- Returns the sum of all values stored into the variable/array index expression. Again, given sample probe in Example 3.19, “stat-aggregates.stp”, the expression
@sum(writes[execname()])
will return the total of all values stored in each unique key in arraywrites
. - min
- Returns the smallest among all the values stored in the variable/array index expression.
- max
- Returns the largest among all the values stored in the variable/array index expression.
- avg
- Returns the average of all values stored in the variable/array index expression.
Example 3.20. Multiple Array Indexes
global reads probe vfs.read { reads[execname(),pid()] <<< 1 } probe timer.s(3) { foreach([var1,var2] in reads) printf("%s (%d) : %d \n", var1, var2, @count(reads[var1,var2])) }
reads
. Note how the foreach
statement uses the same number of variables (var1
and var2
) contained in the first instance of the array reads
from the first probe.
3.6. Tapsets
.stp
. The standard library of tapsets is located in the /usr/share/systemtap/tapset/
directory by default. However, unlike SystemTap scripts, tapsets are not meant for direct execution; rather, they constitute the library from which other scripts can pull definitions.
thread_indent()
is defined in indent.stp
.
Chapter 4. Useful SystemTap Scripts
/usr/share/systemtap/testsuite/systemtap.examples/
directory once you install the systemtap-testsuite package.
4.1. Network
4.1.1. Network Profiling
Example 4.1. nettop.stp
#! /usr/bin/env stap global ifxmit, ifrecv global ifmerged probe netdev.transmit { ifxmit[pid(), dev_name, execname(), uid()] <<< length } probe netdev.receive { ifrecv[pid(), dev_name, execname(), uid()] <<< length } function print_activity() { printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n", "PID", "UID", "DEV", "XMIT_PK", "RECV_PK", "XMIT_KB", "RECV_KB", "COMMAND") foreach ([pid, dev, exec, uid] in ifrecv) { ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifxmit) { ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]); } foreach ([pid, dev, exec, uid] in ifmerged-) { n_xmit = @count(ifxmit[pid, dev, exec, uid]) n_recv = @count(ifrecv[pid, dev, exec, uid]) printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n", pid, uid, dev, n_xmit, n_recv, n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0, n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0, exec) } print("\n") delete ifxmit delete ifrecv delete ifmerged } probe timer.ms(5000), end, error { print_activity() }
print_activity()
function uses the following expressions:
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0
if
or else
conditionals. The second statement is simply a more concise way of writing the following pseudo code:
if n_recv != 0 then @sum(ifrecv[pid, dev, exec, uid])/1024 else 0
PID
— the ID of the listed process.UID
— user ID. A user ID of0
refers to the root user.DEV
— which ethernet device the process used to send or receive data (for example, eth0, eth1)XMIT_PK
— number of packets transmitted by the processRECV_PK
— number of packets received by the processXMIT_KB
— amount of data sent by the process, in kilobytesRECV_KB
— amount of data received by the service, in kilobytes
probe timer.ms(5000)
accordingly. Example 4.2, “Example 4.1, “nettop.stp” Sample Output” contains an excerpt of the output from Example 4.1, “nettop.stp” over a 20-second period:
Example 4.2. Example 4.1, “nettop.stp” Sample Output
[...] PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 5 0 0 swapper 11178 0 eth0 2 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 2886 4 eth0 79 0 5 0 cups-polld 11362 0 eth0 0 61 0 5 firefox 0 0 eth0 3 32 0 3 swapper 2886 4 lo 4 4 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 6 0 0 swapper 2886 4 lo 2 2 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc 3611 0 eth0 0 1 0 0 Xorg PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 3 42 0 2 swapper 11178 0 eth0 43 1 3 0 synergyc 11362 0 eth0 0 7 0 0 firefox 3897 0 eth0 0 1 0 0 multiload-apple [...]
4.1.2. Tracing Functions Called in Network Socket Code
net/socket.c
file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level.
Example 4.3. socket-trace.stp
#!/usr/bin/stap probe kernel.function("*@net/socket.c").call { printf ("%s -> %s\n", thread_indent(1), probefunc()) } probe kernel.function("*@net/socket.c").return { printf ("%s <- %s\n", thread_indent(-1), probefunc()) }
thread_indent()
works.
Example 4.4. Example 4.3, “socket-trace.stp” Sample Output
[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 gnome-terminal(11106): -> sock_poll 5 gnome-terminal(11106): <- sock_poll 0 scim-bridge(3883): -> sock_poll 3 scim-bridge(3883): <- sock_poll 0 scim-bridge(3883): -> sys_socketcall 4 scim-bridge(3883): -> sys_recv 8 scim-bridge(3883): -> sys_recvfrom 12 scim-bridge(3883):-> sock_from_file 16 scim-bridge(3883):<- sock_from_file 20 scim-bridge(3883):-> sock_recvmsg 24 scim-bridge(3883):<- sock_recvmsg 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall [...]
thread_indent()
, see SystemTap Functions Example 3.6, “thread_indent.stp”.
4.1.3. Monitoring Incoming TCP Connections
Example 4.5. tcp_connections.stp
#! /usr/bin/env stap probe begin { printf("%6s %16s %6s %6s %16s\n", "UID", "CMD", "PID", "PORT", "IP_SOURCE") } probe kernel.function("tcp_accept").return?, kernel.function("inet_csk_accept").return? { sock = $return if (sock != 0) printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(), inet_get_local_port(sock), inet_get_ip_source(sock)) }
- Current
UID
CMD
- the command accepting the connectionPID
of the command- Port used by the connection
- IP address from which the TCP connection originated
Example 4.6. Example 4.5, “tcp_connections.stp” Sample Output
UID CMD PID PORT IP_SOURCE 0 sshd 3165 22 10.64.0.227 0 sshd 3165 22 10.64.0.227
4.1.4. Monitoring Network Packets Drops in Kernel
kernel.trace("kfree_skb")
, which easily tracks where packets are discarded. Example 4.7, “dropwatch.stp” uses kernel.trace("kfree_skb")
to trace packet discards; the script summarizes which locations discard packets every five-second interval.
Example 4.7. dropwatch.stp
#!/usr/bin/stap ############################################################ # Dropwatch.stp # Author: Neil Horman <nhorman@redhat.com> # An example script to mimic the behavior of the dropwatch utility # http://fedorahosted.org/dropwatch ############################################################ # Array to hold the list of drop points we find global locations # Note when we turn the monitor on and off probe begin { printf("Monitoring for dropped packets\n") } probe end { printf("Stopping dropped packet monitor\n") } # increment a drop counter for every location we drop at probe kernel.trace("kfree_skb") { locations[$location] <<< 1 } # Every 5 seconds report our drop locations probe timer.sec(5) { printf("\n") foreach (l in locations-) { printf("%d packets dropped at location %p\n", @count(locations[l]), l) } delete locations }
kernel.trace("kfree_skb")
traces which places in the kernel drop network packets. The kernel.trace("kfree_skb")
has two arguments: a pointer to the buffer being freed ($skb
) and the location in kernel code the buffer is being freed ($location
).
Example 4.8. Example 4.7, “dropwatch.stp” Sample Output
Monitoring for dropped packets 51 packets dropped at location 0xffffffff8024cd0f 2 packets dropped at location 0xffffffff8044b472 51 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 97 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 Stopping dropped packet monitor
/boot/System.map-$(uname -r)
file. This file lists the starting addresses for each function, allowing you to map the addresses in the output of Example 4.8, “Example 4.7, “dropwatch.stp” Sample Output” to a specific function name. Given the following snippet of the /boot/System.map-$(uname -r)
file, the address 0xffffffff8024cd0f maps to the function unix_stream_recvmsg
and the address 0xffffffff8044b472 maps to the function arp_rcv
:
[...] ffffffff8024c5cd T unlock_new_inode ffffffff8024c5da t unix_stream_sendmsg ffffffff8024c920 t unix_stream_recvmsg ffffffff8024cea1 t udp_v4_lookup_longway [...] ffffffff8044addc t arp_process ffffffff8044b360 t arp_rcv ffffffff8044b487 t parp_redo ffffffff8044b48c t arp_solicit [...]
4.2. Disk
4.2.1. Summarizing Disk Read/Write Traffic
Example 4.9. disktop.stp
#!/usr/bin/stap # # Copyright (C) 2007 Oracle Corp. # # Get the status of reading/writing disk every 5 seconds, # output top ten entries # # This is free software,GNU General Public License (GPL); # either version 2, or (at your option) any later version. # # Usage: # ./disktop.stp # global io_stat,device global read_bytes,write_bytes probe vfs.read.return { if ($return>0) { if (devname!="N/A") {/*skip read from cache*/ io_stat[pid(),execname(),uid(),ppid(),"R"] += $return device[pid(),execname(),uid(),ppid(),"R"] = devname read_bytes += $return } } } probe vfs.write.return { if ($return>0) { if (devname!="N/A") { /*skip update cache*/ io_stat[pid(),execname(),uid(),ppid(),"W"] += $return device[pid(),execname(),uid(),ppid(),"W"] = devname write_bytes += $return } } } probe timer.ms(5000) { /* skip non-read/write disk */ if (read_bytes+write_bytes) { printf("\n%-25s, %-8s%4dKb/sec, %-7s%6dKb, %-7s%6dKb\n\n", ctime(gettimeofday_s()), "Average:", ((read_bytes+write_bytes)/1024)/5, "Read:",read_bytes/1024, "Write:",write_bytes/1024) /* print header */ printf("%8s %8s %8s %25s %8s %4s %12s\n", "UID","PID","PPID","CMD","DEVICE","T","BYTES") } /* print top ten I/O */ foreach ([process,cmd,userid,parent,action] in io_stat- limit 10) printf("%8d %8d %8d %25s %8s %4s %12d\n", userid,process,parent,cmd, device[process,cmd,userid,parent,action], action,io_stat[process,cmd,userid,parent,action]) /* clear data */ delete io_stat delete device read_bytes = 0 write_bytes = 0 } probe end{ delete io_stat delete device delete read_bytes delete write_bytes }
UID
— user ID. A user ID of0
refers to the root user.PID
— the ID of the listed process.PPID
— the process ID of the listed process's parent process.CMD
— the name of the listed process.DEVICE
— which storage device the listed process is reading from or writing to.T
— the type of action performed by the listed process;W
refers to write, andR
refers to read.BYTES
— the amount of data read to or written from disk.
ctime()
and gettimeofday_s()
. ctime()
derives calendar time in terms of seconds passed since the start of the Unix time (January 1, 1970). gettimeofday_s()
counts the actual number of seconds since the start of the Unix time, which gives a fairly accurate human-readable timestamp for the output.
$return
is a local variable that stores the actual number of bytes each process reads or writes from the virtual file system. $return
can only be used in return probes (for example, vfs.read.return
and vfs.read.return
).
Example 4.10. Example 4.9, “disktop.stp” Sample Output
[...] Mon Sep 29 03:38:28 2008 , Average: 19Kb/sec, Read: 7Kb, Write: 89Kb UID PID PPID CMD DEVICE T BYTES 0 26319 26294 firefox sda5 W 90229 0 2758 2757 pam_timestamp_c sda5 R 8064 0 2885 1 cupsd sda5 W 1678 Mon Sep 29 03:38:38 2008 , Average: 1Kb/sec, Read: 7Kb, Write: 1Kb UID PID PPID CMD DEVICE T BYTES 0 2758 2757 pam_timestamp_c sda5 R 8064 0 2885 1 cupsd sda5 W 1678
4.2.2. Tracking I/O Time For Each File Read or Write
Example 4.11. iotime.stp
global start global entry_io global fd_io global time_io function timestamp:long() { return gettimeofday_us() - start } function proc:string() { return sprintf("%d (%s)", pid(), execname()) } probe begin { start = gettimeofday_us() } global filenames global filehandles global fileread global filewrite probe syscall.open { filenames[pid()] = user_string($filename) } probe syscall.open.return { if ($return != -1) { filehandles[pid(), $return] = filenames[pid()] fileread[pid(), $return] = 0 filewrite[pid(), $return] = 0 } else { printf("%d %s access %s fail\n", timestamp(), proc(), filenames[pid()]) } delete filenames[pid()] } probe syscall.read { if ($count > 0) { fileread[pid(), $fd] += $count } t = gettimeofday_us(); p = pid() entry_io[p] = t fd_io[p] = $fd } probe syscall.read.return { t = gettimeofday_us(); p = pid() fd = fd_io[p] time_io[p,fd] <<< t - entry_io[p] } probe syscall.write { if ($count > 0) { filewrite[pid(), $fd] += $count } t = gettimeofday_us(); p = pid() entry_io[p] = t fd_io[p] = $fd } probe syscall.write.return { t = gettimeofday_us(); p = pid() fd = fd_io[p] time_io[p,fd] <<< t - entry_io[p] } probe syscall.close { if (filehandles[pid(), $fd] != "") { printf("%d %s access %s read: %d write: %d\n", timestamp(), proc(), filehandles[pid(), $fd], fileread[pid(), $fd], filewrite[pid(), $fd]) if (@count(time_io[pid(), $fd])) printf("%d %s iotime %s time: %d\n", timestamp(), proc(), filehandles[pid(), $fd], @sum(time_io[pid(), $fd])) } delete fileread[pid(), $fd] delete filewrite[pid(), $fd] delete filehandles[pid(), $fd] delete fd_io[pid()] delete entry_io[pid()] delete time_io[pid(),$fd] }
$count
to track the amount of data (in bytes) that any system call attempts to read or write. Note that $return
(as used in Example 4.9, “disktop.stp” from Section 4.2.1, “Summarizing Disk Read/Write Traffic”) stores the actual amount of data read/written. $count
can only be used on probes that track data reads or writes (e.g. syscall.read
and syscall.write
).
Example 4.12. Example 4.11, “iotime.stp” Sample Output
[...] 825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0 825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9 [...] 117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0 117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7 [...] 3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0 3973744 2886 (sendmail) iotime /proc/loadavg time: 11 [...]
- A timestamp, in microseconds.
- Process ID and process name.
- An
access
oriotime
flag. - The file accessed.
access
and iotime
lines should appear together. The access
line's timestamp refers to the time that a given process started accessing a file; at the end of the line, it will show the amount of data read/written (in bytes). The iotime
line will show the amount of time (in microseconds) that the process took in order to perform the read or write.
access
line is not followed by an iotime
line, it simply means that the process did not read or write any data.
4.2.3. Track Cumulative IO
Example 4.13. traceio.stp
#! /usr/bin/env stap # traceio.stp # Copyright (C) 2007 Red Hat, Inc., Eugene Teo <eteo@redhat.com> # Copyright (C) 2009 Kai Meyer <kai@unixlords.com> # Fixed a bug that allows this to run longer # And added the humanreadable function # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as # published by the Free Software Foundation. # global reads, writes, total_io probe vfs.read.return { reads[pid(),execname()] += $return total_io[pid(),execname()] += $return } probe vfs.write.return { writes[pid(),execname()] += $return total_io[pid(),execname()] += $return } function humanreadable(bytes) { if (bytes > 1024*1024*1024) { return sprintf("%d GiB", bytes/1024/1024/1024) } else if (bytes > 1024*1024) { return sprintf("%d MiB", bytes/1024/1024) } else if (bytes > 1024) { return sprintf("%d KiB", bytes/1024) } else { return sprintf("%d B", bytes) } } probe timer.s(1) { foreach([p,e] in total_io- limit 10) printf("%8d %15s r: %12s w: %12s\n", p, e, humanreadable(reads[p,e]), humanreadable(writes[p,e])) printf("\n") # Note we don't zero out reads, writes and total_io, # so the values are cumulative since the script started. }
$return
, which is also used by Example 4.9, “disktop.stp” from Section 4.2.1, “Summarizing Disk Read/Write Traffic”.
Example 4.14. Example 4.13, “traceio.stp” Sample Output
[...] Xorg r: 583401 KiB w: 0 KiB floaters r: 96 KiB w: 7130 KiB multiload-apple r: 538 KiB w: 537 KiB sshd r: 71 KiB w: 72 KiB pam_timestamp_c r: 138 KiB w: 0 KiB staprun r: 51 KiB w: 51 KiB snmpd r: 46 KiB w: 0 KiB pcscd r: 28 KiB w: 0 KiB irqbalance r: 27 KiB w: 4 KiB cupsd r: 4 KiB w: 18 KiB Xorg r: 588140 KiB w: 0 KiB floaters r: 97 KiB w: 7143 KiB multiload-apple r: 543 KiB w: 542 KiB sshd r: 72 KiB w: 72 KiB pam_timestamp_c r: 138 KiB w: 0 KiB staprun r: 51 KiB w: 51 KiB snmpd r: 46 KiB w: 0 KiB pcscd r: 28 KiB w: 0 KiB irqbalance r: 27 KiB w: 4 KiB cupsd r: 4 KiB w: 18 KiB
4.2.4. I/O Monitoring (By Device)
Example 4.15. traceio2.stp
#! /usr/bin/env stap global device_of_interest probe begin { /* The following is not the most efficient way to do this. One could directly put the result of usrdev2kerndev() into device_of_interest. However, want to test out the other device functions */ dev = usrdev2kerndev($1) device_of_interest = MKDEV(MAJOR(dev), MINOR(dev)) } probe vfs.write, vfs.read { if (dev == device_of_interest) printf ("%s(%d) %s 0x%x\n", execname(), pid(), probefunc(), dev) }
stat -c "0x%D" directory
, where directory is located on the device you wish to monitor.
usrdev2kerndev()
function converts the whole device number into the format understood by the kernel. The output produced by usrdev2kerndev()
is used in conjunction with the MKDEV()
, MINOR()
, and MAJOR()
functions to determine the major and minor numbers of a specific device.
vfs_read
or vfs_write
), and the kernel device number.
stap traceio2.stp 0x805
, where 0x805
is the whole device number of /home
. /home
resides in /dev/sda5
, which is the device we wish to monitor.
Example 4.16. Example 4.15, “traceio2.stp” Sample Output
[...] synergyc(3722) vfs_read 0x800005 synergyc(3722) vfs_read 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 cupsd(2889) vfs_write 0x800005 [...]
4.2.5. Monitoring Reads and Writes to a File
Example 4.17. inodewatch.stp
#! /usr/bin/env stap probe vfs.write, vfs.read { # dev and ino are defined by vfs.write and vfs.read if (dev == MKDEV($1,$2) # major/minor device && ino == $3) printf ("%s(%d) %s 0x%x/%u\n", execname(), pid(), probefunc(), dev, ino) }
- The file's major device number.
- The file's minor device number.
- The file's
inode
number.
stat -c '%D %i' filename
, where filename is an absolute path.
/etc/crontab
file, run stat -c '%D %i' /etc/crontab
first. This outputs the following output:
805 1078319
805
is the base-16 (hexadecimal) device number. The lower two digits are the minor device number, and the upper digits are the major number. 1078319
is the inode
number. To start monitoring /etc/crontab
, run stap inodewatch.stp 0x8 0x05 1078319
(The 0x
prefixes indicate base-16 values).
vfs_read
or vfs_write
), the device number (in hex format), and the inode
number. Example 4.18, “Example 4.17, “inodewatch.stp” Sample Output” contains the output of stap inodewatch.stp 0x8 0x05 1078319
(when cat /etc/crontab
is executed while the script is running):
Example 4.18. Example 4.17, “inodewatch.stp” Sample Output
cat(16437) vfs_read 0x800005/1078319 cat(16437) vfs_read 0x800005/1078319
4.2.6. Monitoring Changes to File Attributes
Example 4.19. inodewatch2-simple.stp
global ATTR_MODE = 1 probe kernel.function("inode_setattr") { dev_nr = $inode->i_sb->s_dev inode_nr = $inode->i_ino if (dev_nr == ($1 << 20 | $2) # major/minor device && inode_nr == $3 && $attr->ia_valid & ATTR_MODE) printf ("%s(%d) %s 0x%x/%u %o %d\n", execname(), pid(), probefunc(), dev_nr, inode_nr, $attr->ia_mode, uid()) }
inode
number as arguments. For more information on how to retrieve this information, see Section 4.2.5, “Monitoring Reads and Writes to a File”.
uid()
). Example 4.20, “Example 4.19, “inodewatch2-simple.stp” Sample Output” shows the output of Example 4.19, “inodewatch2-simple.stp” while monitoring /home/joe/bigfile
when user joe
executes chmod 777 /home/joe/bigfile
and chmod 666 /home/joe/bigfile
.
Example 4.20. Example 4.19, “inodewatch2-simple.stp” Sample Output
chmod(17448) inode_setattr 0x800005/6011835 100777 500 chmod(17449) inode_setattr 0x800005/6011835 100666 500
4.3. Profiling
4.3.1. Counting Function Calls Made
Example 4.21. functioncallcount.stp
#! /usr/bin/env stap # The following line command will probe all the functions # in kernel's memory management code: # # stap functioncallcount.stp "*@mm/*.c" probe kernel.function(@1).call { # probe functions listed on commandline called[probefunc()] <<< 1 # add a count efficiently } global called probe end { foreach (fn in called-) # Sort by call count (in decreasing order) # (fn+ in called) # Sort by function name printf("%s %d\n", fn, @count(called[fn])) exit() }
stap functioncallcount.stp "*@mm/*.c"
:
Example 4.22. Example 4.21, “functioncallcount.stp” Sample Output
[...] __vma_link 97 __vma_link_file 66 __vma_link_list 97 __vma_link_rb 97 __xchg 103 add_page_to_active_list 102 add_page_to_inactive_list 19 add_to_page_cache 19 add_to_page_cache_lru 7 all_vm_events 6 alloc_pages_node 4630 alloc_slabmgmt 67 anon_vma_alloc 62 anon_vma_free 62 anon_vma_lock 66 anon_vma_prepare 98 anon_vma_unlink 97 anon_vma_unlock 66 arch_get_unmapped_area_topdown 94 arch_get_unmapped_exec_area 3 arch_unmap_area_topdown 97 atomic_add 2 atomic_add_negative 97 atomic_dec_and_test 5153 atomic_inc 470 atomic_inc_and_test 1 [...]
4.3.2. Call Graph Tracing
Example 4.23. para-callgraph.stp
#! /usr/bin/env stap function trace(entry_p, extra) { %( $# > 1 %? if (tid() in trace) %) printf("%s%s%s %s\n", thread_indent (entry_p), (entry_p>0?"->":"<-"), probefunc (), extra) } %( $# > 1 %? global trace probe $2.call { trace[tid()] = 1 } probe $2.return { delete trace[tid()] } %) probe $1.call { trace(1, $$parms) } probe $1.return { trace(-1, $$return) }
- The function(s) whose entry/exit you'd like to trace (
$1
). - A second optional trigger function (
$2
), which enables or disables tracing on a per-thread basis. Tracing in each thread will continue as long as the trigger function has not exited yet.
thread_indent()
; as such, its output contains the timestamp, process name, and thread ID of $1
(the probe function you are tracing). For more information about thread_indent()
, see its entry in SystemTap Functions.
stap para-callgraph.stp 'kernel.function("*@fs/*.c")' 'kernel.function("sys_read")'
:
Example 4.24. Example 4.23, “para-callgraph.stp” Sample Output
[...] 267 gnome-terminal(2921): <-do_sync_read return=0xfffffffffffffff5 269 gnome-terminal(2921):<-vfs_read return=0xfffffffffffffff5 0 gnome-terminal(2921):->fput file=0xffff880111eebbc0 2 gnome-terminal(2921):<-fput 0 gnome-terminal(2921):->fget_light fd=0x3 fput_needed=0xffff88010544df54 3 gnome-terminal(2921):<-fget_light return=0xffff8801116ce980 0 gnome-terminal(2921):->vfs_read file=0xffff8801116ce980 buf=0xc86504 count=0x1000 pos=0xffff88010544df48 4 gnome-terminal(2921): ->rw_verify_area read_write=0x0 file=0xffff8801116ce980 ppos=0xffff88010544df48 count=0x1000 7 gnome-terminal(2921): <-rw_verify_area return=0x1000 12 gnome-terminal(2921): ->do_sync_read filp=0xffff8801116ce980 buf=0xc86504 len=0x1000 ppos=0xffff88010544df48 15 gnome-terminal(2921): <-do_sync_read return=0xfffffffffffffff5 18 gnome-terminal(2921):<-vfs_read return=0xfffffffffffffff5 0 gnome-terminal(2921):->fput file=0xffff8801116ce980
4.3.3. Determining Time Spent in Kernel and User Space
Example 4.25. thread-times.stp
#! /usr/bin/env stap probe perf.sw.cpu_clock!, timer.profile { // NB: To avoid contention on SMP machines, no global scalars/arrays used, // only contention-free statistics aggregates. tid=tid(); e=execname() if (!user_mode()) kticks[e,tid] <<< 1 else uticks[e,tid] <<< 1 ticks <<< 1 tids[e,tid] <<< 1 } global uticks, kticks, ticks global tids probe timer.s(5), end { allticks = @count(ticks) printf ("%16s %5s %7s %7s (of %d ticks)\n", "comm", "tid", "%user", "%kernel", allticks) foreach ([e,tid] in tids- limit 20) { uscaled = @count(uticks[e,tid])*10000/allticks // SystemTap only performs integer arithmetic. // To avoid losing precision the decimal point is shifted // to the right four places (*100000). Think of this as // the original result value x.xxyy becoming xxxyy.0. // The integer percentage xxx is obtained // by dividing by 100 and the fractional percentage // is obtained with a modulo 100 operation. kscaled = @count(kticks[e,tid])*10000/allticks printf ("%16s %5d %3d.%02d%% %3d.%02d%%\n", e, tid, uscaled/100, uscaled%100, kscaled/100, kscaled%100) } printf("\n") delete uticks delete kticks delete ticks delete tids }
Example 4.26. Example 4.25, “thread-times.stp” Sample Output
tid %user %kernel (of 20002 ticks) 0 0.00% 87.88% 32169 5.24% 0.03% 9815 3.33% 0.36% 9859 0.95% 0.00% 3611 0.56% 0.12% 9861 0.62% 0.01% 11106 0.37% 0.02% 32167 0.08% 0.08% 3897 0.01% 0.08% 3800 0.03% 0.00% 2886 0.02% 0.00% 3243 0.00% 0.01% 3862 0.01% 0.00% 3782 0.00% 0.00% 21767 0.00% 0.00% 2522 0.00% 0.00% 3883 0.00% 0.00% 3775 0.00% 0.00% 3943 0.00% 0.00% 3873 0.00% 0.00%
4.3.4. Monitoring Polling Applications
Example 4.27. timeout.stp
#! /usr/bin/env stap # Copyright (C) 2009 Red Hat, Inc. # Written by Ulrich Drepper <drepper@redhat.com> # Modified by William Cohen <wcohen@redhat.com> global process, timeout_count, to global poll_timeout, epoll_timeout, select_timeout, itimer_timeout global nanosleep_timeout, futex_timeout, signal_timeout probe syscall.poll, syscall.epoll_wait { if (timeout) to[pid()]=timeout } probe syscall.poll.return { p = pid() if ($return == 0 && to[p] > 0 ) { poll_timeout[p]++ timeout_count[p]++ process[p] = execname() delete to[p] } } probe syscall.epoll_wait.return { p = pid() if ($return == 0 && to[p] > 0 ) { epoll_timeout[p]++ timeout_count[p]++ process[p] = execname() delete to[p] } } probe syscall.select.return { if ($return == 0) { p = pid() select_timeout[p]++ timeout_count[p]++ process[p] = execname() } } probe syscall.futex.return { if (errno_str($return) == "ETIMEDOUT") { p = pid() futex_timeout[p]++ timeout_count[p]++ process[p] = execname() } } probe syscall.nanosleep.return { if ($return == 0) { p = pid() nanosleep_timeout[p]++ timeout_count[p]++ process[p] = execname() } } probe kernel.function("it_real_fn") { p = pid() itimer_timeout[p]++ timeout_count[p]++ process[p] = execname() } probe syscall.rt_sigtimedwait.return { if (errno_str($return) == "EAGAIN") { p = pid() signal_timeout[p]++ timeout_count[p]++ process[p] = execname() } } probe syscall.exit { p = pid() if (p in process) { delete process[p] delete timeout_count[p] delete poll_timeout[p] delete epoll_timeout[p] delete select_timeout[p] delete itimer_timeout[p] delete futex_timeout[p] delete nanosleep_timeout[p] delete signal_timeout[p] } } probe timer.s(1) { ansi_clear_screen() printf (" pid | poll select epoll itimer futex nanosle signal| process\n") foreach (p in timeout_count- limit 20) { printf ("%5d |%7d %7d %7d %7d %7d %7d %7d| %-.38s\n", p, poll_timeout[p], select_timeout[p], epoll_timeout[p], itimer_timeout[p], futex_timeout[p], nanosleep_timeout[p], signal_timeout[p], process[p]) } }
poll
select
epoll
itimer
futex
nanosleep
signal
Example 4.28. Example 4.27, “timeout.stp” Sample Output
uid | poll select epoll itimer futex nanosle signal| process 28937 | 148793 0 0 4727 37288 0 0| firefox 22945 | 0 56949 0 1 0 0 0| scim-bridge 0 | 0 0 0 36414 0 0 0| swapper 4275 | 23140 0 0 1 0 0 0| mixer_applet2 4191 | 0 14405 0 0 0 0 0| scim-launcher 22941 | 7908 1 0 62 0 0 0| gnome-terminal 4261 | 0 0 0 2 0 7622 0| escd 3695 | 0 0 0 0 0 7622 0| gdm-binary 3483 | 0 7206 0 0 0 0 0| dhcdbd 4189 | 6916 0 0 2 0 0 0| scim-panel-gtk 1863 | 5767 0 0 0 0 0 0| iscsid 2562 | 0 2881 0 1 0 1438 0| pcscd 4257 | 4255 0 0 1 0 0 0| gnome-power-man 4278 | 3876 0 0 60 0 0 0| multiload-apple 4083 | 0 1331 0 1728 0 0 0| Xorg 3921 | 1603 0 0 0 0 0 0| gam_server 4248 | 1591 0 0 0 0 0 0| nm-applet 3165 | 0 1441 0 0 0 0 0| xterm 29548 | 0 1440 0 0 0 0 0| httpd 1862 | 0 0 0 0 0 1438 0| iscsid
timer.s()
). The output of Example 4.21, “functioncallcount.stp” contains the name and UID of the top 20 polling applications, along with how many times each application performed each polling system call (over time). Example 4.28, “Example 4.27, “timeout.stp” Sample Output” contains an excerpt of the script:
4.3.5. Tracking Most Frequently Used System Calls
poll
select
epoll
itimer
futex
nanosleep
signal
Example 4.29. topsys.stp
#! /usr/bin/env stap # # This script continuously lists the top 20 systemcalls in the interval # 5 seconds # global syscalls_count probe syscall.* { syscalls_count[name]++ } function print_systop () { printf ("%25s %10s\n", "SYSCALL", "COUNT") foreach (syscall in syscalls_count- limit 20) { printf("%25s %10d\n", syscall, syscalls_count[syscall]) } delete syscalls_count } probe timer.s(5) { print_systop () printf("--------------------------------------------------------------\n") }
Example 4.30. Example 4.29, “topsys.stp” Sample Output
-------------------------------------------------------------- SYSCALL COUNT gettimeofday 1857 read 1821 ioctl 1568 poll 1033 close 638 open 503 select 455 write 391 writev 335 futex 303 recvmsg 251 socket 137 clock_gettime 124 rt_sigprocmask 121 sendto 120 setitimer 106 stat 90 time 81 sigreturn 72 fstat 66 --------------------------------------------------------------
4.3.6. Tracking System Call Volume Per Process
Example 4.31. syscalls_by_proc.stp
#! /usr/bin/env stap # Copyright (C) 2006 IBM Corp. # # This file is part of systemtap, and is free software. You can # redistribute it and/or modify it under the terms of the GNU General # Public License (GPL); either version 2, or (at your option) any # later version. # # Print the system call count by process name in descending order. # global syscalls probe begin { print ("Collecting data... Type Ctrl-C to exit and display results\n") } probe syscall.* { syscalls[execname()]++ } probe end { printf ("%-10s %-s\n", "#SysCalls", "Process Name") foreach (proc in syscalls-) printf("%-10d %-s\n", syscalls[proc], proc) }
Example 4.32. Example 4.29, “topsys.stp” Sample Output
Collecting data... Type Ctrl-C to exit and display results #SysCalls Process Name 1577 multiload-apple 692 synergyc 408 pcscd 376 mixer_applet2 299 gnome-terminal 293 Xorg 206 scim-panel-gtk 95 gnome-power-man 90 artsd 85 dhcdbd 84 scim-bridge 78 gnome-screensav 66 scim-launcher [...]
Example 4.33. syscalls_by_pid.stp
#! /usr/bin/env stap # Copyright (C) 2006 IBM Corp. # # This file is part of systemtap, and is free software. You can # redistribute it and/or modify it under the terms of the GNU General # Public License (GPL); either version 2, or (at your option) any # later version. # # Print the system call count by process ID in descending order. # global syscalls probe begin { print ("Collecting data... Type Ctrl-C to exit and display results\n") } probe syscall.* { syscalls[pid()]++ } probe end { printf ("%-10s %-s\n", "#SysCalls", "PID") foreach (pid in syscalls-) printf("%-10d %-d\n", syscalls[pid], pid) }
timer.s()
probe; for example, to instruct the script to expire after 5 seconds, add the following probe to the script:
probe timer.s(5) { exit() }
4.4. Identifying Contended User-Space Locks
futex
contentions.
futex
contention occurs when multiple processes are trying to access the same region of memory. In some cases, this can result in a deadlock between the processes in contention, thereby appearing as an application hang.
futex
system call.
Example 4.34. futexes.stp
#! /usr/bin/env stap # This script tries to identify contended user-space locks by hooking # into the futex system call. global thread_thislock # short global thread_blocktime # global FUTEX_WAIT = 0 /*, FUTEX_WAKE = 1 */ global lock_waits # long-lived stats on (tid,lock) blockage elapsed time global process_names # long-lived pid-to-execname mapping probe syscall.futex { if (op != FUTEX_WAIT) next # don't care about WAKE event originator t = tid () process_names[pid()] = execname() thread_thislock[t] = $uaddr thread_blocktime[t] = gettimeofday_us() } probe syscall.futex.return { t = tid() ts = thread_blocktime[t] if (ts) { elapsed = gettimeofday_us() - ts lock_waits[pid(), thread_thislock[t]] <<< elapsed delete thread_blocktime[t] delete thread_thislock[t] } } probe end { foreach ([pid+, lock] in lock_waits) printf ("%s[%d] lock %p contended %d times, %d avg us\n", process_names[pid], pid, lock, @count(lock_waits[pid,lock]), @avg(lock_waits[pid,lock])) }
- Name and ID of the process responsible for a contention
- The region of memory it contested
- How many times the region of memory was contended
- Average time of contention throughout the probe
Example 4.35. Example 4.34, “futexes.stp” Sample Output
[...] automount[2825] lock 0x00bc7784 contended 18 times, 999931 avg us synergyc[3686] lock 0x0861e96c contended 192 times, 101991 avg us synergyc[3758] lock 0x08d98744 contended 192 times, 101990 avg us synergyc[3938] lock 0x0982a8b4 contended 192 times, 101997 avg us [...]
Chapter 5. Understanding SystemTap Errors
[man warning::example1] [man error::example2]
warning::example1
and error::example2
which provides explanatory information specific to that diagnostic.
5.1. Parse and Semantic Errors
The script contains a grammatical/typographical error. SystemTap detected type of construct that is incorrect, given the context of the probe.
probe vfs.read probe vfs.write
probe
keyword in column 1 of line 2:
parse error: expected one of '. , ( ? ! { = +=' saw: keyword at perror.stp:2:1 1 parse error(s).
The script contains unsafe embedded C code (blocks of code surrounded by %{
%}
. SystemTap allows you to embed C code in a script, which is useful if there are no tapsets to suit your purposes. However, embedded C constructs are not safe; as such, SystemTap warns you with this error if such constructs appear in the script.
stapdev
group (or have root privileges), run the script in guru mode by using the -g
option:
stap -g script
The ident
function in the script used the wrong type (%s
or %d
). This error presents itself in Example 5.1, “error-variable.stp”. Because the execname()
function returns a string, the format specifier should be %s
, not %d
.
Example 5.1. error-variable.stp
probe syscall.open { printf ("%d(%d) open\n", execname(), pid()) }
The identifier (a variable, for example) was used, but no type (integer or string) could be determined. This occurs, for instance, if you use a variable in a printf
statement while the script never assigns a value to the variable.
SystemTap could not assign a value to a variable or to a location in an array. The destination for the assignment is not a valid destination. The following example code would generate this error:
probe begin { printf("x") = 1 }
A function call or array index expression in the script used an invalid number of arguments/parameters. In SystemTap arity can either see the number of indices for an array, or the number of parameters to a function.
The script used an array operation without declaring the array as a global variable (global variables can be declared after their use in SystemTap scripts). Similar messages appear if an array is used, but with inconsistent arities.
The var
array is being modifed (being assigned to or deleted from) within an active foreach
loop. This error also displays if an operation within the script performs a function call within the foreach
loop.
SystemTap did not understand what the event or SystemTap function pnt
refers to. This usually means that SystemTap could not find a match for pnt
in the tapset library. The N refers to the line and column of the error.
The pnt
events and handler function could not be resolved for a variety of reasons. This error occurs when the script contains the kernel.function("name")
event, and name does not exist. In some cases, the error could also mean the script contains an invalid kernel file name or source-line number.
A handler in the script references a target variable, but the value of the variable could not be resolved. This error could also mean that a handler is referencing a target variable that is not valid in the context when it was referenced. This may be a result of compiler optimization of the generated code.
There was a problem processing the debugging information. In most cases, this error results from the installation of a kernel-debuginfo package. The installed kernel-debuginfo package itself may have some consistency or correctness problems.
SystemTap could not find a suitable kernel-debuginfo at all.
5.2. Run Time Errors and Warnings
Errors, skipped probes, or both occurred during this run. N and M are the numbers of errors encountered and probes skipped due to conditions, such as too much time required to execute event handlers over an interval of time.
A statistics extractor function other than @count
was invoked on an aggregate that has not had any values accumulated yet. This is similar to a division by zero.
An array containing aggregate values contains too many distinct key pairs at this time.
Too many levels of function call nesting were attempted. The default nesting of function calls allowed is 10.
The probe handler attempted to execute too many statements in the probe handler. The default number of actions allowed in a probe handler is 1000.
Chapter 6. References
- SystemTap Wiki
- The SystemTap Wiki is a collection of links and articles related to the deployment, usage, and development of SystemTap. You can find it at http://sourceware.org/systemtap/wiki/HomePage.
- SystemTap Tutorial
- Much of the content in this book comes from the SystemTap Tutorial. The SystemTap Tutorial is a more appropriate reference for users with intermediate to advanced knowledge of C++ and kernel development, and can be found at http://sourceware.org/systemtap/tutorial/.
- man stapprobes
- The
stapprobes
man page enumerates a variety of probe points supported by SystemTap, along with additional aliases defined by the SystemTap tapset library. The bottom of the man page includes a list of other man pages enumerating similar probe points for specific system components, such asstapprobes.scsi
,stapprobes.kprocess
,stapprobes.signal
, etc. - man stapfuncs
- The
stapfuncs
man page enumerates numerous functions supported by the SystemTap tapset library, along with the prescribed syntax for each one. Note, however, that this is not a complete list of all supported functions; there are more undocumented functions available. - SystemTap Language Reference
- This document is a comprehensive reference of SystemTap's language constructs and syntax. It is recommended for users with a rudimentary to intermediate knowledge of C++ and other similar programming languages. The SystemTap Language Reference is available to all users at http://sourceware.org/systemtap/langref/
- Tapset Developers Guide
- Once you have sufficient proficiency in writing SystemTap scripts, you can then try your hand out on writing your own tapsets. The Tapset Developers Guide describes how to add functions to your tapset library.
- Test Suite
- The
systemtap-testsuite
package allows you to test the entire SystemTap toolchain without having to build from source. In addition, it also contains numerous examples of SystemTap scripts you can study and test; some of these scripts are also documented in Chapter 4, Useful SystemTap Scripts.By default, the example scripts included insystemtap-testsuite
are located in/usr/share/systemtap/testsuite/systemtap.examples
.
Appendix A. Revision History
Revision History | |||
---|---|---|---|
Revision 7-7 | Mon Aug 5 2019 | ||
| |||
Revision 7-6 | Tue Oct 30 2018 | ||
| |||
Revision 7-5.1 | Mon Jun 18 2018 | ||
| |||
Revision 7-5 | Tue Jan 09 2018 | ||
| |||
Revision 7-4 | Wed Jul 26 2017 | ||
| |||
Revision 7-3.9 | Tue May 16 2017 | ||
| |||
Revision 1-8 | Wed Oct 19 2016 | ||
| |||
Revision 1-6 | Wed Jan 20 2016 | ||
| |||
Revision 1-5 | Thu Nov 11 2015 | ||
| |||
Revision 0-3 | Fri Dec 6 2013 | ||
|
Index
Symbols
- $count
- sample usage
- local variables, Tracking I/O Time For Each File Read or Write
- $return
- sample usage
- local variables, Summarizing Disk Read/Write Traffic, Track Cumulative IO
- @avg (integer extractor)
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- @count (integer extractor)
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- @max (integer extractor)
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- @min (integer extractor)
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- @sum (integer extractor)
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
A
- adding values to statistical aggregates
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- advantages of cross-instrumentation, Generating Instrumentation for Other Computers
- aggregate element not found
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- aggregates (statistical)
- array operations, Computing for Statistical Aggregates
- aggregation overflow
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- algebraic formulas using arrays
- reading values from arrays
- array operations, Reading Values From Arrays
- architecture notation, determining, Installing Required Kernel Information Packages
- architecture of SystemTap, Architecture
- array locals not supported
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- array operations
- assigning associated values, Assigning an Associated Value
- associating timestamps to process names, Assigning an Associated Value
- associative arrays, Array Operations in SystemTap
- clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
- delete operator, Clearing/Deleting Arrays and Array Elements
- multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
- virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
- computing for statistical aggregates, Computing for Statistical Aggregates
- @avg (integer extractor), Computing for Statistical Aggregates
- @count (integer extractor), Computing for Statistical Aggregates
- @max (integer extractor), Computing for Statistical Aggregates
- @min (integer extractor), Computing for Statistical Aggregates
- @sum (integer extractor), Computing for Statistical Aggregates
- adding values to statistical aggregates, Computing for Statistical Aggregates
- count (operator), Computing for Statistical Aggregates
- extracting data collected by statistical aggregates, Computing for Statistical Aggregates
- conditional statements, using arrays in, Using Arrays in Conditional Statements
- testing for array membership, Using Arrays in Conditional Statements
- deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
- incrementing associated values, Incrementing Associated Values
- tallying virtual file system reads (VFS reads), Incrementing Associated Values
- multiple elements in an array, Processing Multiple Elements in an Array
- processing multiple elements in an array, Processing Multiple Elements in an Array
- cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
- foreach, Processing Multiple Elements in an Array
- iterations, processing elements in an array as, Processing Multiple Elements in an Array
- limiting the output of foreach, Processing Multiple Elements in an Array
- ordering the output of foreach, Processing Multiple Elements in an Array
- reading values from arrays, Reading Values From Arrays
- computing for timestamp deltas, Reading Values From Arrays
- empty unique keys, Reading Values From Arrays
- using arrays in simple computations, Reading Values From Arrays
- arrays, Associative Arrays
- (see also associative arrays)
- assigning associated values
- array operations, Assigning an Associated Value
- associating timestamps to process names, Assigning an Associated Value
- associating timestamps to process names
- array operations, Assigning an Associated Value
- associated values
- introduction
- arrays, Associative Arrays
- associating timestamps to process names
- assigning associated values
- array operations, Assigning an Associated Value
- associative arrays
- introduction, Associative Arrays
- associated values, Associative Arrays
- example, Associative Arrays
- index expression, Associative Arrays
- key pairs, Associative Arrays
- syntax, Associative Arrays
- unique keys, Associative Arrays
- asynchronous events
- Events, Event
B
- begin
- Events, Event
- building instrumentation modules from SystemTap scripts, Generating Instrumentation for Other Computers
- building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
C
- call graph tracing
- examples of SystemTap scripts, Call Graph Tracing
- capabilities of SystemTap
- Introduction, SystemTap Capabilities
- changes to file attributes, monitoring
- examples of SystemTap scripts, Monitoring Changes to File Attributes
- clearing arrays/array elements
- array operations, Clearing/Deleting Arrays and Array Elements
- delete operator, Clearing/Deleting Arrays and Array Elements
- multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
- virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
- command-line arguments
- SystemTap handler constructs
- handlers, Command-Line Arguments
- compiling instrumentation/kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
- components
- SystemTap scripts
- introduction, SystemTap Scripts
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- @avg (integer extractor), Computing for Statistical Aggregates
- @count (integer extractor), Computing for Statistical Aggregates
- @max (integer extractor), Computing for Statistical Aggregates
- @min (integer extractor), Computing for Statistical Aggregates
- @sum (integer extractor), Computing for Statistical Aggregates
- adding values to statistical aggregates, Computing for Statistical Aggregates
- count (operator), Computing for Statistical Aggregates
- extracting data collected by statistical aggregates, Computing for Statistical Aggregates
- computing for timestamp deltas
- reading values from arrays
- array operations, Reading Values From Arrays
- conditional operators
- conditional statements
- handlers, Conditional Statements
- conditional statements, using arrays in
- array operations, Using Arrays in Conditional Statements
- testing for array membership, Using Arrays in Conditional Statements
- CONFIG_HZ, computing for, Variables
- contended user-space locks (futex contentions), identifying
- examples of SystemTap scripts, Identifying Contended User-Space Locks
- copy fault
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- count operator
- computing for statistical aggregates
- array (operator), Computing for Statistical Aggregates
- counting function calls
- examples of SystemTap scripts, Counting Function Calls Made
- CPU ticks
- examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
- cpu()
- functions, Systemtap Handler/Body
- cross-compiling, Generating Instrumentation for Other Computers
- cross-instrumentation
- advantages of, Generating Instrumentation for Other Computers
- building kernel modules from SystemTap scripts, Generating Instrumentation for Other Computers
- configuration
- host system and target system, Generating Instrumentation for Other Computers
- generating instrumentation from SystemTap scripts, Generating Instrumentation for Other Computers
- host system, Generating Instrumentation for Other Computers
- instrumentation module, Generating Instrumentation for Other Computers
- target kernel, Generating Instrumentation for Other Computers
- target system, Generating Instrumentation for Other Computers
- ctime()
- functions, Systemtap Handler/Body
- ctime(), example of usage
- script examples, Summarizing Disk Read/Write Traffic
- cumulative I/O, tracking
- examples of SystemTap scripts, Track Cumulative IO
- cumulative virtual file system reads, tallying
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
D
- delete operator
- clearing arrays/array elements
- array operations, Clearing/Deleting Arrays and Array Elements
- determining architecture notation, Installing Required Kernel Information Packages
- determining the kernel version, Installing Required Kernel Information Packages
- determining time spent in kernel and user space
- examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
- device I/O, monitoring
- examples of SystemTap scripts, I/O Monitoring (By Device)
- device number of a file (integer format)
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- disk I/O traffic, summarizing
- script examples, Summarizing Disk Read/Write Traffic
- division by 0
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- documentation goals
- Introduction, Documentation Goals
E
- embedded code in unprivileged script
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- empty unique keys
- reading values from arrays
- array operations, Reading Values From Arrays
- end
- Events, Event
- errors
- parse/semantics error, Parse and Semantic Errors
- embedded code in unprivileged script, Parse and Semantic Errors
- expected symbol/array index expression, Parse and Semantic Errors
- grammatical/typographical script error, Parse and Semantic Errors
- guru mode, Parse and Semantic Errors
- invalid values to variables/arrays, Parse and Semantic Errors
- libdwfl failure, Parse and Semantic Errors
- no match for probe point, Parse and Semantic Errors
- non-global arrays, Parse and Semantic Errors
- probe mismatch, Parse and Semantic Errors
- type mismatch for identifier, Parse and Semantic Errors
- unresolved function call, Parse and Semantic Errors
- unresolved target-symbol expression, Parse and Semantic Errors
- unresolved type for identifier, Parse and Semantic Errors
- variable modified during 'foreach', Parse and Semantic Errors
- runtime errors/warnings, Run Time Errors and Warnings
- aggregate element not found, Run Time Errors and Warnings
- aggregation overflow, Run Time Errors and Warnings
- copy fault, Run Time Errors and Warnings
- division by 0, Run Time Errors and Warnings
- MAXACTION exceeded, Run Time Errors and Warnings
- MAXNESTING exceeded, Run Time Errors and Warnings
- number of errors: N, skipped probes: M, Run Time Errors and Warnings
- pointer dereference fault, Run Time Errors and Warnings
- event types
- Understanding How SystemTap Works, Understanding How SystemTap Works
- Events
- asynchronous events, Event
- begin, Event
- end, Event
- examples of synchronous and asynchronous events, Event
- introduction, Event
- kernel.function("function"), Event
- kernel.trace("tracepoint"), Event
- module("module"), Event
- synchronous events, Event
- syscall.system_call, Event
- timer events, Event
- vfs.file_operation, Event
- wildcards, Event
- events and handlers, Understanding How SystemTap Works
- events wildcards, Event
- example
- introduction
- arrays, Associative Arrays
- example of multiple command-line arguments
- examples of SystemTap scripts, Call Graph Tracing
- examples of synchronous and asynchronous events
- Events, Event
- examples of SystemTap scripts, Useful SystemTap Scripts
- call graph tracing, Call Graph Tracing
- CPU ticks, Determining Time Spent in Kernel and User Space
- ctime(), example of usage, Summarizing Disk Read/Write Traffic
- determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
- file device number (integer format), Monitoring Reads and Writes to a File
- futex system call, Identifying Contended User-Space Locks
- identifying contended user-space locks (futex contentions), Identifying Contended User-Space Locks
- if/else conditionals, alternative syntax, Network Profiling
- inode number, Monitoring Reads and Writes to a File
- monitoring changes to file attributes, Monitoring Changes to File Attributes
- monitoring device I/O, I/O Monitoring (By Device)
- monitoring I/O time, Tracking I/O Time For Each File Read or Write
- monitoring incoming TCP connections, Monitoring Incoming TCP Connections
- monitoring polling applications, Monitoring Polling Applications
- monitoring reads and writes to a file, Monitoring Reads and Writes to a File
- monitoring system calls, Tracking Most Frequently Used System Calls
- monitoring system calls (volume per process), Tracking System Call Volume Per Process
- multiple command-line arguments, example of, Call Graph Tracing
- net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
- network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
- process deadlocks (arising from futex contentions), Identifying Contended User-Space Locks
- stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
- stat -c, determining whole device number, I/O Monitoring (By Device)
- summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
- tallying function calls, Counting Function Calls Made
- thread_indent(), sample usage, Call Graph Tracing
- timer.ms(), sample usage, Counting Function Calls Made
- timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
- tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
- tracking cumulative I/O, Track Cumulative IO
- trigger function, Call Graph Tracing
- usrdev2kerndev(), I/O Monitoring (By Device)
- whole device number (usage as a command-line argument), I/O Monitoring (By Device)
- exceeded MAXACTION
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- exceeded MAXNESTING
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- exit()
- functions, Systemtap Handler/Body
- expected symbol/array index expression
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- extracting data collected by statistical aggregates
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
F
- file attributes, monitoring changes to
- examples of SystemTap scripts, Monitoring Changes to File Attributes
- file device number (integer format)
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- file reads/writes, monitoring
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- flight recorder mode, SystemTap Flight Recorder Mode
- file mode, File Flight Recorder
- in-memory mode, In-memory Flight Recorder
- for loops
- conditional statements
- handlers, Conditional Statements
- foreach
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
- format
- introduction
- arrays, Associative Arrays
- format and syntax
- printf(), Systemtap Handler/Body
- SystemTap handler constructs
- handlers, Variables
- SystemTap scripts
- introduction, SystemTap Scripts
- format specifiers
- printf(), Systemtap Handler/Body
- format strings
- printf(), Systemtap Handler/Body
- function call (unresolved)
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- function calls (incoming/outgoing), tracing
- examples of SystemTap scripts, Call Graph Tracing
- function calls, tallying
- examples of SystemTap scripts, Counting Function Calls Made
- functions, Systemtap Handler/Body
- cpu(), Systemtap Handler/Body
- ctime(), Systemtap Handler/Body
- gettimeofday_s(), Systemtap Handler/Body
- pp(), Systemtap Handler/Body
- SystemTap scripts
- introduction, SystemTap Scripts
- target(), Systemtap Handler/Body
- thread_indent(), Systemtap Handler/Body
- tid(), Systemtap Handler/Body
- uid(), Systemtap Handler/Body
- functions (used in handlers)
- exit(), Systemtap Handler/Body
- functions called in network socket code, tracing
- examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
- futex contention, definition
- examples of SystemTap scripts, Identifying Contended User-Space Locks
- futex contentions, identifying
- examples of SystemTap scripts, Identifying Contended User-Space Locks
- futex system call
- examples of SystemTap scripts, Identifying Contended User-Space Locks
G
- gettimeofday_s()
- functions, Systemtap Handler/Body
- global
- SystemTap handler constructs
- handlers, Variables
- goals, documentation
- Introduction, Documentation Goals
- grammatical/typographical script error
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- guru mode
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
H
- handler functions, Systemtap Handler/Body
- handlers
- conditional statements, Conditional Statements
- conditional operators, Conditional Statements
- for loops, Conditional Statements
- if/else, Conditional Statements
- while loops, Conditional Statements
- introduction, Systemtap Handler/Body
- SystemTap handler constructs, Basic SystemTap Handler Constructs
- command-line arguments, Command-Line Arguments
- global, Variables
- syntax and format, Basic SystemTap Handler Constructs
- variables, Variables
- handlers and events, Understanding How SystemTap Works
- SystemTap scripts
- introduction, SystemTap Scripts
- heaviest disk reads/writes, identifying
- script examples, Summarizing Disk Read/Write Traffic
- host system
- cross-instrumentation, Generating Instrumentation for Other Computers
- host system and target system
- cross-instrumentation
- configuration, Generating Instrumentation for Other Computers
I
- I/O monitoring (by device)
- examples of SystemTap scripts, I/O Monitoring (By Device)
- I/O time, monitoring
- examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
- I/O traffic, summarizing
- script examples, Summarizing Disk Read/Write Traffic
- identifier type mismatch
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- identifying contended user-space locks (futex contentions)
- examples of SystemTap scripts, Identifying Contended User-Space Locks
- identifying heaviest disk reads/writes
- script examples, Summarizing Disk Read/Write Traffic
- if/else
- conditional statements
- handlers, Conditional Statements
- if/else conditionals, alternative syntax
- examples of SystemTap scripts, Network Profiling
- if/else statements, using arrays in
- array operations, Using Arrays in Conditional Statements
- incoming TCP connections, monitoring
- examples of SystemTap scripts, Monitoring Incoming TCP Connections
- incoming/outgoing function calls, tracing
- examples of SystemTap scripts, Call Graph Tracing
- incrementing associated values
- array operations, Incrementing Associated Values
- tallying virtual file system reads (VFS reads), Incrementing Associated Values
- index expression
- introduction
- arrays, Associative Arrays
- initial testing, Initial Testing
- inode number
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- Installation
- initial testing, Initial Testing
- kernel information packages, Installing Required Kernel Information Packages
- kernel version, determining the, Installing Required Kernel Information Packages
- required packages, Installing Required Kernel Information Packages
- Setup and Installation, Installation and Setup
- systemtap package, Installing SystemTap
- systemtap-runtime package, Installing SystemTap
- instrumentation module
- cross-instrumentation, Generating Instrumentation for Other Computers
- instrumentation modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
- integer extractors
- computing for statistical aggregates
- array operations, Computing for Statistical Aggregates
- Introduction
- capabilities of SystemTap, SystemTap Capabilities
- documentation goals, Documentation Goals
- goals, documentation, Documentation Goals
- performance monitoring, Introduction
- invalid division
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- invalid values to variables/arrays
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- iterations, processing elements in an array as
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
K
- kernel and user space, determining time spent in
- examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
- kernel information packages, Installing Required Kernel Information Packages
- kernel modules from SystemTap scripts, building, Generating Instrumentation for Other Computers
- kernel version, determining the, Installing Required Kernel Information Packages
- kernel.function("function")
- Events, Event
- kernel.trace("tracepoint")
- Events, Event
- key pairs
- introduction
- arrays, Associative Arrays
L
- libdwfl failure
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- limiting the output of foreach
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
- local variables
- name, Systemtap Handler/Body
- sample usage
M
- MAXACTION exceeded
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- MAXNESTING exceeded
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- membership (in array), testing for
- conditional statements, using arrays in
- array operations, Using Arrays in Conditional Statements
- module("module")
- Events, Event
- monitoring changes to file attributes
- examples of SystemTap scripts, Monitoring Changes to File Attributes
- monitoring cumulative I/O
- examples of SystemTap scripts, Track Cumulative IO
- monitoring device I/O
- examples of SystemTap scripts, I/O Monitoring (By Device)
- monitoring I/O time
- examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
- monitoring incoming TCP connections
- examples of SystemTap scripts, Monitoring Incoming TCP Connections
- monitoring polling applications
- examples of SystemTap scripts, Monitoring Polling Applications
- monitoring reads and writes to a file
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- monitoring system calls
- examples of SystemTap scripts, Tracking Most Frequently Used System Calls
- monitoring system calls (volume per process)
- examples of SystemTap scripts, Tracking System Call Volume Per Process
- multiple array operations within the same probe
- clearing arrays/array elements
- array operations, Clearing/Deleting Arrays and Array Elements
- multiple command-line arguments, example of
- examples of SystemTap scripts, Call Graph Tracing
- multiple elements in an array
- array operations, Processing Multiple Elements in an Array
N
- name
- local variables, Systemtap Handler/Body
- net/socket.c, tracing functions from
- examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
- network profiling
- examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
- network socket code, tracing functions called in
- examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
- network traffic, monitoring
- examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
- no match for probe point
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- non-global arrays
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- number of errors: N, skipped probes: M
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
O
- operations
- assigning associated values
- associating timestamps to process names, Assigning an Associated Value
- associative arrays, Array Operations in SystemTap
- clearing arrays/array elements, Clearing/Deleting Arrays and Array Elements
- delete operator, Clearing/Deleting Arrays and Array Elements
- multiple array operations within the same probe, Clearing/Deleting Arrays and Array Elements
- virtual file system reads (non-cumulative), tallying, Clearing/Deleting Arrays and Array Elements
- computing for statistical aggregates, Computing for Statistical Aggregates
- @avg (integer extractor), Computing for Statistical Aggregates
- @count (integer extractor), Computing for Statistical Aggregates
- @max (integer extractor), Computing for Statistical Aggregates
- @min (integer extractor), Computing for Statistical Aggregates
- @sum (integer extractor), Computing for Statistical Aggregates
- adding values to statistical aggregates, Computing for Statistical Aggregates
- count (operator), Computing for Statistical Aggregates
- extracting data collected by statistical aggregates, Computing for Statistical Aggregates
- conditional statements, using arrays in, Using Arrays in Conditional Statements
- testing for array membership, Using Arrays in Conditional Statements
- deleting arrays and array elements, Clearing/Deleting Arrays and Array Elements
- incrementing associated values, Incrementing Associated Values
- tallying virtual file system reads (VFS reads), Incrementing Associated Values
- multiple elements in an array, Processing Multiple Elements in an Array
- processing multiple elements in an array, Processing Multiple Elements in an Array
- cumulative virtual file system reads, tallying, Processing Multiple Elements in an Array
- foreach, Processing Multiple Elements in an Array
- iterations, processing elements in an array as, Processing Multiple Elements in an Array
- limiting the output of foreach, Processing Multiple Elements in an Array
- ordering the output of foreach, Processing Multiple Elements in an Array
- reading values from arrays, Reading Values From Arrays
- computing for timestamp deltas, Reading Values From Arrays
- empty unique keys, Reading Values From Arrays
- using arrays in simple computations, Reading Values From Arrays
- options, stap
- Usage, Running SystemTap Scripts
- ordering the output of foreach
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
- overflow of aggregation
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
P
- packages required to run SystemTap, Installing Required Kernel Information Packages
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- embedded code in unprivileged script, Parse and Semantic Errors
- expected symbol/array index expression, Parse and Semantic Errors
- grammatical/typographical script error, Parse and Semantic Errors
- guru mode, Parse and Semantic Errors
- invalid values to variables/arrays, Parse and Semantic Errors
- libdwfl failure, Parse and Semantic Errors
- no match for probe point, Parse and Semantic Errors
- non-global arrays, Parse and Semantic Errors
- probe mismatch, Parse and Semantic Errors
- type mismatch for identifier, Parse and Semantic Errors
- unresolved function call, Parse and Semantic Errors
- unresolved target-symbol expression, Parse and Semantic Errors
- unresolved type for identifier, Parse and Semantic Errors
- variable modified during 'foreach', Parse and Semantic Errors
- performance monitoring
- Introduction, Introduction
- pointer dereference fault
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- polling applications, monitoring
- examples of SystemTap scripts, Monitoring Polling Applications
- pp()
- functions, Systemtap Handler/Body
- printf()
- format specifiers, Systemtap Handler/Body
- format strings, Systemtap Handler/Body
- syntax and format, Systemtap Handler/Body
- printing I/O activity (cumulative)
- examples of SystemTap scripts, Track Cumulative IO
- probe mismatch
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- probe point (no match for)
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- probes
- SystemTap scripts
- introduction, SystemTap Scripts
- process deadlocks (arising from futex contentions)
- examples of SystemTap scripts, Identifying Contended User-Space Locks
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
- cumulative virtual file system reads, tallying
- array operations, Processing Multiple Elements in an Array
- foreach
- array operations, Processing Multiple Elements in an Array
- limiting the output of foreach
- array operations, Processing Multiple Elements in an Array
- ordering the output of foreach
- array operations, Processing Multiple Elements in an Array
- profiling the network
- examples of SystemTap scripts, Network Profiling, Monitoring Network Packets Drops in Kernel
R
- reading values from arrays
- array operations, Reading Values From Arrays
- empty unique keys, Reading Values From Arrays
- using arrays in simple computations, Reading Values From Arrays
- computing for timestamp deltas
- array operations, Reading Values From Arrays
- reads/writes to a file, monitoring
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- required packages, Installing Required Kernel Information Packages
- RPMs required to run SystemTap, Installing Required Kernel Information Packages
- running scripts from standard input, Running SystemTap Scripts
- running SystemTap scripts
- Usage, Running SystemTap Scripts
- runtime errors/warnings
- understanding SystemTap errors, Run Time Errors and Warnings
- aggregate element not found, Run Time Errors and Warnings
- aggregation overflow, Run Time Errors and Warnings
- copy fault, Run Time Errors and Warnings
- division by 0, Run Time Errors and Warnings
- MAXACTION exceeded, Run Time Errors and Warnings
- MAXNESTING exceeded, Run Time Errors and Warnings
- number of errors: N, skipped probes: M, Run Time Errors and Warnings
- pointer dereference fault, Run Time Errors and Warnings
S
- script examples
- call graph tracing, Call Graph Tracing
- CPU ticks, Determining Time Spent in Kernel and User Space
- ctime(), example of usage, Summarizing Disk Read/Write Traffic
- determining time spent in kernel and user space, Determining Time Spent in Kernel and User Space
- file device number (integer format), Monitoring Reads and Writes to a File
- futex system call, Identifying Contended User-Space Locks
- identifying contended user-space locks (futex contentions), Identifying Contended User-Space Locks
- if/else conditionals, alternative syntax, Network Profiling
- inode number, Monitoring Reads and Writes to a File
- monitoring changes to file attributes, Monitoring Changes to File Attributes
- monitoring device I/O, I/O Monitoring (By Device)
- monitoring I/O time, Tracking I/O Time For Each File Read or Write
- monitoring incoming TCP connections, Monitoring Incoming TCP Connections
- monitoring polling applications, Monitoring Polling Applications
- monitoring reads and writes to a file, Monitoring Reads and Writes to a File
- monitoring system calls, Tracking Most Frequently Used System Calls
- monitoring system calls (volume per process), Tracking System Call Volume Per Process
- multiple command-line arguments, example of, Call Graph Tracing
- net/socket.c, tracing functions from, Tracing Functions Called in Network Socket Code
- network profiling, Network Profiling, Monitoring Network Packets Drops in Kernel
- process deadlocks (arising from futex contentions), Identifying Contended User-Space Locks
- stat -c, determining file device number (integer format), Monitoring Reads and Writes to a File
- stat -c, determining whole device number, I/O Monitoring (By Device)
- summarizing disk I/O traffic, Summarizing Disk Read/Write Traffic
- tallying function calls, Counting Function Calls Made
- thread_indent(), sample usage, Call Graph Tracing
- timer.ms(), sample usage, Counting Function Calls Made
- timer.s(), sample usage, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
- tracing functions called in network socket code, Tracing Functions Called in Network Socket Code
- tracking cumulative I/O, Track Cumulative IO
- trigger function, Call Graph Tracing
- usrdev2kerndev(), I/O Monitoring (By Device)
- whole device number (usage as a command-line argument), I/O Monitoring (By Device)
- scripts
- introduction, SystemTap Scripts
- components, SystemTap Scripts
- events and handlers, SystemTap Scripts
- format and syntax, SystemTap Scripts
- functions, SystemTap Scripts
- probes, SystemTap Scripts
- statement blocks, SystemTap Scripts
- sessions, SystemTap, Architecture
- Setup and Installation, Installation and Setup
- standard input, running scripts from
- Usage, Running SystemTap Scripts
- stap
- Usage, Running SystemTap Scripts
- stap options, Running SystemTap Scripts
- stapdev
- Usage, Running SystemTap Scripts
- staprun
- Usage, Running SystemTap Scripts
- stapusr
- Usage, Running SystemTap Scripts
- stat -c, determining file device number (integer format)
- examples of SystemTap scripts, Monitoring Reads and Writes to a File
- stat -c, determining whole device number
- examples of SystemTap scripts, I/O Monitoring (By Device)
- statement blocks
- SystemTap scripts
- introduction, SystemTap Scripts
- statistical aggregates
- array operations, Computing for Statistical Aggregates
- summarizing disk I/O traffic
- script examples, Summarizing Disk Read/Write Traffic
- synchronous events
- Events, Event
- syntax
- introduction
- arrays, Associative Arrays
- syntax and format
- printf(), Systemtap Handler/Body
- SystemTap handler constructs
- handlers, Basic SystemTap Handler Constructs
- SystemTap scripts
- introduction, SystemTap Scripts
- syscall.system_call
- Events, Event
- system calls volume (per process), monitoring
- examples of SystemTap scripts, Tracking System Call Volume Per Process
- system calls, monitoring
- examples of SystemTap scripts, Tracking Most Frequently Used System Calls
- SystemTap architecture, Architecture
- SystemTap handlers
- SystemTap handler constructs, Basic SystemTap Handler Constructs
- syntax and format, Basic SystemTap Handler Constructs
- systemtap package, Installing SystemTap
- SystemTap script functions, Systemtap Handler/Body
- SystemTap scripts
- introduction, SystemTap Scripts
- components, SystemTap Scripts
- events and handlers, SystemTap Scripts
- format and syntax, SystemTap Scripts
- functions, SystemTap Scripts
- probes, SystemTap Scripts
- statement blocks, SystemTap Scripts
- useful examples, Useful SystemTap Scripts
- SystemTap scripts, how to run, Running SystemTap Scripts
- SystemTap sessions, Architecture
- SystemTap statements
- conditional statements, Conditional Statements
- conditional operators, Conditional Statements
- for loops, Conditional Statements
- if/else, Conditional Statements
- while loops, Conditional Statements
- SystemTap handler constructs
- command-line arguments, Command-Line Arguments
- global, Variables
- variables, Variables
- systemtap-runtime package, Installing SystemTap
- systemtap-testsuite package
- sample scripts, Useful SystemTap Scripts
T
- tallying function calls
- examples of SystemTap scripts, Counting Function Calls Made
- tallying virtual file system reads (VFS reads)
- incrementing associated values
- array operations, Incrementing Associated Values
- Tapsets
- definition of, Tapsets
- target kernel
- cross-instrumentation, Generating Instrumentation for Other Computers
- target system
- cross-instrumentation, Generating Instrumentation for Other Computers
- target system and host system
- configuration, Generating Instrumentation for Other Computers
- target()
- functions, Systemtap Handler/Body
- target-symbol expression, unresolved
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- TCP connections (incoming), monitoring
- examples of SystemTap scripts, Monitoring Incoming TCP Connections
- testing for array membership
- conditional statements, using arrays in
- array operations, Using Arrays in Conditional Statements
- testing, initial, Initial Testing
- thread_indent()
- functions, Systemtap Handler/Body
- thread_indent(), sample usage
- examples of SystemTap scripts, Call Graph Tracing
- tid()
- functions, Systemtap Handler/Body
- time of I/O
- examples of SystemTap scripts, Tracking I/O Time For Each File Read or Write
- time spent in kernel/user space, determining
- examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
- timer events
- Events, Event
- timer.ms(), sample usage
- examples of SystemTap scripts, Counting Function Calls Made
- timer.s(), sample usage
- examples of SystemTap scripts, Monitoring Polling Applications, Tracking Most Frequently Used System Calls
- timestamp deltas, computing for
- reading values from arrays
- array operations, Reading Values From Arrays
- timestamps, association thereof to process names
- assigning associated values
- array operations, Assigning an Associated Value
- tracepoint, Event, Monitoring Network Packets Drops in Kernel
- tracing call graph
- examples of SystemTap scripts, Call Graph Tracing
- tracing functions called in network socket code
- examples of SystemTap scripts, Tracing Functions Called in Network Socket Code
- tracing incoming/outgoing function calls
- examples of SystemTap scripts, Call Graph Tracing
- tracking cumulative I/O
- examples of SystemTap scripts, Track Cumulative IO
- trigger function
- examples of SystemTap scripts, Call Graph Tracing
- type mismatch for identifier
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- typographical script error
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
U
- uid()
- functions, Systemtap Handler/Body
- uname -m, Installing Required Kernel Information Packages
- uname -r, Installing Required Kernel Information Packages
- Understanding How SystemTap Works, Understanding How SystemTap Works
- architecture, Architecture
- event types, Understanding How SystemTap Works
- events and handlers, Understanding How SystemTap Works
- SystemTap sessions, Architecture
- understanding SystemTap errors
- parse/semantics error, Parse and Semantic Errors
- embedded code in unprivileged script, Parse and Semantic Errors
- expected symbol/array index expression, Parse and Semantic Errors
- grammatical/typographical script error, Parse and Semantic Errors
- guru mode, Parse and Semantic Errors
- invalid values to variables/arrays, Parse and Semantic Errors
- libdwfl failure, Parse and Semantic Errors
- no match for probe point, Parse and Semantic Errors
- non-global arrays, Parse and Semantic Errors
- probe mismatch, Parse and Semantic Errors
- type mismatch for identifier, Parse and Semantic Errors
- unresolved function call, Parse and Semantic Errors
- unresolved target-symbol expression, Parse and Semantic Errors
- unresolved type for identifier, Parse and Semantic Errors
- variable modified during 'foreach', Parse and Semantic Errors
- runtime errors/warnings, Run Time Errors and Warnings
- aggregate element not found, Run Time Errors and Warnings
- aggregation overflow, Run Time Errors and Warnings
- copy fault, Run Time Errors and Warnings
- division by 0, Run Time Errors and Warnings
- MAXACTION exceeded, Run Time Errors and Warnings
- MAXNESTING exceeded, Run Time Errors and Warnings
- number of errors: N, skipped probes: M, Run Time Errors and Warnings
- pointer dereference fault, Run Time Errors and Warnings
- unique keys
- introduction
- arrays, Associative Arrays
- unprivileged script, embedded code in
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- unresolved function call
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- unresolved target-symbol expression
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- unresolved type for identifier
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- unsafe embedded code in unprivileged script
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- Usage
- options, stap, Running SystemTap Scripts
- running SystemTap scripts, Running SystemTap Scripts
- standard input, running scripts from, Running SystemTap Scripts
- stap, Running SystemTap Scripts
- stapdev, Running SystemTap Scripts
- staprun, Running SystemTap Scripts
- stapusr, Running SystemTap Scripts
- useful examples of SystemTap scripts, Useful SystemTap Scripts
- user and kernel space, determining time spent in
- examples of SystemTap scripts, Determining Time Spent in Kernel and User Space
- using arrays in simple computations
- reading values from arrays
- array operations, Reading Values From Arrays
- Using SystemTap, Using SystemTap
- usrdev2kerndev()
- examples of SystemTap scripts, I/O Monitoring (By Device)
V
- values, assignment of
- array operations, Assigning an Associated Value
- variable modified during 'foreach'
- parse/semantics error
- understanding SystemTap errors, Parse and Semantic Errors
- variables
- SystemTap handler constructs
- handlers, Variables
- variables (local)
- name, Systemtap Handler/Body
- sample usage
- VFS reads, tallying of
- incrementing associated values
- array operations, Incrementing Associated Values
- vfs.file_operation
- Events, Event
- virtual file system reads (cumulative), tallying
- processing multiple elements in an array
- array operations, Processing Multiple Elements in an Array
- virtual file system reads (non-cumulative), tallying
- clearing arrays/array elements
- array operations, Clearing/Deleting Arrays and Array Elements
W
- while loops
- conditional statements
- handlers, Conditional Statements
- whole device number (usage as a command-line argument)
- examples of SystemTap scripts, I/O Monitoring (By Device)
- wildcards in events, Event
- writes/reads to a file, monitoring
- examples of SystemTap scripts, Monitoring Reads and Writes to a File