Chapter 4. Useful SystemTap Scripts
This chapter enumerates several SystemTap scripts you can use to monitor and investigate different subsystems. All of these scripts are available at
/usr/share/systemtap/testsuite/systemtap.examples/ once you install the systemtap-testsuite RPM.
4.1. Network
The following sections showcase scripts that trace network-related functions and build a profile of network activity.
4.1.1. Network Profiling
This section describes how to profile network activity. nettop.stp provides a glimpse into how much network traffic each process is generating on a machine.
nettop.stp
#! /usr/bin/env stap
global ifxmit, ifrecv
global ifmerged
probe netdev.transmit
{
ifxmit[pid(), dev_name, execname(), uid()] <<< length
}
probe netdev.receive
{
ifrecv[pid(), dev_name, execname(), uid()] <<< length
}
function print_activity()
{
printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n",
"PID", "UID", "DEV", "XMIT_PK", "RECV_PK",
"XMIT_KB", "RECV_KB", "COMMAND")
foreach ([pid, dev, exec, uid] in ifrecv) {
ifmerged[pid, dev, exec, uid] += @count(ifrecv[pid,dev,exec,uid]);
}
foreach ([pid, dev, exec, uid] in ifxmit) {
ifmerged[pid, dev, exec, uid] += @count(ifxmit[pid,dev,exec,uid]);
}
foreach ([pid, dev, exec, uid] in ifmerged-) {
n_xmit = @count(ifxmit[pid, dev, exec, uid])
n_recv = @count(ifrecv[pid, dev, exec, uid])
printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n",
pid, uid, dev, n_xmit, n_recv,
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0,
n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0,
exec)
}
print("\n")
delete ifxmit
delete ifrecv
delete ifmerged
}
probe timer.ms(5000), end, error
{
print_activity()
}
Note that
function print_activity() uses the following expressions:
n_xmit ? @sum(ifxmit[pid, dev, exec, uid])/1024 : 0 n_recv ? @sum(ifrecv[pid, dev, exec, uid])/1024 : 0
These expressions are if/else conditionals. The first statement is simply a more concise way of writing the following psuedo code:
if n_recv != 0 then @sum(ifrecv[pid, dev, exec, uid])/1024 else 0
nettop.stp tracks which processes are generating network traffic on the system, and provides the following information about each process:
PID— the ID of the listed process.UID— user ID. A user ID of0refers to the root user.DEV— which ethernet device the process used to send / receive data (e.g. eth0, eth1)XMIT_PK— number of packets transmitted by the processRECV_PK— number of packets received by the processXMIT_KB— amount of data sent by the process, in kilobytesRECV_KB— amount of data received by the service, in kilobytes
nettop.stp provides network profile sampling every 5 seconds. You can change this setting by editing
probe timer.ms(5000) accordingly. Example 4.1, “nettop.stp Sample Output” contains an excerpt of the output from nettop.stp over a 20-second period:
Example 4.1. nettop.stp Sample Output
[...]
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 0 5 0 0 swapper
11178 0 eth0 2 0 0 0 synergyc
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
2886 4 eth0 79 0 5 0 cups-polld
11362 0 eth0 0 61 0 5 firefox
0 0 eth0 3 32 0 3 swapper
2886 4 lo 4 4 0 0 cups-polld
11178 0 eth0 3 0 0 0 synergyc
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 0 6 0 0 swapper
2886 4 lo 2 2 0 0 cups-polld
11178 0 eth0 3 0 0 0 synergyc
3611 0 eth0 0 1 0 0 Xorg
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
0 0 eth0 3 42 0 2 swapper
11178 0 eth0 43 1 3 0 synergyc
11362 0 eth0 0 7 0 0 firefox
3897 0 eth0 0 1 0 0 multiload-apple
[...]
4.1.2. Tracing Functions Called in Network Socket Code
This section describes how to trace functions called from the kernel's
net/socket.c file. This task helps you identify, in finer detail, how each process interacts with the network at the kernel level.
socket-trace.stp
#! /usr/bin/env stap
probe kernel.function("*@net/socket.c").call {
printf ("%s -> %s\n", thread_indent(1), probefunc())
}
probe kernel.function("*@net/socket.c").return {
printf ("%s <- %s\n", thread_indent(-1), probefunc())
}
socket-trace.stp is identical to Example 3.6, “thread_indent.stp”, which was earlier used in SystemTap Functions to illustrate how
thread_indent() works.
Example 4.2. socket-trace.stp Sample Output
[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 gnome-terminal(11106): -> sock_poll 5 gnome-terminal(11106): <- sock_poll 0 scim-bridge(3883): -> sock_poll 3 scim-bridge(3883): <- sock_poll 0 scim-bridge(3883): -> sys_socketcall 4 scim-bridge(3883): -> sys_recv 8 scim-bridge(3883): -> sys_recvfrom 12 scim-bridge(3883):-> sock_from_file 16 scim-bridge(3883):<- sock_from_file 20 scim-bridge(3883):-> sock_recvmsg 24 scim-bridge(3883):<- sock_recvmsg 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall [...]
Example 4.2, “socket-trace.stp Sample Output” contains a 3-second excerpt of the output for socket-trace.stp. For more information about the output of this script as provided by
thread_indent(), refer to SystemTap Functions Example 3.6, “thread_indent.stp”.
4.1.3. Monitoring Incoming TCP Connections
This section illustrates how to monitor incoming TCP connections. This task is useful in identifying any unauthorized, suspicious, or otherwise unwanted network access requests in real time.
tcp_connections.stp
#! /usr/bin/env stap
probe begin {
printf("%6s %16s %6s %6s %16s\n",
"UID", "CMD", "PID", "PORT", "IP_SOURCE")
}
probe kernel.function("tcp_accept").return?,
kernel.function("inet_csk_accept").return? {
sock = $return
if (sock != 0)
printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(),
inet_get_local_port(sock), inet_get_ip_source(sock))
}
While tcp_connections.stp is running, it will print out the following information about any incoming TCP connections accepted by the system in real time:
- Current
UID CMD- the command accepting the connectionPIDof the command- Port used by the connection
- IP address from which the TCP connection originated
Example 4.3. tcp_connections.stp Sample Output
UID CMD PID PORT IP_SOURCE 0 sshd 3165 22 10.64.0.227 0 sshd 3165 22 10.64.0.227
