Chapter 39. Monitoring network activity with SystemTap
You can use helpful example SystemTap scripts available in the /usr/share/systemtap/testsuite/systemtap.examples/ directory, upon installing the systemtap-testsuite package, to monitor and investigate the network activity of your system.
39.1. Profiling network activity with SystemTap
You can use the nettop.stp example SystemTap script to profile network activity. The script tracks which processes are generating network traffic on the system, and provides the following information about each process:
- PID
- The ID of the listed process.
- UID
- User ID. A user ID of 0 refers to the root user.
- DEV
- Which ethernet device the process used to send or receive data (for example, eth0, eth1).
- XMIT_PK
- The number of packets transmitted by the process.
- RECV_PK
- The number of packets received by the process.
- XMIT_KB
- The amount of data sent by the process, in kilobytes.
- RECV_KB
- The amount of data received by the service, in kilobytes.
Prerequisites
- You have installed SystemTap as described in Installing SystemTap.
Procedure
Run the
nettop.stpscript:# stap --example nettop.stp
The
nettop.stpscript provides network profile sampling every 5 seconds.Output of the
nettop.stpscript looks similar to the following:[...] PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 5 0 0 swapper 11178 0 eth0 2 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 2886 4 eth0 79 0 5 0 cups-polld 11362 0 eth0 0 61 0 5 firefox 0 0 eth0 3 32 0 3 swapper 2886 4 lo 4 4 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 0 6 0 0 swapper 2886 4 lo 2 2 0 0 cups-polld 11178 0 eth0 3 0 0 0 synergyc 3611 0 eth0 0 1 0 0 Xorg PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND 0 0 eth0 3 42 0 2 swapper 11178 0 eth0 43 1 3 0 synergyc 11362 0 eth0 0 7 0 0 firefox 3897 0 eth0 0 1 0 0 multiload-apple
39.2. Tracing functions called in network socket code with SystemTap
You can use the socket-trace.stp example SystemTap script to trace functions called from the kernel’s net/socket.c file. This helps you identify, in finer detail, how each process interacts with the network at the kernel level.
Prerequisites
- You have installed SystemTap as described in Installing SystemTap.
Procedure
Run the
socket-trace.stpscript:# stap --example socket-trace.stp
A 3-second excerpt of the output of the
socket-trace.stpscript looks similar to the following:[...] 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 Xorg(3611): -> sock_poll 3 Xorg(3611): <- sock_poll 0 gnome-terminal(11106): -> sock_poll 5 gnome-terminal(11106): <- sock_poll 0 scim-bridge(3883): -> sock_poll 3 scim-bridge(3883): <- sock_poll 0 scim-bridge(3883): -> sys_socketcall 4 scim-bridge(3883): -> sys_recv 8 scim-bridge(3883): -> sys_recvfrom 12 scim-bridge(3883):-> sock_from_file 16 scim-bridge(3883):<- sock_from_file 20 scim-bridge(3883):-> sock_recvmsg 24 scim-bridge(3883):<- sock_recvmsg 28 scim-bridge(3883): <- sys_recvfrom 31 scim-bridge(3883): <- sys_recv 35 scim-bridge(3883): <- sys_socketcall [...]
39.3. Monitoring network packet drops with SystemTap
The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb"), which tracks where packets are discarded.
The dropwatch.stp SystemTap script uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes what locations discard packets in every 5-second interval.
Prerequisites
- You have installed SystemTap as described in Installing SystemTap.
Procedure
Run the
dropwatch.stpscript:# stap --example dropwatch.stp
Running the
dropwatch.stpscript for 15 seconds results in output similar to the following:Monitoring for dropped packets 51 packets dropped at location 0xffffffff8024cd0f 2 packets dropped at location 0xffffffff8044b472 51 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 97 packets dropped at location 0xffffffff8024cd0f 1 packets dropped at location 0xffffffff8044b472 Stopping dropped packet monitor
NoteTo make the location of packet drops more meaningful, see the
/boot/System.map-$(uname -r)file. This file lists the starting addresses for each function, enabling you to map the addresses in the output of thedropwatch.stpscript to a specific function name. Given the following snippet of the/boot/System.map-$(uname -r)file, the address0xffffffff8024cd0fmaps to the functionunix_stream_recvmsgand the address0xffffffff8044b472maps to the functionarp_rcv:[...] ffffffff8024c5cd T unlock_new_inode ffffffff8024c5da t unix_stream_sendmsg ffffffff8024c920 t unix_stream_recvmsg ffffffff8024cea1 t udp_v4_lookup_longway [...] ffffffff8044addc t arp_process ffffffff8044b360 t arp_rcv ffffffff8044b487 t parp_redo ffffffff8044b48c t arp_solicit [...]
