SupportConsoleDevelopersStart a trialContact

Chapter 43. Network tracing using the BPF compiler collection

BPF Compiler Collection (BCC) is a library, which facilitates the creation of the extended Berkeley Packet Filter (eBPF) programs. The main utility of eBPF programs is analyzing the operating system performance and network performance without experiencing overhead or security issues.

BCC removes the need for users to know deep technical details of eBPF, and provides many out-of-the-box starting points, such as the bcc-tools package with pre-created eBPF programs.

Note

The eBPF programs are triggered on events, such as disk I/O, TCP connections, and process creations. It is unlikely that the programs should cause the kernel to crash, loop or become unresponsive because they run in a safe virtual machine in the kernel.

43.1. Installing the bcc-tools package

Install the bcc-tools package, which also installs the BPF Compiler Collection (BCC) library as a dependency.

Procedure

  • Install bcc-tools. 

    # dnf install bcc-tools

    The BCC tools are installed in the /usr/share/bcc/tools/ directory.

Verification

  • Inspect the installed tools: 

    # ls -l /usr/share/bcc/tools/
...
-rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
-rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
-rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
-rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
-rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
-rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
-rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
...

    The doc directory in the listing provides documentation for each tool.

43.2. Displaying TCP connections added to the Kernel’s accept queue

After the kernel receives the ACK packet in a TCP 3-way handshake, the kernel moves the connection from the SYN queue to the accept queue after the connection’s state changes to ESTABLISHED. Therefore, only successful TCP connections are visible in this queue.

The tcpaccept utility uses eBPF features to display all connections the kernel adds to the accept queue. The utility is lightweight because it traces the accept() function of the kernel instead of capturing packets and filtering them. For example, use tcpaccept for general troubleshooting to display new connections the server has accepted.

Procedure

  1. Enter the following command to start the tracing the kernel accept queue: 

    # /usr/share/bcc/tools/tcpaccept
PID   COMM      IP RADDR         RPORT  LADDR    LPORT
843   sshd      4  192.0.2.17    50598  192.0.2.1  22
1107  ns-slapd  4  198.51.100.6  38772  192.0.2.1  389
1107  ns-slapd  4  203.0.113.85  38774  192.0.2.1  389
...

    Each time the kernel accepts a connection, tcpaccept displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpaccept(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpaccept_example.txt file

43.3. Tracing outgoing TCP connection attempts

The tcpconnect utility uses eBPF features to trace outgoing TCP connection attempts. The output of the utility also includes connections that failed.

The tcpconnect utility is lightweight because it traces, for example, the connect() function of the kernel instead of capturing packets and filtering them.

Procedure

  1. Enter the following command to start the tracing process that displays all outgoing connections: 

    # /usr/share/bcc/tools/tcpconnect
PID    COMM         IP SADDR      DADDR          DPORT
31346  curl         4  192.0.2.1  198.51.100.16  80
31348  telnet       4  192.0.2.1  203.0.113.231  23
31361  isc-worker00 4  192.0.2.1  192.0.2.254    53
...

    Each time the kernel processes an outgoing connection, tcpconnect displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpconnect(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpconnect_example.txt file

43.4. Measuring the latency of outgoing TCP connections

The TCP connection latency is the time taken to establish a connection. This typically involves the kernel TCP/IP processing and network round trip time, and not the application runtime.

The tcpconnlat utility uses eBPF features to measure the time between a sent SYN packet and the received response packet.

Procedure

  1. Start measuring the latency of outgoing connections: 

    # /usr/share/bcc/tools/tcpconnlat
PID    COMM         IP SADDR      DADDR          DPORT LAT(ms)
32151  isc-worker00 4  192.0.2.1  192.0.2.254    53    0.60
32155  ssh          4  192.0.2.1  203.0.113.190  22    26.34
32319  curl         4  192.0.2.1  198.51.100.59  443   188.96
...

    Each time the kernel processes an outgoing connection, tcpconnlat displays the details of the connection after the kernel receives the response packet.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpconnlat(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpconnlat_example.txt file

43.5. Displaying details about TCP packets and segments that were dropped by the kernel

The tcpdrop utility enables administrators to display details about TCP packets and segments that were dropped by the kernel. Use this utility to debug high rates of dropped packets that can cause the remote system to send timer-based retransmits. High rates of dropped packets and segments can impact the performance of a server.

Instead of capturing and filtering packets, which is resource-intensive, the tcpdrop utility uses eBPF features to retrieve the information directly from the kernel.

Procedure

  1. Enter the following command to start displaying details about dropped TCP packets and segments: 

    # /usr/share/bcc/tools/tcpdrop
TIME     PID    IP SADDR:SPORT       > DADDR:DPORT   STATE (FLAGS)
13:28:39 32253  4  192.0.2.85:51616  > 192.0.2.1:22  CLOSE_WAIT (FIN|ACK)
	b'tcp_drop+0x1'
	b'tcp_data_queue+0x2b9'
	...

13:28:39 1      4  192.0.2.85:51616  > 192.0.2.1:22   CLOSE (ACK)
	b'tcp_drop+0x1'
	b'tcp_rcv_state_process+0xe2'
	...

    Each time the kernel drops TCP packets and segments, tcpdrop displays the details of the connection, including the kernel stack trace that led to the dropped package.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpdrop(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpdrop_example.txt file

43.6. Tracing TCP sessions

The tcplife utility uses eBPF to trace TCP sessions that open and close, and prints a line of output to summarize each one. Administrators can use tcplife to identify connections and the amount of transferred traffic.

For example, you can display connections to port 22 (SSH) to retrieve the following information:

  • The local process ID (PID)
  • The local process name
  • The local IP address and port number
  • The remote IP address and port number
  • The amount of received and transmitted traffic in KB.
  • The time in milliseconds the connection was active

Procedure

  1. Enter the following command to start the tracing of connections to the local port 22: 

    # /usr/share/bcc/tools/tcplife -L 22
PID   COMM    LADDR      LPORT RADDR       RPORT TX_KB  RX_KB      MS
19392 sshd    192.0.2.1  22    192.0.2.17  43892    53     52 6681.95
19431 sshd    192.0.2.1  22    192.0.2.245 43902    81 249381 7585.09
19487 sshd    192.0.2.1  22    192.0.2.121 43970  6998     7 16740.35
...

    Each time a connection is closed, tcplife displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcplife(8) man page on your system
  • /usr/share/bcc/tools/doc/tcplife_example.txt file

43.7. Tracing TCP retransmissions

The tcpretrans utility displays details about TCP retransmissions, such as the local and remote IP address and port number, as well as the TCP state at the time of the retransmissions.

The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start displaying TCP retransmission details: 

    # /usr/share/bcc/tools/tcpretrans
TIME     PID  IP LADDR:LPORT   T> RADDR:RPORT         STATE
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:45:43 0    4  192.0.2.1:22  R> 198.51.100.0:17634  ESTABLISHED
...

    Each time the kernel calls the TCP retransmit function, tcpretrans displays the details of the connection.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpretrans(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpretrans_example.txt file

43.8. Displaying TCP state change information

During a TCP session, the TCP state changes. The tcpstates utility uses eBPF functions to trace these state changes, and prints details including the duration in each state. For example, use tcpstates to identify if connections spend too much time in the initialization state.

Procedure

  1. Use the following command to start tracing TCP state changes: 

    # /usr/share/bcc/tools/tcpstates
SKADDR           C-PID C-COMM     LADDR     LPORT RADDR       RPORT OLDSTATE    -> NEWSTATE    MS
ffff9cd377b3af80 0     swapper/1  0.0.0.0   22    0.0.0.0     0     LISTEN      -> SYN_RECV    0.000
ffff9cd377b3af80 0     swapper/1  192.0.2.1 22    192.0.2.45  53152 SYN_RECV    -> ESTABLISHED 0.067
ffff9cd377b3af80 818   sssd_nss   192.0.2.1 22    192.0.2.45  53152 ESTABLISHED -> CLOSE_WAIT  65636.773
ffff9cd377b3af80 1432  sshd       192.0.2.1 22    192.0.2.45  53152 CLOSE_WAIT  -> LAST_ACK    24.409
ffff9cd377b3af80 1267  pulseaudio 192.0.2.1 22    192.0.2.45  53152 LAST_ACK    -> CLOSE       0.376
...

    Each time a connection changes its state, tcpstates displays a new line with updated connection details.

    If multiple connections change their state at the same time, use the socket address in the first column (SKADDR) to determine which entries belong to the same connection.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpstates(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpstates_example.txt file

43.9. Summarizing and aggregating TCP traffic sent to specific subnets

The tcpsubnet utility summarizes and aggregates IPv4 TCP traffic that the local host sends to subnets and displays the output on a fixed interval. The utility uses eBPF features to collect and summarize the data to reduce the overhead.

By default, tcpsubnet summarizes traffic for the following subnets:

  • 127.0.0.1/32
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.0.2.0/24/16
  • 0.0.0.0/0

Note that the last subnet (0.0.0.0/0) is a catch-all option. The tcpsubnet utility counts all traffic for subnets different than the first four in this catch-all entry.

Follow the procedure to count the traffic for the 192.0.2.0/24 and 198.51.100.0/24 subnets. Traffic to other subnets will be tracked in the 0.0.0.0/0 catch-all subnet entry.

Procedure

  1. Start monitoring the amount of traffic send to the 192.0.2.0/24, 198.51.100.0/24, and other subnets: 

    # /usr/share/bcc/tools/tcpsubnet 192.0.2.0/24,198.51.100.0/24,0.0.0.0/0
Tracing... Output every 1 secs. Hit Ctrl-C to end
[02/21/20 10:04:50]
192.0.2.0/24           856
198.51.100.0/24       7467
[02/21/20 10:04:51]
192.0.2.0/24          1200
198.51.100.0/24       8763
0.0.0.0/0              673
...

    This command displays the traffic in bytes for the specified subnets once per second.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcpsubnet(8) man page on your system
  • /usr/share/bcc/tools/doc/tcpsubnet.txt file

43.10. Displaying the network throughput by IP address and port

The tcptop utility displays TCP traffic the host sends and receives in kilobytes. The report automatically refreshes and contains only active TCP connections. The utility uses eBPF features and, therefore, has only a very low overhead.

Procedure

  1. To monitor the sent and received traffic, enter: 

    # /usr/share/bcc/tools/tcptop
13:46:29 loadavg: 0.10 0.03 0.01 1/215 3875

PID    COMM         LADDR           RADDR              RX_KB   TX_KB
3853   3853         192.0.2.1:22    192.0.2.165:41838  32     102626
1285   sshd         192.0.2.1:22    192.0.2.45:39240   0           0
...

    The output of the command includes only active TCP connections. If the local or remote system closes a connection, the connection is no longer visible in the output.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcptop(8) man page on your system
  • /usr/share/bcc/tools/doc/tcptop.txt file

43.11. Tracing established TCP connections

The tcptracer utility traces the kernel functions that connect, accept, and close TCP connections. The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start the tracing process: 

    # /usr/share/bcc/tools/tcptracer
Tracing TCP established connections. Ctrl-C to end.
T  PID    COMM        IP SADDR        DADDR       SPORT  DPORT
A  1088   ns-slapd    4  192.0.2.153  192.0.2.1   0      65535
A  845    sshd        4  192.0.2.1    192.0.2.67  22     42302
X  4502   sshd        4  192.0.2.1    192.0.2.67  22     42302
...

    Each time the kernel connects, accepts, or closes a connection, tcptracer displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • tcptracer(8) man page on your system
  • /usr/share/bcc/tools/doc/tcptracer_example.txt file

43.12. Tracing IPv4 and IPv6 listen attempts

The solisten utility traces all IPv4 and IPv6 listen attempts. It traces the listen attempts including that ultimately fail or the listening program that does not accept the connection. The utility traces function that the kernel calls when a program wants to listen for TCP connections.

Procedure

  1. Enter the following command to start the tracing process that displays all listen TCP attempts: 

    # /usr/share/bcc/tools/solisten
PID    COMM           PROTO         BACKLOG     PORT     ADDR
3643   nc             TCPv4         1           4242     0.0.0.0
3659   nc             TCPv6         1           4242     2001:db8:1::1
4221   redis-server   TCPv6         128         6379     ::
4221   redis-server   TCPv4         128         6379     0.0.0.0
....
  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • solisten(9) man page on your system
  • /usr/share/bcc/tools/doc/solisten_example.txt file

43.13. Summarizing the service time of soft interrupts

The softirqs utility summarizes the time spent servicing soft interrupts (soft IRQs) and shows this time as either totals or histogram distributions. This utility uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism.

Procedure

  1. Enter the following command to start the tracing soft irq event time: 

    # /usr/share/bcc/tools/softirqs
Tracing soft irq event time... Hit Ctrl-C to end.
^C
SOFTIRQ          TOTAL_usecs
tasklet                  166
block                   9152
net_rx                 12829
rcu                    53140
sched                 182360
timer                 306256
  2. Press Ctrl+C to stop the tracing process.

Additional resources

  • softirqs(8) and mpstat(1) man pages on your system
  • /usr/share/bcc/tools/doc/softirqs_example.txt file

43.14. Summarizing packets size and count on a network interface

The netqtop utility displays statistics about the attributes of received (RX) and transmitted (TX) packets on each network queue of a particular network interface. The statistics include:

  • Bytes per second (BPS)
  • Packets per second (PPS)
  • The average packet size
  • Total number of packets

To generate these statistics, netqtop traces the kernel functions that perform events of transmitted packets net_dev_start_xmit and received packets netif_receive_skb.

Procedure

  1. Display the number of packets within the range of bytes size of the time interval of 2 seconds: 

    # /usr/share/bcc/tools/netqtop -n enp1s0 -i 2

Fri Jan 31 18:08:55 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------
Fri Jan 31 18:08:57 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------
  2. Press Ctrl+C to stop netqtop.

Additional resources

  • netqtop(8) man page on your system
  • /usr/share/bcc/tools/doc/netqtop_example.txt
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.