Red Hat Summit Red Hat SummitSuporteConsoleDesenvolvedoresComeçar um testeContato

Este conteúdo não está disponível no idioma selecionado.

Chapter 45. Network tracing using the BPF compiler collection

BPF Compiler Collection (BCC) is a library, which facilitates the creation of the extended Berkeley Packet Filter (eBPF) programs. The main utility of eBPF programs is analyzing the operating system performance and network performance without experiencing overhead or security issues.

BCC removes the need for users to know deep technical details of eBPF, and provides many out-of-the-box starting points, such as the bcc-tools package with pre-created eBPF programs.

Note

The eBPF programs are triggered on events, such as disk I/O, TCP connections, and process creations. It is unlikely that the programs should cause the kernel to crash, loop or become unresponsive because they run in a safe virtual machine in the kernel.

45.1. Installing the bcc-tools package
Copiar o link

Install the bcc-tools package, which also installs the BPF Compiler Collection (BCC) library as a dependency.

Procedure

  • Install bcc-tools. 

    # yum install bcc-tools
    Copy to Clipboard Toggle word wrap

    The BCC tools are installed in the /usr/share/bcc/tools/ directory.

Verification

  • Inspect the installed tools: 

    # ls -l /usr/share/bcc/tools/
...
-rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
-rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
-rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
-rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
-rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
-rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
-rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
...
    Copy to Clipboard Toggle word wrap

    The doc directory in the listing provides documentation for each tool.

45.2. Displaying TCP connections added to the Kernel’s accept queue
Copiar o link

By using the tcpaccept utility, you can monitor newly accepted connections by tracing the kernel’s accept() function.

After the kernel receives the ACK packet in a TCP 3-way handshake, the kernel moves the connection from the SYN queue to the accept queue after the connection’s state changes to ESTABLISHED. Therefore, only successful TCP connections are visible in this queue.

The tcpaccept utility uses eBPF features to display all connections the kernel adds to the accept queue. The utility is lightweight because it traces the accept() function of the kernel instead of capturing packets and filtering them. For example, use tcpaccept for general troubleshooting to display new connections the server has accepted.

Procedure

  1. Enter the following command to start the tracing the kernel accept queue: 

    # /usr/share/bcc/tools/tcpaccept
PID   COMM      IP RADDR         RPORT  LADDR    LPORT
843   sshd      4  192.0.2.17    50598  192.0.2.1  22
1107  ns-slapd  4  198.51.100.6  38772  192.0.2.1  389
1107  ns-slapd  4  203.0.113.85  38774  192.0.2.1  389
...
    Copy to Clipboard Toggle word wrap

    Each time the kernel accepts a connection, tcpaccept displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

45.3. Tracing outgoing TCP connection attempts
Copiar o link

The tcpconnect utility uses eBPF features to trace outgoing TCP connection attempts. The output of the utility also includes connections that failed.

The tcpconnect utility is lightweight because it traces, for example, the connect() function of the kernel instead of capturing packets and filtering them.

Procedure

  1. Enter the following command to start the tracing process that displays all outgoing connections: 

    # /usr/share/bcc/tools/tcpconnect
PID    COMM         IP SADDR      DADDR          DPORT
31346  curl         4  192.0.2.1  198.51.100.16  80
31348  telnet       4  192.0.2.1  203.0.113.231  23
31361  isc-worker00 4  192.0.2.1  192.0.2.254    53
...
    Copy to Clipboard Toggle word wrap

    Each time the kernel processes an outgoing connection, tcpconnect displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

45.4. Measuring the latency of outgoing TCP connections
Copiar o link

The tcpconnlat utility uses eBPF features to measure the time between a sent SYN packet and the received response packet.

The TCP connection latency is the time taken to establish a connection. This typically involves the kernel TCP/IP processing and network round trip time, and not the application runtime.

Procedure

  1. Start measuring the latency of outgoing connections: 

    # /usr/share/bcc/tools/tcpconnlat
PID    COMM         IP SADDR      DADDR          DPORT LAT(ms)
32151  isc-worker00 4  192.0.2.1  192.0.2.254    53    0.60
32155  ssh          4  192.0.2.1  203.0.113.190  22    26.34
32319  curl         4  192.0.2.1  198.51.100.59  443   188.96
...
    Copy to Clipboard Toggle word wrap

    Each time the kernel processes an outgoing connection, tcpconnlat displays the details of the connection after the kernel receives the response packet.

  2. Press Ctrl+C to stop the tracing process.

The tcpdrop utility enables administrators to display details about TCP packets and segments that were dropped by the kernel. Use this utility to debug high rates of dropped packets that can cause the remote system to send timer-based retransmits.

High rates of dropped packets and segments can impact the performance of a server. Instead of capturing and filtering packets, which is resource-intensive, the tcpdrop utility uses eBPF features to retrieve the information directly from the kernel.

Procedure

  1. Enter the following command to start displaying details about dropped TCP packets and segments: 

    # /usr/share/bcc/tools/tcpdrop
TIME     PID    IP SADDR:SPORT       > DADDR:DPORT   STATE (FLAGS)
13:28:39 32253  4  192.0.2.85:51616  > 192.0.2.1:22  CLOSE_WAIT (FIN|ACK)
	b'tcp_drop+0x1'
	b'tcp_data_queue+0x2b9'
	...

13:28:39 1      4  192.0.2.85:51616  > 192.0.2.1:22   CLOSE (ACK)
	b'tcp_drop+0x1'
	b'tcp_rcv_state_process+0xe2'
	...
    Copy to Clipboard Toggle word wrap

    Each time the kernel drops TCP packets and segments, tcpdrop displays the details of the connection, including the kernel stack trace that led to the dropped package.

  2. Press Ctrl+C to stop the tracing process.

45.6. Tracing TCP sessions
Copiar o link

The tcplife utility uses eBPF to trace TCP sessions that open and close, and prints a line of output to summarize each one. Administrators can use tcplife to identify connections and the amount of transferred traffic.

For example, you can display connections to port 22 (SSH) to retrieve the following information:

  • The local process ID (PID)
  • The local process name
  • The local IP address and port number
  • The remote IP address and port number
  • The amount of received and transmitted traffic in KB.
  • The time in milliseconds the connection was active

Procedure

  1. Enter the following command to start the tracing of connections to the local port 22: 

    # /usr/share/bcc/tools/tcplife -L 22
PID   COMM    LADDR      LPORT RADDR       RPORT TX_KB  RX_KB      MS
19392 sshd    192.0.2.1  22    192.0.2.17  43892    53     52 6681.95
19431 sshd    192.0.2.1  22    192.0.2.245 43902    81 249381 7585.09
19487 sshd    192.0.2.1  22    192.0.2.121 43970  6998     7 16740.35
...
    Copy to Clipboard Toggle word wrap

    Each time a connection is closed, tcplife displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

45.7. Tracing TCP retransmissions
Copiar o link

The tcpretrans utility displays details about TCP retransmissions, such as the local and remote IP address and port number, as well as the TCP state at the time of the retransmissions.

The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start displaying TCP retransmission details: 

    # /usr/share/bcc/tools/tcpretrans
TIME     PID  IP LADDR:LPORT   T> RADDR:RPORT         STATE
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:45:43 0    4  192.0.2.1:22  R> 198.51.100.0:17634  ESTABLISHED
...
    Copy to Clipboard Toggle word wrap

    Each time the kernel calls the TCP retransmit function, tcpretrans displays the details of the connection.

  2. Press Ctrl+C to stop the tracing process.

45.8. Displaying TCP state change information
Copiar o link

During a TCP session, the TCP state changes. The tcpstates utility uses eBPF functions to trace these state changes, and prints details including the duration in each state. For example, use tcpstates to identify if connections spend too much time in the initialization state.

Procedure

  1. Use the following command to start tracing TCP state changes: 

    # /usr/share/bcc/tools/tcpstates
SKADDR           C-PID C-COMM     LADDR     LPORT RADDR       RPORT OLDSTATE    -> NEWSTATE    MS
ffff9cd377b3af80 0     swapper/1  0.0.0.0   22    0.0.0.0     0     LISTEN      -> SYN_RECV    0.000
ffff9cd377b3af80 0     swapper/1  192.0.2.1 22    192.0.2.45  53152 SYN_RECV    -> ESTABLISHED 0.067
ffff9cd377b3af80 818   sssd_nss   192.0.2.1 22    192.0.2.45  53152 ESTABLISHED -> CLOSE_WAIT  65636.773
ffff9cd377b3af80 1432  sshd       192.0.2.1 22    192.0.2.45  53152 CLOSE_WAIT  -> LAST_ACK    24.409
ffff9cd377b3af80 1267  pulseaudio 192.0.2.1 22    192.0.2.45  53152 LAST_ACK    -> CLOSE       0.376
...
    Copy to Clipboard Toggle word wrap

    Each time a connection changes its state, tcpstates displays a new line with updated connection details.

    If multiple connections change their state at the same time, use the socket address in the first column (SKADDR) to determine which entries belong to the same connection.

  2. Press Ctrl+C to stop the tracing process.

45.9. Summarizing and aggregating TCP traffic sent to specific subnets
Copiar o link

The tcpsubnet utility summarizes and aggregates IPv4 TCP traffic that the local host sends to subnets and displays the output on a fixed interval. The utility uses eBPF features to collect and summarize the data to reduce the overhead.

By default, tcpsubnet summarizes traffic for the following subnets:

  • 127.0.0.1/32
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.0.2.0/24/16
  • 0.0.0.0/0

Note that the last subnet (0.0.0.0/0) is a catch-all option. The tcpsubnet utility counts all traffic for subnets different than the first four in this catch-all entry.

Follow the procedure to count the traffic for the 192.0.2.0/24 and 198.51.100.0/24 subnets. Traffic to other subnets will be tracked in the 0.0.0.0/0 catch-all subnet entry.

Procedure

  1. Start monitoring the amount of traffic sent to the 192.0.2.0/24, 198.51.100.0/24, and other subnets: 

    # /usr/share/bcc/tools/tcpsubnet 192.0.2.0/24,198.51.100.0/24,0.0.0.0/0
Tracing... Output every 1 secs. Hit Ctrl-C to end
[02/21/20 10:04:50]
192.0.2.0/24           856
198.51.100.0/24       7467
[02/21/20 10:04:51]
192.0.2.0/24          1200
198.51.100.0/24       8763
0.0.0.0/0              673
...
    Copy to Clipboard Toggle word wrap

    This command displays the traffic in bytes for the specified subnets once per second.

  2. Press Ctrl+C to stop the tracing process.

45.10. Displaying the network throughput by IP address and port
Copiar o link

The tcptop utility displays TCP traffic the host sends and receives in kilobytes. The report automatically refreshes and contains only active TCP connections. The utility uses eBPF features and, therefore, has only a very low overhead.

Procedure

  1. To monitor the sent and received traffic, enter: 

    # /usr/share/bcc/tools/tcptop
13:46:29 loadavg: 0.10 0.03 0.01 1/215 3875

PID    COMM         LADDR           RADDR              RX_KB   TX_KB
3853   3853         192.0.2.1:22    192.0.2.165:41838  32     102626
1285   sshd         192.0.2.1:22    192.0.2.45:39240   0           0
...
    Copy to Clipboard Toggle word wrap

    The output of the command includes only active TCP connections. If the local or remote system closes a connection, the connection is no longer visible in the output.

  2. Press Ctrl+C to stop the tracing process.

45.11. Tracing established TCP connections
Copiar o link

The tcptracer utility traces the kernel functions that connect, accept, and close TCP connections. The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start the tracing process: 

    # /usr/share/bcc/tools/tcptracer
Tracing TCP established connections. Ctrl-C to end.
T  PID    COMM        IP SADDR        DADDR       SPORT  DPORT
A  1088   ns-slapd    4  192.0.2.153  192.0.2.1   0      65535
A  845    sshd        4  192.0.2.1    192.0.2.67  22     42302
X  4502   sshd        4  192.0.2.1    192.0.2.67  22     42302
...
    Copy to Clipboard Toggle word wrap

    Each time the kernel connects, accepts, or closes a connection, tcptracer displays the details of the connections.

  2. Press Ctrl+C to stop the tracing process.

45.12. Tracing IPv4 and IPv6 listen attempts
Copiar o link

The solisten utility traces all IPv4 and IPv6 listen attempts. It traces the attempts including which ultimately fail or the listening program that does not accept the connection. The utility traces functions that the kernel calls when a program wants to listen for TCP connections.

Procedure

  1. Enter the following command to start the tracing process that displays all listen TCP attempts: 

    # /usr/share/bcc/tools/solisten
PID    COMM           PROTO         BACKLOG     PORT     ADDR
3643   nc             TCPv4         1           4242     0.0.0.0
3659   nc             TCPv6         1           4242     2001:db8:1::1
4221   redis-server   TCPv6         128         6379     ::
4221   redis-server   TCPv4         128         6379     0.0.0.0
....
    Copy to Clipboard Toggle word wrap
  2. Press Ctrl+C to stop the tracing process.

45.13. Summarizing the service time of soft interrupts
Copiar o link

The softirqs utility summarizes the time spent servicing soft interrupts (soft IRQs) and shows this time as either totals or histogram distributions. This utility uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism.

Procedure

  1. Enter the following command to start the tracing soft irq event time: 

    # /usr/share/bcc/tools/softirqs
Tracing soft irq event time... Hit Ctrl-C to end.
^C
SOFTIRQ          TOTAL_usecs
tasklet                  166
block                   9152
net_rx                 12829
rcu                    53140
sched                 182360
timer                 306256
    Copy to Clipboard Toggle word wrap
  2. Press Ctrl+C to stop the tracing process.

45.14. Summarizing packets size and count on a network interface
Copiar o link

The netqtop utility displays statistics about the attributes of received (RX) and transmitted (TX) packets on each network queue of a particular network interface.

The statistics include:

  • Bytes per second (BPS)
  • Packets per second (PPS)
  • The average packet size
  • Total number of packets

To generate these statistics, netqtop traces the kernel functions that perform events of transmitted packets net_dev_start_xmit and received packets netif_receive_skb.

Procedure

  1. Display the number of packets within the range of bytes size of the time interval of 2 seconds: 

    # /usr/share/bcc/tools/netqtop -n enp1s0 -i 2

Fri Jan 31 18:08:55 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------
Fri Jan 31 18:08:57 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------
    Copy to Clipboard Toggle word wrap
  2. Press Ctrl+C to stop netqtop.
Voltar ao topo
Red Hat logoGithubredditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar. Explore nossas atualizações recentes.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja o Blog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

Theme

© 2025 Red Hat