Red Hat Summit Red Hat Summitサポートコンソール開発者トライアルの開始お問い合わせ

3.2. Network tracing using the BPF compiler collection

BPF Compiler Collection (BCC) is a library, which facilitates the creation of the extended Berkeley Packet Filter (eBPF) programs. The main utility of eBPF programs is analyzing the operating system performance and network performance without experiencing overhead or security issues.

BCC removes the need for users to know deep technical details of eBPF, and provides many out-of-the-box starting points, such as the bcc-tools package with pre-created eBPF programs.

注記

The eBPF programs are triggered on events, such as disk I/O, TCP connections, and process creations. It is unlikely that the programs should cause the kernel to crash, loop or become unresponsive because they run in a safe virtual machine in the kernel.

3.2.1. Installing the bcc-tools package
リンクのコピー

Install the bcc-tools package, which also installs the BPF Compiler Collection (BCC) library as a dependency.

Procedure

  • Install bcc-tools: 

    # dnf install bcc-tools

    The BCC tools are installed in the /usr/share/bcc/tools/ directory.

Verification

  • Inspect the installed tools: 

    # ls -l /usr/share/bcc/tools/
...
-rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
-rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
-rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
-rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
-rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
-rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
-rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
...

    The doc directory in the listing above contains documentation for each tool.

3.2.2. Displaying TCP connections added to the Kernel’s accept queue
リンクのコピー

By using the tcpaccept utility, you can monitor newly accepted connections by tracing the kernel’s accept() function.

After the kernel receives the ACK packet in a TCP 3-way handshake, the kernel moves the connection from the SYN queue to the accept queue after the connection’s state changes to ESTABLISHED. Therefore, only successful TCP connections are visible in this queue.

The tcpaccept utility uses eBPF features to display all connections the kernel adds to the accept queue. The utility is lightweight because it traces the accept() function of the kernel instead of capturing packets and filtering them. For example, use tcpaccept for general troubleshooting to display new connections the server has accepted.

Procedure

  1. Enter the following command to start the tracing the kernel accept queue: 

    # /usr/share/bcc/tools/tcpaccept
PID   COMM      IP RADDR         RPORT  LADDR    LPORT
843   sshd      4  192.0.2.17    50598  192.0.2.1  22
1107  ns-slapd  4  198.51.100.6  38772  192.0.2.1  389
1107  ns-slapd  4  203.0.113.85  38774  192.0.2.1  389
...

    Each time the kernel accepts a connection, tcpaccept displays the details of the connections.

    For further details, see the tcpaccept(8) man page and the /usr/share/bcc/tools/doc/tcpaccept_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.3. Tracing outgoing TCP connection attempts
リンクのコピー

The tcpconnect utility uses eBPF features to trace outgoing TCP connection attempts. The output of the utility also includes connections that failed.

The tcpconnect utility is lightweight because it traces, for example, the connect() function of the kernel instead of capturing packets and filtering them.

Procedure

  1. Enter the following command to start the tracing process that displays all outgoing connections: 

    # /usr/share/bcc/tools/tcpconnect
PID    COMM         IP SADDR      DADDR          DPORT
31346  curl         4  192.0.2.1  198.51.100.16  80
31348  telnet       4  192.0.2.1  203.0.113.231  23
31361  isc-worker00 4  192.0.2.1  192.0.2.254    53
...

    Each time the kernel processes an outgoing connection, tcpconnect displays the details of the connections.

    For further details, see the tcpconnect(8) man page and the /usr/share/bcc/tools/doc/tcpconnect_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.4. Measuring the latency of outgoing TCP connections
リンクのコピー

The tcpconnlat utility uses eBPF features to measure the time between a sent SYN packet and the received response packet.

The TCP connection latency is the time taken to establish a connection. This typically involves the kernel TCP/IP processing and network round trip time, and not the application runtime.

Procedure

  1. Start measuring the latency of outgoing connections: 

    # /usr/share/bcc/tools/tcpconnlat
PID    COMM         IP SADDR      DADDR          DPORT LAT(ms)
32151  isc-worker00 4  192.0.2.1  192.0.2.254    53    0.60
32155  ssh          4  192.0.2.1  203.0.113.190  22    26.34
32319  curl         4  192.0.2.1  198.51.100.59  443   188.96
...

    Each time the kernel processes an outgoing connection, tcpconnlat displays the details of the connection after the kernel receives the response packet.

    For further details, see the tcpconnlat(8) man page and the /usr/share/bcc/tools/doc/tcpconnlat_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.5. Displaying details about TCP packets and segments that were dropped by the kernel
リンクのコピー

The tcpdrop utility enables administrators to display details about TCP packets and segments that were dropped by the kernel. Use this utility to debug high rates of dropped packets that can cause the remote system to send timer-based retransmits.

High rates of dropped packets and segments can impact the performance of a server. Instead of capturing and filtering packets, which is resource-intensive, the tcpdrop utility uses eBPF features to retrieve the information directly from the kernel.

Procedure

  1. Enter the following command to start displaying details about dropped TCP packets and segments: 

    # /usr/share/bcc/tools/tcpdrop
TIME     PID    IP SADDR:SPORT       > DADDR:DPORT   STATE (FLAGS)
13:28:39 32253  4  192.0.2.85:51616  > 192.0.2.1:22  CLOSE_WAIT (FIN|ACK)
	b'tcp_drop+0x1'
	b'tcp_data_queue+0x2b9'
	...

13:28:39 1      4  192.0.2.85:51616  > 192.0.2.1:22   CLOSE (ACK)
	b'tcp_drop+0x1'
	b'tcp_rcv_state_process+0xe2'
	...

    Each time the kernel drops TCP packets and segments, tcpdrop displays the details of the connection, including the kernel stack trace that led to the dropped package.

    For further details, see the tcpdrop(8) man page and the /usr/share/bcc/tools/doc/tcpdrop_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.6. Tracing TCP sessions
リンクのコピー

The tcplife utility uses eBPF to trace TCP sessions that open and close, and prints a line of output to summarize each one. Administrators can use tcplife to identify connections and the amount of transferred traffic.

For example, you can display connections to port 22 (SSH) to retrieve the following information:

  • The local process ID (PID)
  • The local process name
  • The local IP address and port number
  • The remote IP address and port number
  • The amount of received and transmitted traffic in KB.
  • The time in milliseconds the connection was active

Procedure

  1. Enter the following command to start the tracing of connections to the local port 22: 

    # /usr/share/bcc/tools/tcplife -L 22
PID   COMM    LADDR      LPORT RADDR       RPORT TX_KB  RX_KB      MS
19392 sshd    192.0.2.1  22    192.0.2.17  43892    53     52 6681.95
19431 sshd    192.0.2.1  22    192.0.2.245 43902    81 249381 7585.09
19487 sshd    192.0.2.1  22    192.0.2.121 43970  6998     7 16740.35
...

    Each time a connection is closed, tcplife displays the details of the connections.

    For further details, see the tcplife(8) man page and the /usr/share/bcc/tools/doc/tcplife_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.7. Tracing TCP retransmissions
リンクのコピー

The tcpretrans utility displays details about TCP retransmissions, such as the local and remote IP address and port number, as well as the TCP state at the time of the retransmissions.

The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start displaying TCP retransmission details: 

    # /usr/share/bcc/tools/tcpretrans
TIME     PID  IP LADDR:LPORT   T> RADDR:RPORT         STATE
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:23:02 0    4  192.0.2.1:22  R> 198.51.100.0:26788  ESTABLISHED
00:45:43 0    4  192.0.2.1:22  R> 198.51.100.0:17634  ESTABLISHED
...

    Each time the kernel calls the TCP retransmit function, tcpretrans displays the details of the connection.

    For further details, see the tcpretrans(8) man page and the /usr/share/bcc/tools/doc/tcpretrans_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.8. Displaying TCP state change information
リンクのコピー

During a TCP session, the TCP state changes. The tcpstates utility uses eBPF functions to trace these state changes, and prints details including the duration in each state. For example, use tcpstates to identify if connections spend too much time in the initialization state.

Procedure

  1. Use the following command to start tracing TCP state changes: 

    # /usr/share/bcc/tools/tcpstates
SKADDR           C-PID C-COMM     LADDR     LPORT RADDR       RPORT OLDSTATE    -> NEWSTATE    MS
ffff9cd377b3af80 0     swapper/1  0.0.0.0   22    0.0.0.0     0     LISTEN      -> SYN_RECV    0.000
ffff9cd377b3af80 0     swapper/1  192.0.2.1 22    192.0.2.45  53152 SYN_RECV    -> ESTABLISHED 0.067
ffff9cd377b3af80 818   sssd_nss   192.0.2.1 22    192.0.2.45  53152 ESTABLISHED -> CLOSE_WAIT  65636.773
ffff9cd377b3af80 1432  sshd       192.0.2.1 22    192.0.2.45  53152 CLOSE_WAIT  -> LAST_ACK    24.409
ffff9cd377b3af80 1267  pulseaudio 192.0.2.1 22    192.0.2.45  53152 LAST_ACK    -> CLOSE       0.376
...

    Each time a connection changes its state, tcpstates displays a new line with updated connection details.

    If multiple connections change their state at the same time, use the socket address in the first column (SKADDR) to determine which entries belong to the same connection.

    For further details, see the tcpstates(8) man page and the /usr/share/bcc/tools/doc/tcpstates_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.9. Summarizing and aggregating TCP traffic sent to specific subnets
リンクのコピー

The tcpsubnet utility summarizes and aggregates IPv4 TCP traffic that the local host sends to subnets and displays the output on a fixed interval. The utility uses eBPF features to collect and summarize the data to reduce the overhead.

By default, tcpsubnet summarizes traffic for the following subnets:

  • 127.0.0.1/32
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.0.2.0/24/16
  • 0.0.0.0/0

Note that the last subnet (0.0.0.0/0) is a catch-all option. The tcpsubnet utility counts all traffic for subnets different from the first four in this catch-all entry.

Follow the procedure to count the traffic for the 192.0.2.0/24 and 198.51.100.0/24 subnets. Traffic to other subnets will be tracked in the 0.0.0.0/0 catch-all subnet entry.

Procedure

  1. Start monitoring the amount of traffic sent to the 192.0.2.0/24, 198.51.100.0/24, and other subnets: 

    # /usr/share/bcc/tools/tcpsubnet 192.0.2.0/24,198.51.100.0/24,0.0.0.0/0
Tracing... Output every 1 secs. Hit Ctrl-C to end
[02/21/20 10:04:50]
192.0.2.0/24           856
198.51.100.0/24       7467
[02/21/20 10:04:51]
192.0.2.0/24          1200
198.51.100.0/24       8763
0.0.0.0/0              673
...

    This command displays the traffic in bytes for the specified subnets once per second.

    For further details, see the tcpsubnet(8) man page and the /usr/share/bcc/tools/doc/tcpsubnet.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.10. Displaying the network throughput by IP address and port
リンクのコピー

The tcptop utility displays TCP traffic the host sends and receives in kilobytes. The report automatically refreshes and contains only active TCP connections. The utility uses eBPF features and, therefore, has only a very low overhead.

Procedure

  1. To monitor the sent and received traffic, enter: 

    # /usr/share/bcc/tools/tcptop
13:46:29 loadavg: 0.10 0.03 0.01 1/215 3875

PID    COMM         LADDR           RADDR              RX_KB   TX_KB
3853   3853         192.0.2.1:22    192.0.2.165:41838  32     102626
1285   sshd         192.0.2.1:22    192.0.2.45:39240   0           0
...

    The output of the command includes only active TCP connections. If the local or remote system closes a connection, the connection is no longer visible in the output.

    For further details, see the tcptop(8) man page and the /usr/share/bcc/tools/doc/tcptop.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.11. Tracing established TCP connections
リンクのコピー

The tcptracer utility traces the kernel functions that connect, accept, and close TCP connections. The utility uses eBPF features and, therefore, has a very low overhead.

Procedure

  1. Use the following command to start the tracing process: 

    # /usr/share/bcc/tools/tcptracer
Tracing TCP established connections. Ctrl-C to end.
T  PID    COMM        IP SADDR        DADDR       SPORT  DPORT
A  1088   ns-slapd    4  192.0.2.153  192.0.2.1   0      65535
A  845    sshd        4  192.0.2.1    192.0.2.67  22     42302
X  4502   sshd        4  192.0.2.1    192.0.2.67  22     42302
...

    Each time the kernel connects, accepts, or closes a connection, tcptracer displays the details of the connections.

    For further details, see the tcptracer(8) man page and the /usr/share/bcc/tools/doc/tcptracer_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.12. Tracing IPv4 and IPv6 listen attempts
リンクのコピー

The solisten utility traces all IPv4 and IPv6 listen attempts. It traces the attempts including which ultimately fail or the listening program that does not accept the connection. The utility traces functions that the kernel calls when a program wants to listen for TCP connections.

Procedure

  1. Enter the following command to start the tracing process that displays all listen TCP attempts: 

    # /usr/share/bcc/tools/solisten
PID    COMM           PROTO         BACKLOG     PORT     ADDR
3643   nc             TCPv4         1           4242     0.0.0.0
3659   nc             TCPv6         1           4242     2001:db8:1::1
4221   redis-server   TCPv6         128         6379     ::
4221   redis-server   TCPv4         128         6379     0.0.0.0
....

    For further details, see the solisten(9) man page and the /usr/share/bcc/tools/doc/solisten_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.13. Summarizing the service time of soft interrupts
リンクのコピー

The softirqs utility summarizes the time spent servicing soft interrupts (soft IRQs) and shows this time as either totals or histogram distributions. This utility uses the irq:softirq_enter and irq:softirq_exit kernel tracepoints, which is a stable tracing mechanism.

Procedure

  1. Enter the following command to start the tracing soft irq event time: 

    # /usr/share/bcc/tools/softirqs
Tracing soft irq event time... Hit Ctrl-C to end.
^C
SOFTIRQ          TOTAL_usecs
tasklet                  166
block                   9152
net_rx                 12829
rcu                    53140
sched                 182360
timer                 306256

    For further details, see the softirqs(8) and mpstat(1) man pages and the /usr/share/bcc/tools/doc/softirqs_example.txt file on your system.

  2. Press Ctrl+C to stop the tracing process.

3.2.14. Summarizing packets size and count on a network interface
リンクのコピー

The netqtop utility displays statistics about the attributes of received (RX) and transmitted (TX) packets on each network queue of a particular network interface.

The statistics include:

  • Bytes per second (BPS)
  • Packets per second (PPS)
  • The average packet size
  • Total number of packets

To generate these statistics, netqtop traces the kernel functions that perform events of transmitted packets net_dev_start_xmit and received packets netif_receive_skb.

Procedure

  1. Display the number of packets within the range of bytes size of the time interval of 2 seconds: 

    # /usr/share/bcc/tools/netqtop -n enp1s0 -i 2

Fri Jan 31 18:08:55 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------
Fri Jan 31 18:08:57 2023
TX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	0      	0      	0      	0      	0      	0
 Total  	0      	0      	0      	0      	0      	0

RX
 QueueID	avg_size   [0, 64)	[64, 512)  [512, 2K)  [2K, 16K)  [16K, 64K)
 0      	38.0   	1      	0      	0      	0      	0
 Total  	38.0   	1      	0      	0      	0      	0
-----------------------------------------------------------------------------

    For further details, see the netqtop(8) man page and the /usr/share/bcc/tools/doc/netqtop_example.txt file on your system.

  2. Press Ctrl+C to stop netqtop.
Red Hat logoGithubredditYoutubeTwitter

詳細情報

試用、購入および販売

コミュニティー

Red Hat ドキュメントについて

Red Hat をお使いのお客様が、信頼できるコンテンツが含まれている製品やサービスを活用することで、イノベーションを行い、目標を達成できるようにします。 最新の更新を見る.

多様性を受け入れるオープンソースの強化

Red Hat では、コード、ドキュメント、Web プロパティーにおける配慮に欠ける用語の置き換えに取り組んでいます。このような変更は、段階的に実施される予定です。詳細情報: Red Hat ブログ.

会社概要

Red Hat は、企業がコアとなるデータセンターからネットワークエッジに至るまで、各種プラットフォームや環境全体で作業を簡素化できるように、強化されたソリューションを提供しています。

Theme

© 2026 Red Hatトップに戻る