5.3. Increasing the system-wide TCP socket buffers

Procedure

1. Determine the latency of the connection. For example, ping from the client to server to measure the average Round Trip Time (RTT):

   # ping -c 10 server.example.com
   ...
   --- server.example.com ping statistics ---
   10 packets transmitted, 10 received, 0% packet loss, time 9014ms
   rtt min/avg/max/mdev = 17.208/17.056/19.333/0.616 ms

   In this example, the latency is 17 ms.

2. Use the following formula to calculate the Bandwidth Delay Product (BDP) for the traffic you want to tune:

   connection speed in bytes * latency in seconds = BDP in bytes

   For example, to calculate the BDP for a 10 Gbps connection that has a 17 ms latency:

   (10 * 1000 * 1000 * 1000 / 8) * 0.017 = 21,250,000 bytes

3. Create the /etc/sysctl.d/10-tcp-socket-buffers.conf file and either set the maximum read or write buffer size, or both, based on your requirements:

   net.ipv4.tcp_rmem = 4096 262144 42500000
   net.ipv4.tcp_wmem = 4096 262144 42500000

   Specify the values in bytes. Use the following rule of thumb when you try to identify optimized values for your environment:

   • Default buffer size (second value): Increase this value only slightly or set it to 524288 (512 KiB) at most. A too high default buffer size can cause buffer collapsing and, consequently, latency spikes.
   • Maximum buffer size (third value): A value double to triple of the BDP is often sufficient.

4. Load the settings from the /etc/sysctl.d/10-tcp-socket-buffers.conf file:

   # sysctl -p /etc/sysctl.d/10-tcp-socket-buffers.conf

5. If you have changed the second value in the net.ipv4.tcp_rmem or net.ipv4.tcp_wmem parameter, restart the applications to use the new TCP buffer sizes.

   If you have changed only the third value, you do not need to restart the application because auto-tuning applies these settings dynamically.

Verification

1. Optional: Test the TCP throughput by using iperf3.

2. Monitor the packet drop statistics by using the same method that you used when you encountered the packet drops.

   If packet drops still occur but at a lower rate, increase the buffer sizes further.