Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 6. Tuning UDP connections

Before you start tuning Red Hat Enterprise Linux to improve the throughput of UDP traffic, it is important to have the realistic expectations. UDP is a simple protocol. Compared to TCP, UDP does not contain features, such as flow control, congestion control, and data reliability. This makes it difficult to reach reliable communication over UDP with a throughput rate that is close to the maximum speed of the network interface controller (NIC).

6.1. Detecting packet drops

There are multiple levels in the network stack in which the kernel can drop packets. Red Hat Enterprise Linux provides different utilities to display statistics of these levels. Use them to identify potential problems.

Note that you can ignore a very small rate of dropped packets. However, if you encounter a significant rate, consider tuning measures.

Note

The kernel drops network packets if the networking stack cannot handle the incoming traffic.

Procedure

  • Identify UDP protocol-specific packet drops due to too small socket buffers or slow application processing: 

    # nstat -az UdpSndbufErrors UdpRcvbufErrors
#kernel
UdpSndbufErrors           4    0.0
UdpRcvbufErrors    45716659    0.0
    Copy to Clipboard

    The second column in the output lists the counters.

6.2. Testing the UDP throughput by using iperf3

The iperf3 utility provides a server and client mode to perform network throughput tests between two hosts.

Note

The throughput of applications depend on many factors, such as the buffer sizes that the application uses. Therefore, the results measured with testing utilities, such as iperf3, can significantly be different from those of applications on a server under production workload.

Prerequisites

  • The iperf3 package is installed on both the client and server.
  • No other services on both hosts cause network traffic that substantially affects the test result.
  • Optional: You increased the maximum UDP socket sizes on both the server and the client. For details, see Increasing the system-wide UDP socket buffers.

Procedure

  1. Optional: Display the maximum network speed of the network interface controller (NIC) on both the server and client: 

    # ethtool enp1s0 | grep "Speed"
   Speed: 10000Mb/s
    Copy to Clipboard

  2. On the server:

    1. Display the maximum UDP socket read buffer size, and note the value: 

      # sysctl net.core.rmem_max
net.core.rmem_max = 16777216
      Copy to Clipboard

      The displayed value is in bytes.

    2. Temporarily open the default iperf3 port 5201 in the firewalld service: 

      # firewall-cmd --add-port=5201/tcp --add-port=5201/udp
      Copy to Clipboard

      Note that, iperf3 opens only a TCP socket on the server. If a client wants to use UDP, it first connects to this TCP port, and then the server opens a UDP socket on the same port number for performing the UDP traffic throughput test. For this reason, you must open port 5201 for both the TCP and UDP protocol in the local firewall.

    3. Start iperf3 in server mode: 

      # iperf3 --server
      Copy to Clipboard

      The service now waits for incoming client connections.

  3. On the client:

    1. Display the Maximum Transmission Unit (MTU) of the interface that the client will use for the connection to the server, and note the value: 

      # ip link show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
...
      Copy to Clipboard

    2. Display the maximum UDP socket write buffer size, and note the value: 

      # sysctl net.core.wmem_max
net.core.wmem_max = 16777216
      Copy to Clipboard

      The displayed value is in bytes.

    3. Start measuring the throughput: 

      # iperf3 --udp --time 60 --window 16777216 --length 1472 --bitrate 2G --client 192.0.2.1
      Copy to Clipboard
      • --udp: Use the UDP protocol for the test.
      • --time <seconds>: Defines the time in seconds when the client stops the transmission.
      • --window <size>: Sets the UDP socket buffer size. Ideally, the sizes are the same on both the client and server. In case that they are different, set this parameter to the value that is smaller: net.core.wmem_max on the client or net.core.rmem_max on the server.
      • --length <size>: Sets the length of the buffer to read and write. Set this option to the largest unfragmented payload. Calculate the ideal value as follows: MTU - IP header (20 bytes for IPv4 and 40 bytes for IPv6) - 8 bytes UDP header.

      • --bitrate <rate>: Limits the bit rate to the specified value in bits per second. You can specify units, such as 2G for 2 Gbps.

        Set this parameter to a value that you expect to work and increase it in later measurements. If the client sends packets at a faster rate than the devices on the transmit path or the server can process them, packets can be dropped.

      • --client <server>: Enables the client mode and sets the IP address or name of the server that runs the iperf3 server.

  4. Wait until iperf3 completes the test. Both the server and the client display statistics every second and a summary at the end. For example, the following is a summary displayed on a client: 

    [ ID] Interval       Transfer      Bitrate         Jitter    Lost/Total Datagrams
[ 5] 0.00-60.00 sec  14.0 GBytes   2.00 Gbits/sec  0.000 ms  0/10190216 (0%) sender
[ 5] 0.00-60.04 sec  14.0 GBytes   2.00 Gbits/sec  0.002 ms  0/10190216 (0%) receiver
    Copy to Clipboard

    In this example, the average bit rate was 2 Gbps, and no packets were lost.

  5. On the server:

    1. Press Ctrl+C to stop the iperf3 server.

    2. Close port 5201 in firewalld: 

      # firewall-cmd --remove-port=5201/tcp --remove-port=5201/udp
      Copy to Clipboard

6.3. Impact of the MTU size on UDP traffic throughput

If your application uses a large UDP message size, using jumbo frames can improve the throughput. According to the IEEE 802.3 standard, a default Ethernet frame without Virtual Local Area Network (VLAN) tag has a maximum size of 1518 bytes. Each of these frames includes an 18 bytes header, leaving 1500 bytes for payload. Consequently, for every 1500 bytes of data the server transmits over the network, 18 bytes (1.2%) are overhead.

Jumbo frames are non-standardized frames that have a larger Maximum Transmission Unit (MTU) than the standard Ethernet payload size of 1500 bytes. For example, if you configure jumbo frames with the maximum allowed MTU of 9000 bytes payload, the overhead of each frame reduces to 0.2%.

Important

All network devices on the transmission path and the involved broadcast domains must support jumbo frames and use the same MTU. Packet fragmentation and reassembly due to inconsistent MTU settings on the transmission path reduces the network throughput.

Different connection types have certain MTU limitations:

  • Ethernet: the MTU is limited to 9000 bytes.
  • IP over InfiniBand (IPoIB) in datagram mode: The MTU is limited to 4 bytes less than the InfiniBand MTU.
  • In-memory networking commonly supports larger MTUs. For details, see the respective documentation.

6.4. Impact of the CPU speed on UDP traffic throughput

In bulk transfers, the UDP protocol is much less efficient than TCP, mainly due to the missing packet aggregation in UDP. By default, the Generic Receive Offload (GRO) and UDP Segmentation Offload (USO) features are not enabled. Consequently, the CPU frequency can limit the UDP throughput for bulk transfer on high speed links.

For example, on a tuned host with a high Maximum Transmission Unit (MTU) and large socket buffers, a 3 GHz CPU can process the traffic of a 10 GBit NIC that sends or receives UDP traffic at full speed. However, you can expect about 1-2 Gbps speed loss for every 100 MHz CPU speed under 3 GHz when you transmit UDP traffic. Also, if a CPU speed of 3 GHz can closely achieve 10 Gbps, the same CPU restricts UDP traffic on a 40 GBit NIC to roughly 20-25 Gbps.

6.5. Increasing the system-wide UDP socket buffers

Socket buffers temporarily store data that the kernel has received or should send:

  • The read socket buffer holds packets that the kernel has received but which the application has not read yet.
  • The write socket buffer holds packets that an application has written to the buffer but which the kernel has not passed to the IP stack and network driver yet.

If a UDP packet is too large and exceeds the buffer size or packets are sent or received at a too fast rate, the kernel drops any new incoming UDP packet until the data is removed from the buffer. In this case, increasing the socket buffers can prevent packet loss.

Important

Setting too large buffer sizes wastes memory. Each socket can be set to the size that the application requests, and the kernel doubles this value. For example, if an application requests a 256 KiB socket buffer size and opens 1 million sockets, the system requires 512 GB RAM (512 KiB x 1 million) only for the potential socket buffer space.

Prerequisites

  • You encountered a significant rate of dropped UDP packets.

Procedure

  1. Create the /etc/sysctl.d/10-udp-socket-buffers.conf file and either set the maximum read or write buffer size, or both, based on your requirements: 

    net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
    Copy to Clipboard

    Specify the values in bytes. The values in this example set the maximum size of buffers to 16 MiB. The default values of both parameters are 212992 bytes (208 KiB).

  2. Load the settings from the /etc/sysctl.d/10-udp-socket-buffers.conf file: 

    # sysctl -p /etc/sysctl.d/10-udp-socket-buffers.conf
    Copy to Clipboard

  3. Configure your applications to use the larger socket buffer sizes.

    The net.core.rmem_max and net.core.wmem_max parameters define the maximum buffer size that the setsockopt() function in an application can request. Note that, if you configure your application to not use the setsockopt() function, the kernel uses the values from the rmem_default and wmem_default parameters.

    For further details, see the documentation of the programming language of your application. If you are not the developer of the application, contact the developer.

  4. Restart the applications to use the new UDP buffer sizes.

Verification

  • Monitor the packet drop statistics by using the same method as you used when you encountered the packet drops.

    If packet drops still occur but at a lower rate, increase the buffer sizes further.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat