Chapter 26. Improving network latency using TCP_NODELAY
By default, TCP uses Nagle’s algorithm to collect small outgoing packets to send all at once. This can cause higher rates of latency.
Prerequisites
- You have administrator privileges.
26.1. The effects of using TCP_NODELAY
Applications that require low latency on every packet sent must be run on sockets with the TCP_NODELAY option enabled. This sends buffer writes to the kernel as soon as an event occurs.
- Note
-
For
TCP_NODELAYto be effective, applications must avoid doing small, logically related buffer writes. Otherwise, these small writes causeTCPto send these multiple buffers as individual packets, resulting in poor overall performance.
If applications have several buffers that are logically related and must be sent as one packet, apply one of the following workarounds to avoid poor performance:
-
Build a contiguous packet in memory and then send the logical packet to
TCPon a socket configured withTCP_NODELAY. -
Create an I/O vector and pass it to the kernel using the
writevcommand on a socket configured withTCP_NODELAY. -
Use the
TCP_CORKoption.TCP_CORKtellsTCPto wait for the application to remove the cork before sending any packets. This command causes the buffers it receives to be appended to the existing buffers. This allows applications to build a packet in kernel space, which can be required when using different libraries that provide abstractions for layers.
When a logical packet has been built in the kernel by the various components in the application, the socket should be uncorked, allowing TCP to send the accumulated logical packet immediately.
26.2. Enabling TCP_NODELAY
The TCP_NODELAY option sends buffer writes to the kernel when events occur, with no delays. Enable TCP_NODELAY using the setsockopt() function.
Procedure
Add the following lines to the
TCPapplication’s.cfile.int one = 1; setsockopt(descriptor, SOL_TCP, TCP_NODELAY, &one, sizeof(one));
- Save the file and exit the editor.
Apply one of the following workarounds to prevent poor performance.
-
Build a contiguous packet in memory and then send the logical packet to
TCPon a socket configured withTCP_NODELAY. -
Create an I/O vector and pass it to the kernel using
writevon a socket configured withTCP_NODELAY.
-
Build a contiguous packet in memory and then send the logical packet to
26.3. Enabling TCP_CORK
The TCP_CORK option prevents TCP from sending any packets until the socket is "uncorked".
Procedure
Add the following lines to the
TCPapplication’s.cfile.int one = 1; setsockopt(descriptor, SOL_TCP, TCP_CORK, &one, sizeof(one));
- Save the file and exit the editor.
After the logical packet has been built in the kernel by the various components in the application, disable
TCP_CORK.int zero = 0; setsockopt(descriptor, SOL_TCP, TCP_CORK, &zero, sizeof(zero));
TCPsends the accumulated logical packet immediately, without waiting for any further packets from the application.
26.4. Additional resources
-
tcp(7),setsockopt(3p), andsetsockopt(2)man pages on your system
