Red Hat SummitChapter 11. Socket options in RHEL for Real Time
The real-time socket is a two way data transfer mechanism between two processes on same systems such as the UNIX domain and loopback devices or on different systems such as network sockets.
Transmission Control Protocol (TCP) is the most common transport protocol and is often used to achieve consistent low latency for a service that requires constant communication or to cork the sockets in a low priority restricted environment.
With new applications, hardware features, and kernel architecture optimizations, TCP has to introduce new approaches to handle the changes effectively. The new approaches can cause unstable program behaviors. Because the program behavior changes as the underlying operating system components change, they must be handled with care.
One example of such behavior in TCP is the delay in sending small buffers. This allows sending them as one network packet. Buffering small writes to TCP and sending them all at once generally works well, but it can also create latencies. For real-time applications, the TCP_NODELAY socket option disables the delay and sends small writes as soon as they are ready.
The relevant socket options for data transfer are TCP_NODELAY and TCP_CORK.
11.1. TCP_NODELAY socket option
The TCP_NODELAY socket option disables Nagle’s algorithm. Configuring TCP_NODELAY with the setsockopt sockets API function sends multiple small buffer writes as individual packets as soon as they are ready.
Sending multiple logically related buffers as a single packet by building a contiguous packet before sending, achieves better latency and performance. Alternatively, if the memory buffers are logically related but not contiguous, you can create an I/O vector and pass it to the kernel using writev on a socket with TCP_NODELAY enabled.
The following example illustrates enabling TCP_NODELAY through the setsockopt sockets API.
int one = 1; setsockopt(descriptor, SOL_TCP, TCP_NODELAY, &one, sizeof(one));
To use TCP_NODELAY effectively, avoid small, logically related buffer writes. With TCP_NODELAY, small writes make TCP send multiple buffers as individual packets, which may result in poor overall performance.
Additional resources
-
sendfile(2)man page on your system
11.2. TCP_CORK socket option
The TCP_CORK option collects all data packets in a socket and prevents from transmitting them until the buffer fills to a specified limit. This enables applications to build a packet in the kernel space and send data when TCP_CORK is disabled. TCP_CORK is set on a socket file descriptor using the setsocketopt() function. When developing programs, if you must send bulk data from a file, consider using TCP_CORK with the sendfile() function.
When a logical packet is built in the kernel by various components, enable TCP_CORK by configuring it to a value of 1 using the setsockopt sockets API. This is known as "corking the socket”. TCP_CORK can cause bugs if the cork is not removed at an appropriate time.
The following example illustrates enabling TCP_CORK through the setsockopt sockets API.
int one = 1; setsockopt(descriptor, SOL_TCP, TCP_CORK, &one, sizeof(one));
In some environments, if the kernel is not able to identify when to remove the cork, you can manually remove it as follows:
int zero = 0; setsockopt(descriptor, SOL_TCP, TCP_CORK, &zero, sizeof(zero));
Additional resources
-
sendfile(2)man page on your system
11.3. Example programs using socket options
The TCP_NODELAY and TCP_CORK socket options significantly influence the behavior of a network connection. TCP_NODELAY disables the Nagle’s algorithm on applications that benefit by sending data packets as soon as they are ready. With TCP_CORK, you can transfer multiple data packets simultaneously, with no delays between them.
To enable the socket options, for example TCP_NODELAY, build it with the following code and then set appropriate options.
gcc tcp_nodelay_client.c -o tcp_nodelay_client -lrt
When you run the tcp_nodelay_server and tcp_nodelay_client programs without any arguments, the client uses the default socket options. For more information about tcp_nodelay_server and tcp_nodelay_client programs, see the Red Hat Knowledgebase solution TCP changes result in latency performance when small buffers are used.
The example programs provide information about the performance impact these socket options can have on your applications.
Performance impact on a client
You can send small buffer writes to a client without using the TCP_NODELAY and TCP_CORK socket options. When run without any arguments, the client uses the default socket options.
To initiate data transfer, define the server TCP port and the number of packets it must process. For example, 10,000 packets in this test.
$ ./tcp_nodelay_server 5001 10000The code sends 15 packets, each of two bytes, and waits for a response from the server. It adopts the default TCP behavior here
Performance impact on a loopback interface
To enable the socket option, build it using gcc tcp_nodelay_client.c -o tcp_nodelay_client -lrt and then set the appropriate options.
Following examples use a loopback interface to demonstrate three variations:
To send buffer writes immediately, set the
no_delayoption on a socket configured withTCP_NODELAY.$ ./tcp_nodelay_client localhost 5001 10000 no_delay 10000 packets of 30 bytes sent in 1649.771240 ms: 181.843399 bytes/ms using TCP_NODELAYTCP sends the buffers right away, disabling the algorithm that combines the small packets. This improves performance but can cause a flurry of small packets to be sent for each logical packet.
To collect multiple data packets and send them with one system call, configure the
TCP_CORKsocket option.$ ./tcp_nodelay_client localhost 5001 10000 cork 10000 packets of 30 bytes sent in 850.796448 ms: 352.610779 bytes/ms using TCP_CORKUsing the cork technique significantly reduces the time required to send data packets as it combines full logical packets in its buffers and sends fewer overall network packets. You must ensure to remove the
corkat the appropriate time.When developing programs, if you must send bulk data from a file, consider using
TCP_CORKwith thesendfile()option.To measure performance without using socket options.
$ ./tcp_nodelay_client localhost 5001 10000 10000 packets of 30 bytes sent in 400129.781250 ms: 0.749757 bytes/msThis is the baseline measure when TCP combines buffer writes and waits to check for more data than can optimally fit in the network packet.
Additional resources
-
sendfile(2)man page on your system
