Chapter 8. Sockets
A socket is a bi-directional data transfer mechanism. They are used to transfer data between two processes. The two processes can be running on the same system as Unix-domain or loopback sockets, or on different systems as network sockets.
There are no special options or restriction to using sockets on a Red Hat Enterprise Linux for Real Time system.
8.1. Socket Options
There are two socket options that are relevant to Red Hat Enterprise Linux for Real Time applications:
TCP_NODELAY and TCP_CORK.
TCP_NODELAYTCP is the most common transport protocol, which means it is often used to solve many different needs. As new application and hardware features are developed, and kernel architecture optimizations are made, TCP has had to introduce new heuristics to handle the changes effectively.
These heuristics can result in a program becoming unstable. Because the behavior changes as the underlying operating system components change, they should be treated with care.
One example of heuristic behavior in TCP is that small buffers are delayed. This allows them to be sent as one network packet. This generally works well, but it can also create latencies. For Red Hat Enterprise Linux for Real Time applications,
TCP_NODELAY is a socket option that can be used to turn this behavior off. It can be enabled through the setsockopt sockets API, with the following function:
int one = 1; setsockopt(descriptor, SOL_TCP, TCP_NODELAY, &one, sizeof(one));
For this option to be used effectively, the applications must avoid doing small buffer writes, as TCP will send these buffers as individual packets.
TCP_NODELAY can also interact with other optimization heuristics to result in poor overall performance.
If applications have several buffers that are logically related and that should be sent as one packet it will achieve better latency and performance by building a contiguous packet before sending. The packet can then be sent as one using a socket with
TCP_NODELAY enabled.
Alternatively, if the memory buffers are logically related but not already contiguous, use them to build an I/O vector. It can then be passed to the kernel using
writev on a socket with TCP_NODELAY enabled.
TCP_CORK
Another TCP socket option that works in a similar way is TCP_CORK. When enabled, TCP will delay all packets until the application removes the cork, and allows the stored packets to be sent. This allows applications to build a packet in kernel space, which is useful when different libraries are being used to provide layer abstractions.
The
TCP_CORK option can can be enabled by using the following function:
int one = 1; setsockopt(descriptor, SOL_TCP, TCP_CORK, &one, sizeof(one));
Enabling
TCP_CORK is often referred to as corking the socket.
In a situation where the kernel is not able to identify when to remove the cork, it can be manually removed with the function:
int zero = 0; setsockopt(descriptor, SOL_TCP, TCP_CORK, &zero, sizeof(zero));Once the socket is uncorked, TCP will send the accumulated logical package immediately, without waiting for further packets from the application.
Example 8.1. Using TCP_NODELAY and TCP_CORK
This example demonstrates the performance impact that
TCP_NODELAY and TCP_CORK can have on an application.
The server waits for packets of 30 bytes and then sends a 2 byte packet in response. To start with, define the TCP port and the number of packets it should process. In this example, it is 10,000 packets:
~]$ ./tcp_nodelay_server 5001 10000
The server does not need to have any socket options set.
If the client is run without any arguments, the default socket options will be used. Use the
no_delay option to enable TCP_NODELAY socket options. Use the cork option to enable TCP_CORK. In all cases it will send 15 packets, each of two bytes, and wait for a response from the server.
This example uses a loopback interface to demonstrate three variations.
In the first variation, neither
TCP_NODELAY nor TCP_CORK are in use. This is a baseline measurement. TCP coalesces writes and has to wait to check if the application has more data than can optimally fit in the network packet:
~]$ ./tcp_nodelay_client localhost 5001 10000
10000 packets of 30 bytes sent in 400129.781250 ms: 0.749757 bytes/ms
The second variation uses
TCP_NODELAY only. TCP is instructed not to coalesce small packets, but to send buffers immediately. This improves performance significantly, but creates a large number of network packets for each logical packet:
~]$ ./tcp_nodelay_client localhost 5001 10000 no_delay
10000 packets of 30 bytes sent in 1649.771240 ms: 181.843399 bytes/ms using TCP_NODELAY
The third variation uses
TCP_CORK only. It halves the time required to the send the same number of logical packets. This is because TCP coalesces full logical packets in its buffers, and sends fewer overall network packets:
~]$ ./tcp_nodelay_client localhost 5001 10000 cork
10000 packets of 30 bytes sent in 850.796448 ms: 352.610779 bytes/ms using TCP_CORK
In this scenario,
TCP_CORK is the best technique to use. It allows the application to precisely convey the information that a packet is finished and must be sent without delay. When developing programs, if they need to send bulk data from a file, consider using TCP_CORK with sendfile.
Note
For more information, or for further reading, the following man page and example applications are related to the information given in this section:
- sendfile(2)
- “TCP nagle sample applications”, which are example applications of both socket options, written in C. To download them, right-click and save from the following links:
