Chapter 9. Avoiding listen queue lock contention

1. Enable the rhel-9-for-x86_64-baseos-debug-rpms repository:

   # subscription-manager repos --enable rhel-10-for-x86_64-baseos-debug-rpms

   Copy to Clipboard Toggle word wrap

2. Install the kernel-debuginfo-common-x86_64 package:

   # dnf install kernel-debuginfo-common-x86_64

   Copy to Clipboard Toggle word wrap
3. The code example is now available in the /usr/src/debug/kernel-<version>/linux-<version>/tools/testing/selftests/net/reuseport_bpf_cpu.c file.

Procedure

1. Display the count of available queues:

   # ethtool -l enp1s0
   Channel parameters for enp1s0:
   Pre-set maximums:
   RX:		0
   TX:		0
   Other:		0
   Combined:	4
   Current hardware settings:
   RX:		0
   TX:		0
   Other:		0
   Combined:	1

   Copy to Clipboard Toggle word wrap

   The Pre-set maximums section shows the total number of queues and Current hardware settings the number of queues that are currently assigned to the receive, transmit, other, or combined queues.

2. Optional: If you require queues on specific channels, assign them accordingly. For example, to assign the 4 queues to the Combined channel, enter:

   # ethtool -L enp1s0 combined 4

   Copy to Clipboard Toggle word wrap

3. Display to which Non-Uniform Memory Access (NUMA) node the NIC is assigned:

   # cat /sys/class/net/enp1s0/device/numa_node
   0

   Copy to Clipboard Toggle word wrap

   If the file is not found or the command returns -1, the host is not a NUMA system.

4. If the host is a NUMA system, display which CPUs are assigned to which NUMA node:

   # lscpu | grep NUMA
   NUMA node(s):       2
   NUMA node0 CPU(s):  0-3
   NUMA node1 CPU(s):  4-7

   Copy to Clipboard Toggle word wrap

5. In the example above, the NIC has 4 queues and the NIC is assigned to NUMA node 0. This node uses the CPU cores 0-3. Consequently, map each transmit queue to one of the CPU cores from 0-3:

   # echo 1 > /sys/class/net/enp1s0/queues/tx-0/xps_cpus
   # echo 2 > /sys/class/net/enp1s0/queues/tx-1/xps_cpus
   # echo 4 > /sys/class/net/enp1s0/queues/tx-2/xps_cpus
   # echo 8 > /sys/class/net/enp1s0/queues/tx-3/xps_cpus

   Copy to Clipboard Toggle word wrap

   If the number of CPU cores and transmit (TX) queues is the same, use a 1 to 1 mapping to avoid any kind of contention on the TX queue. Otherwise, if you map multiple CPUs on the same TX queue, transmit operations on different CPUs will cause TX queue lock contention and negatively impact the transmit throughput.

   Note that you must pass the bitmap, containing the CPU’s core numbers, to the queues. Use the following command to calculate the bitmap:

   # printf %x $((1 << <core_number> ))

   Copy to Clipboard Toggle word wrap

Verification

1. Identify the process IDs (PIDs) of services that send traffic:

   # pidof <process_name>
   12345 98765

   Copy to Clipboard Toggle word wrap

2. Pin the PIDs to cores that use XPS:

   # numactl -C 0-3 12345 98765

   Copy to Clipboard Toggle word wrap

3. Monitor the requeues counter while the process send traffic:

   # tc -s qdisc
   qdisc fq_codel 0: dev enp10s0u1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
    Sent 125728849 bytes 1067587 pkt (dropped 0, overlimits 0 requeues 30)
    backlog 0b 0p requeues 30
    ...

   Copy to Clipboard Toggle word wrap

   If the requeues counter no longer increases at a significant rate, TX queue lock contention no longer happens.

Procedure

1. Optional: Display the NetworkManager connection profiles:

   # nmcli connection show
   NAME     UUID                                  TYPE      DEVICE
   example  f2f33f29-bb5c-3a07-9069-be72eaec3ecf  ethernet  enp1s0

   Copy to Clipboard Toggle word wrap

2. Disable GRO support in the connection profile:

   # nmcli connection modify example ethtool.feature-gro off

   Copy to Clipboard Toggle word wrap

3. Reactivate the connection profile:

   # nmcli connection up example

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that GRO is disabled:

   # ethtool -k enp1s0 | grep generic-receive-offload
   generic-receive-offload: off

   Copy to Clipboard Toggle word wrap
2. Monitor the throughput on the server. Re-enable GRO in the NetworkManager profile if the setting has negative side effects to other applications on the host.