9.2. Avoiding TX queue lock contention: Transmit packet steering

Procedure

1. Display the count of available queues:

   # ethtool -l enp1s0
   Channel parameters for enp1s0:
   Pre-set maximums:
   RX:		0
   TX:		0
   Other:		0
   Combined:	4
   Current hardware settings:
   RX:		0
   TX:		0
   Other:		0
   Combined:	1

   The Pre-set maximums section shows the total number of queues and Current hardware settings the number of queues that are currently assigned to the receive, transmit, other, or combined queues.

2. Optional: If you require queues on specific channels, assign them accordingly. For example, to assign the 4 queues to the Combined channel, enter:

   # ethtool -L enp1s0 combined 4

3. Display to which Non-Uniform Memory Access (NUMA) node the NIC is assigned:

   # cat /sys/class/net/enp1s0/device/numa_node
   0

   If the file is not found or the command returns -1, the host is not a NUMA system.

4. If the host is a NUMA system, display which CPUs are assigned to which NUMA node:

   # lscpu | grep NUMA
   NUMA node(s):       2
   NUMA node0 CPU(s):  0-3
   NUMA node1 CPU(s):  4-7

5. In the example above, the NIC has 4 queues and the NIC is assigned to NUMA node 0. This node uses the CPU cores 0-3. Consequently, map each transmit queue to one of the CPU cores from 0-3:

   # echo 1 > /sys/class/net/enp1s0/queues/tx-0/xps_cpus
   # echo 2 > /sys/class/net/enp1s0/queues/tx-1/xps_cpus
   # echo 4 > /sys/class/net/enp1s0/queues/tx-2/xps_cpus
   # echo 8 > /sys/class/net/enp1s0/queues/tx-3/xps_cpus

   If the number of CPU cores and transmit (TX) queues is the same, use a 1 to 1 mapping to avoid any kind of contention on the TX queue. Otherwise, if you map multiple CPUs on the same TX queue, transmit operations on different CPUs will cause TX queue lock contention and negatively impact the transmit throughput.

   Note that you must pass the bitmap, containing the CPU’s core numbers, to the queues. Use the following command to calculate the bitmap:

   # printf %x $((1 << <core_number> ))

Verification

1. Identify the process IDs (PIDs) of services that send traffic:

   # pidof <process_name>
   12345 98765

2. Pin the PIDs to cores that use XPS:

   # numactl -C 0-3 12345 98765

3. Monitor the requeues counter while the process send traffic:

   # tc -s qdisc
   qdisc fq_codel 0: dev enp10s0u1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
    Sent 125728849 bytes 1067587 pkt (dropped 0, overlimits 0 requeues 30)
    backlog 0b 0p requeues 30
    ...

   If the requeues counter no longer increases at a significant rate, TX queue lock contention no longer happens.