Chapter 12. Tuning interrupt coalescence settings

Procedure

1. Identify the network interface that is experiencing the bottleneck:

   # ethtool -S enp1s0
   NIC statistics:
        rx_packets: 1234
        tx_packets: 5678
        rx_bytes: 12345678
        tx_bytes: 87654321
        rx_errors: 0
        tx_errors: 0
        rx_missed: 0
        tx_dropped: 0
        coalesced_pkts: 0
        coalesced_events: 0
        coalesced_aborts: 0

   Identify the packet counters containing drop, discard, or error in their name. These particular statistics measure the actual packet loss at the network interface controller (NIC) packet buffer, which can be caused by NIC coalescence.

2. Monitor values of packet counters you identified in the previous step.

   Compare them to the expected values for your network to determine whether any particular interface experiences a bottleneck. Some common signs of a network bottleneck include, but are not limited to:

   • Many errors on a network interface
   • High packet loss

   • Heavy usage of the network interface

     Note

     Other important factors are for example CPU usage, memory usage, and disk I/O when identifying a network bottleneck.

3. Check the current interrupt coalescence settings:

   # ethtool -c enp1s0
   Coalesce parameters for enp1s0:
           Adaptive RX: off
           Adaptive TX: off
           RX usecs: 100
           RX frames: 8
           RX usecs irq: 100
           RX frames irq: 8
           TX usecs: 100
           TX frames: 8
           TX usecs irq: 100
           TX frames irq: 8

   • The usecs values refer to the number of microseconds that the receiver or transmitter waits before generating an interrupt.
   • The frames values refer to the number of frames that the receiver or transmitter waits before generating an interrupt.

   • The irq values are used to configure the interrupt moderation when the network interface is already handling an interrupt.

     Note

     Not all network interface cards support reporting and changing all values from the example output.

   • The Adaptive RX/TX value represents the adaptive interrupt coalescence mechanism, which adjusts the interrupt coalescence settings dynamically. Based on the packet conditions, the NIC driver auto-calculates coalesce values when Adaptive RX/TX are enabled (the algorithm differs for every NIC driver).

4. Modify the coalescence settings as needed. For example:

   • While ethtool.coalesce-adaptive-rx is disabled, configure ethtool.coalesce-rx-usecs to set the delay before generating an interrupt to 100 microseconds for the RX packets:

     # nmcli connection modify enp1s0 ethtool.coalesce-rx-usecs 100

   • Enable ethtool.coalesce-adaptive-rx while ethtool.coalesce-rx-usecs is set to its default value:

     # nmcli connection modify enp1s0 ethtool.coalesce-adaptive-rx on

     Modify the Adaptive-RX setting as follows:

     • Users concerned with low latency (sub-50us) should not enable Adaptive-RX.
     • Users concerned with throughput can probably enable Adaptive-RX with no harm. If they do not want to use the adaptive interrupt coalescence mechanism, they can try setting large values such as 100us or 250us to ethtool.coalesce-rx-usecs.
     • Users unsure about their needs should not modify this setting until an issue occurs.

5. Re-activate the connection:

   # nmcli connection up enp1s0