Performance Tuning Guide

Procedure 6.1. Balancing Interrupts Automatically

• If your BIOS exports its NUMA topology, the irqbalance service can automatically serve interrupt requests on the node that is local to the hardware requesting service.
  For details on configuring irqbalance, see Section A.1, “irqbalance”.

Procedure 6.2. Balancing Interrupts Manually

1. Check which devices correspond to the interrupt requests that you want to configure.
   Starting with Red Hat Enterprise Linux 7.5, the system configures the optimal interrupt affinity for certain devices and their drivers automatically. You can no longer configure their affinity manually. This applies to the following devices:
   • Devices using the be2iscsi driver
   • NVMe PCI devices
2. Find the hardware specification for your platform. Check if the chipset on your system supports distributing interrupts.
   • If it does, you can configure interrupt delivery as described in the following steps.
     Additionally, check which algorithm your chipset uses to balance interrupts. Some BIOSes have options to configure interrupt delivery.
   • If it does not, your chipset will always route all interrupts to a single, static CPU. You cannot configure which CPU is used.
3. Check which Advanced Programmable Interrupt Controller (APIC) mode is in use on your system.
   Only non-physical flat mode (flat) supports distributing interrupts to multiple CPUs. This mode is available only for systems that have up to 8 CPUs.

   $ journalctl --dmesg | grep APIC

   Copy to Clipboard Toggle word wrap
   In the command output:
   • If your system uses a mode other than flat, you can see a line similar to Setting APIC routing to physical flat.
   • If you can see no such message, your system uses flat mode.
   If your system uses x2apic mode, you can disable it by adding the nox2apic option to the kernel command line in the bootloader configuration.
4. Calculate the smp_affinity mask.
   The smp_affinity value is stored as a hexadecimal bit mask representing all processors in the system. Each bit configures a different CPU. The least significant bit is CPU 0.
   The default value of the mask is f, meaning that an interrupt request can be handled on any processor in the system. Setting this value to 1 means that only processor 0 can handle the interrupt.

   Procedure 6.3. Calculating the Mask

   1. In binary, use the value 1 for CPUs that will handle the interrupts.
      For example, to handle interrupts by CPU 0 and CPU 7, use 0000000010000001 as the binary code:
      Expand

      Table 6.1. Binary Bits for CPUs

      CPU	15	14	13	12	11	10	9	8	7	6	5	4	3	2	1	0
      Binary	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1

      Show more
   2. Convert the binary code to hexadecimal.
      For example, to convert the binary code using Python:

      >>> hex(int('0000000010000001', 2))
      
      '0x81'
      

      Copy to Clipboard Toggle word wrap
   On systems with more than 32 processors, you must delimit smp_affinity values for discrete 32 bit groups. For example, if you want only the first 32 processors of a 64 processor system to service an interrupt request, use 0xffffffff,00000000.
5. Set the smp_affinity mask.
   The interrupt affinity value for a particular interrupt request is stored in the associated /proc/irq/irq_number/smp_affinity file.
   Write the calculated mask to the associated file:

   # echo mask > /proc/irq/irq_number/smp_affinity

   Copy to Clipboard Toggle word wrap

Procedure 7.1. Reserving 1 GB Pages During Early Boot

1. Create a HugeTLB pool for 1 GB pages by appending the following line to the kernel command-line options in the /etc/default/grub file as root:

   default_hugepagesz=1G hugepagesz=1G
   

   Copy to Clipboard Toggle word wrap
2. Regenerate the GRUB2 configuration using the edited default file. If your system uses BIOS firmware, execute the following command:

   # grub2-mkconfig -o /boot/grub2/grub.cfg
   

   Copy to Clipboard Toggle word wrap
   On a system with UEFI firmware, execute the following command:

     # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
   

   Copy to Clipboard Toggle word wrap
3. Create a file named /usr/lib/systemd/system/hugetlb-gigantic-pages.service with the following content:

   [Unit]
   Description=HugeTLB Gigantic Pages Reservation
   DefaultDependencies=no
   Before=dev-hugepages.mount
   ConditionPathExists=/sys/devices/system/node
   ConditionKernelCommandLine=hugepagesz=1G
   
   [Service]
   Type=oneshot
   RemainAfterExit=yes
   ExecStart=/usr/lib/systemd/hugetlb-reserve-pages.sh
   
   [Install]
   WantedBy=sysinit.target
   

   Copy to Clipboard Toggle word wrap
4. Create a file named /usr/lib/systemd/hugetlb-reserve-pages.sh with the following content:

   #!/bin/sh
   
   nodes_path=/sys/devices/system/node/
   if [ ! -d $nodes_path ]; then
   	echo "ERROR: $nodes_path does not exist"
   	exit 1
   fi
   
   reserve_pages()
   {
   	echo $1 > $nodes_path/$2/hugepages/hugepages-1048576kB/nr_hugepages
   }
   
   reserve_pages number_of_pages node

   Copy to Clipboard Toggle word wrap
   On the last line, replace number_of_pages with the number of 1GB pages to reserve and node with the name of the node on which to reserve these pages.

   Example 7.1. Reserving Pages on node0 and node1

   For example, to reserve two 1GB pages on node0 and one 1GB page on node1, replace the last line with the following code:

   reserve_pages 2 node0
   reserve_pages 1 node1
   

   Copy to Clipboard Toggle word wrap

   You can modify it to your needs or add more lines to reserve memory in other nodes.
5. Make the script executable:

   # chmod +x /usr/lib/systemd/hugetlb-reserve-pages.sh
   

   Copy to Clipboard Toggle word wrap
6. Enable early boot reservation:

   # systemctl enable hugetlb-gigantic-pages
   

   Copy to Clipboard Toggle word wrap