Configure NUMA in a virtual machine

To improve virtual machine (VM) performance on NUMA-compatible hosts, you can configure Non-Uniform Memory Access (NUMA) settings by using automated utilities or manual XML editing.

Before you begin

  • The host is a NUMA-compatible machine. To detect whether this is the case, use the virsh nodeinfo command and see the NUMA cell(s) line:
    # virsh nodeinfo
    CPU model:           x86_64
    CPU(s):              48
    CPU frequency:       1200 MHz
    CPU socket(s):       1
    Core(s) per socket:  12
    Thread(s) per core:  2
    NUMA cell(s):        2
    Memory size:         67012964 KiB

    If the value of the line is 2 or greater, the host is NUMA-compatible.

  • Optional: You have the numactl package installed on the host.
    # dnf install numactl

About this task

Note that manual NUMA setup is more likely to yield a significant performance improvement than using automated utilities.

Procedure

  1. Set the VM’s NUMA policy to Preferred. For example, to configure the testguest5 VM:
    # virt-xml testguest5 --edit --vcpus placement=auto
    # virt-xml testguest5 --edit --numatune mode=preferred
  2. Configure the numad service to automatically align the VM CPU with memory resources.
    # echo 1 > /proc/sys/kernel/numa_balancing
  3. Start the numad service.
    # systemctl start numad
  4. Optional: Tune NUMA settings manually. Specify which host NUMA nodes you want to assign specifically to a certain VM. This can improve the host memory usage by the VM’s vCPU.
    1. Use the numactl command to view the NUMA topology on the host:
      # numactl --hardware
      available: 2 nodes (0-1)
      node 0 size: 18156 MB
      node 0 free: 9053 MB
      node 1 size: 18180 MB
      node 1 free: 6853 MB
      node distances:
      node   0   1
        0:  10  20
        1:  20  10
    2. Edit the XML configuration of a VM to assign CPU and memory resources to specific NUMA nodes. For example, the following configuration sets testguest6 to use vCPUs 0-7 on NUMA node 0 and vCPUs 8-15 on NUMA node 1. Both nodes are also assigned 16 GiB of VM’s memory.
      # virsh edit <testguest6>
      <domain type='kvm'>
        <name>testguest6</name>
        ...
        <vcpu placement='static'>16</vcpu>
        ...
        <cpu ...>
          <numa>
            <cell id='0' cpus='0-7' memory='16' unit='GiB'/>
            <cell id='1' cpus='8-15' memory='16' unit='GiB'/>
          </numa>
        ...
      </domain>
      Note

      For best performance results, it is a good practice to respect the maximum memory size for each NUMA node on the host.

    3. If the VM is running, restart it to apply the configuration.