Chapter 18. Optimizing virtual machine performance

Procedure

1. Stop libvirtd and its sockets.

   $ systemctl stop libvirtd.service
   $ systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket

2. Disable libvirtd to prevent it from starting on boot.

   $ systemctl disable libvirtd.service
   $ systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket

3. Enable the modular libvirt daemons.

   # for drv in qemu interface network nodedev nwfilter secret storage; do systemctl unmask virt${drv}d.service; systemctl unmask virt${drv}d{,-ro,-admin}.socket; systemctl enable virt${drv}d.service; systemctl enable virt${drv}d{,-ro,-admin}.socket; done

4. Start the sockets for the modular daemons.

   # for drv in qemu network nodedev nwfilter secret storage; do systemctl start virt${drv}d{,-ro,-admin}.socket; done

5. Optional: If you require connecting to your host from remote hosts, enable and start the virtualization proxy daemon.

   1. Check whether the libvirtd-tls.socket service is enabled on your system.

      # grep listen_tls /etc/libvirt/libvirtd.conf
      
      listen_tls = 0

   2. If libvirtd-tls.socket is not enabled (listen_tls = 0), activate virtproxyd as follows:

      # systemctl unmask virtproxyd.service
      # systemctl unmask virtproxyd{,-ro,-admin}.socket
      # systemctl enable virtproxyd.service
      # systemctl enable virtproxyd{,-ro,-admin}.socket
      # systemctl start virtproxyd{,-ro,-admin}.socket

   3. If libvirtd-tls.socket is enabled (listen_tls = 1), activate virtproxyd as follows:

      # systemctl unmask virtproxyd.service
      # systemctl unmask virtproxyd{,-ro,-admin,-tls}.socket
      # systemctl enable virtproxyd.service
      # systemctl enable virtproxyd{,-ro,-admin,-tls}.socket
      # systemctl start virtproxyd{,-ro,-admin,-tls}.socket

      To enable the TLS socket of virtproxyd, your host must have TLS certificates configured to work with libvirt. For more information, see the Upstream libvirt documentation.

Verification

1. Activate the enabled virtualization daemons.

   # virsh uri
   qemu:///system

2. Verify that your host is using the virtqemud modular daemon.

   # systemctl is-active virtqemud.service
   active

   If the status is active, you have successfully enabled modular libvirt daemons.

Procedure

1. Optional: Obtain the information about the maximum memory and currently used memory for a VM. This will serve as a baseline for your changes, and also for verification.

   # virsh dominfo testguest
   Max memory:     2097152 KiB
   Used memory:    2097152 KiB

1. Log in to the RHEL 9 web console.

   For details, see Logging in to the web console.

2. In the Virtual Machines interface, click the VM whose information you want to see.

   A new page opens with an Overview section with basic information about the selected VM and a Console section to access the VM’s graphical interface.

3. Click edit next to the Memory line in the Overview pane.

   The Memory Adjustment dialog appears.

   

4. Configure the virtual memory for the selected VM.

   • Maximum allocation - Sets the maximum amount of host memory that the VM can use for its processes. You can specify the maximum memory when creating the VM or increase it later. You can specify memory as multiples of MiB or GiB.

     Adjusting maximum memory allocation is only possible on a shut-off VM.

   • Current allocation - Sets the actual amount of memory allocated to the VM. This value can be less than the Maximum allocation but cannot exceed it. You can adjust the value to regulate the memory available to the VM for its processes. You can specify memory as multiples of MiB or GiB.

     If you do not specify this value, the default allocation is the Maximum allocation value.

5. Click Save.

   The memory allocation of the VM is adjusted.

Procedure

1. Optional: Obtain the information about the maximum memory and currently used memory for a VM. This will serve as a baseline for your changes, and also for verification.

   # virsh dominfo testguest
   Max memory:     2097152 KiB
   Used memory:    2097152 KiB

2. Adjust the maximum memory allocated to a VM. Increasing this value improves the performance potential of the VM, and reducing the value lowers the performance footprint the VM has on your host. Note that this change can only be performed on a shut-off VM, so adjusting a running VM requires a reboot to take effect.

   For example, to change the maximum memory that the testguest VM can use to 4096 MiB:

   # virt-xml testguest --edit --memory memory=4096,currentMemory=4096
   Domain 'testguest' defined successfully.
   Changes will take effect after the domain is fully powered off.

   To increase the maximum memory of a running VM, you can attach a memory device to the VM. This is also referred to as memory hot plug. For details, see Attaching memory devices to virtual machines.

   Warning

   Removing memory devices from a running VM (also referred as a memory hot unplug) is not supported, and highly discouraged by Red Hat.

3. Optional: You can also adjust the memory currently used by the VM, up to the maximum allocation. This regulates the memory load that the VM has on the host until the next reboot, without changing the maximum VM allocation.

   # virsh setmem testguest --current 2048

Verification

1. Confirm that the memory used by the VM has been updated:

   # virsh dominfo testguest
   Max memory:     4194304 KiB
   Used memory:    2097152 KiB

2. Optional: If you adjusted the current VM memory, you can obtain the memory balloon statistics of the VM to evaluate how effectively it regulates its memory use.

    # virsh domstats --balloon testguest
   Domain: 'testguest'
     balloon.current=365624
     balloon.maximum=4194304
     balloon.swap_in=0
     balloon.swap_out=0
     balloon.major_fault=306
     balloon.minor_fault=156117
     balloon.unused=3834448
     balloon.available=4035008
     balloon.usable=3746340
     balloon.last-update=1587971682
     balloon.disk_caches=75444
     balloon.hugetlb_pgalloc=0
     balloon.hugetlb_pgfail=0
     balloon.rss=1005456

Procedure

1. Shut down the selected VM if it is running.

2. To configure a VM to use 1 GiB huge pages, open the XML definition of a VM for editing. For example, to edit a testguest VM, run the following command:

   # virsh edit testguest

3. Add the following lines to the <memoryBacking> section in the XML definition:

   <memoryBacking>
     <hugepages>
       <page size='1' unit='GiB'/>
     </hugepages>
   </memoryBacking>

Verification

1. Start the VM.

2. Confirm that the host has successfully allocated huge pages for the running VM. On the host, run the following command:

   # cat /proc/meminfo | grep Huge
   
   HugePages_Total:    4
   HugePages_Free:     2
   HugePages_Rsvd:     1
   Hugepagesize:       1024000 kB

   When you add together the number of free and reserved huge pages (HugePages_Free + HugePages_Rsvd), the result should be less than the total number of huge pages (HugePages_Total). The difference is the number of huge pages that is used by the running VM.

1. Display the current <blkio> parameters for a VM:

   # virsh dumpxml VM-name

   <domain>
     [...]
     <blkiotune>
       <weight>800</weight>
       <device>
         <path>/dev/sda</path>
         <weight>1000</weight>
       </device>
       <device>
         <path>/dev/sdb</path>
         <weight>500</weight>
       </device>
     </blkiotune>
     [...]
   </domain>

2. Edit the I/O weight of a specified device:

   # virsh blkiotune VM-name --device-weights device, I/O-weight

   For example, the following changes the weight of the /dev/sda device in the testguest1 VM to 500.

   # virsh blkiotune testguest1 --device-weights /dev/sda, 500

Procedure

1. Use the virsh domblklist command to list the names of all the disk devices on a specified VM.

   # virsh domblklist rollin-coal
   Target     Source
   ------------------------------------------------
   vda        /var/lib/libvirt/images/rollin-coal.qcow2
   sda        -
   sdb        /home/horridly-demanding-processes.iso

2. Find the host block device where the virtual disk that you want to throttle is mounted.

   For example, if you want to throttle the sdb virtual disk from the previous step, the following output shows that the disk is mounted on the /dev/nvme0n1p3 partition.

   $ lsblk
   NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   zram0                                         252:0    0     4G  0 disk  [SWAP]
   nvme0n1                                       259:0    0 238.5G  0 disk
   ├─nvme0n1p1                                   259:1    0   600M  0 part  /boot/efi
   ├─nvme0n1p2                                   259:2    0     1G  0 part  /boot
   └─nvme0n1p3                                   259:3    0 236.9G  0 part
     └─luks-a1123911-6f37-463c-b4eb-fxzy1ac12fea 253:0    0 236.9G  0 crypt /home

3. Set I/O limits for the block device by using the virsh blkiotune command.

   # virsh blkiotune VM-name --parameter device,limit

   The following example throttles the sdb disk on the rollin-coal VM to 1000 read and write I/O operations per second and to 50 MB per second read and write throughput.

   # virsh blkiotune rollin-coal --device-read-iops-sec /dev/nvme0n1p3,1000 --device-write-iops-sec /dev/nvme0n1p3,1000 --device-write-bytes-sec /dev/nvme0n1p3,52428800 --device-read-bytes-sec /dev/nvme0n1p3,52428800

Procedure

1. To enable multi-queue on a storage device, edit the XML configuration of the VM.

   # virsh edit <example_vm>

2. In the XML configuration, find the intended storage device and change the queues parameter to use multiple I/O queues. Replace N with the number of vCPUs in the VM, up to 16.

   • A virtio-blk example:

     <disk type='block' device='disk'>
       <driver name='qemu' type='raw' queues='N'/>
       <source dev='/dev/sda'/>
       <target dev='vda' bus='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
     </disk>

   • A virtio-scsi example:

     <controller type='scsi' index='0' model='virtio-scsi'>
        <driver queues='N' />
     </controller>

3. Restart the VM for the changes to take effect.

Procedure

1. Shut down the selected VM if it is running.

2. On the host, add or edit the <iothreads> tag in the XML configuration of the VM. For example, to create a single IOThread for a testguest1 VM:

   # virsh edit <testguest1>
   
   <domain type='kvm'>
     <name>testguest1</name>
     ...
     <vcpu placement='static'>8</vcpu>
     <iothreads>1</iothreads>
     ...
   </domain>

   Note

   For optimal results, use only 1-2 IOThreads per CPU on the host.

3. Assign a dedicated IOThread` to a VM disk. For example, to assign an IOThread with ID of 1 to a disk on the testguest1 VM:

   # virsh edit <testguest1>
   
   <domain type='kvm'>
     <name>testguest1</name>
     ...
     <devices>
       <disk type='file' device='disk'>
         <driver name='qemu' type='raw' cache='none' io='native' iothread='1'/>
         <source file='/var/lib/libvirt/images/test-disk.raw'/>
         <target dev='vda' bus='virtio'/>
         <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
       </disk>
       ...
     </devices>
     ...
   </domain>

   Note

   IOThread IDs start from 1 and you must dedicate only a single IOThread to a disk.

   Usually, a one dedicated IOThread per VM is sufficient for optimal performance.

4. When using virtio-scsi storage devices, assign a dedicated IOThread` to the virtio-scsi controller. For example, to assign an IOThread with ID of 1 to a controller on the testguest1 VM:

   # virsh edit <testguest1>
   
   <domain type='kvm'>
     <name>testguest1</name>
     ...
     <devices>
       <controller type='scsi' index='0' model='virtio-scsi'>
         <driver iothread='1'/>
         <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
       </controller>
       ...
     </devices>
     ...
   </domain>

Procedure

1. Shut down the selected VM if it is running.

2. Edit the XML configuration of the selected VM.

   # virsh edit <vm_name>

3. Find the disk device and edit the cache option in the driver tag.

   <domain type='kvm'>
     <name>testguest1</name>
     ...
     <devices>
       <disk type='file' device='disk'>
         <driver name='qemu' type='raw' cache='none' io='native' iothread='1'/>
         <source file='/var/lib/libvirt/images/test-disk.raw'/>
         <target dev='vda' bus='virtio'/>
         <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
       </disk>
       ...
     </devices>
     ...
   </domain>

Procedure

1. Adjust the maximum number of vCPUs that can be attached to a VM, which takes effect on the VM’s next boot.

   For example, to increase the maximum vCPU count for the testguest VM to 8:

   # virsh setvcpus testguest 8 --maximum --config

   Note that the maximum may be limited by the CPU topology, host hardware, the hypervisor, and other factors.

2. Adjust the current number of vCPUs attached to a VM, up to the maximum configured in the previous step. For example:

   • To increase the number of vCPUs attached to the running testguest VM to 4:

     # virsh setvcpus testguest 4 --live

     This increases the VM’s performance and host load footprint of testguest until the VM’s next boot.

   • To permanently decrease the number of vCPUs attached to the testguest VM to 1:

     # virsh setvcpus testguest 1 --config

     This decreases the VM’s performance and host load footprint of testguest after the VM’s next boot. However, if needed, additional vCPUs can be hot plugged to the VM to temporarily increase its performance.

Procedure

1. Log in to the RHEL 9 web console.

   For details, see Logging in to the web console.

2. In the Virtual Machines interface, click the VM whose information you want to see.

   A new page opens with an Overview section with basic information about the selected VM and a Console section to access the VM’s graphical interface.

3. Click edit next to the number of vCPUs in the Overview pane.

   The vCPU details dialog appears.

   

1. Configure the virtual CPUs for the selected VM.

   • vCPU Count - The number of vCPUs currently in use.

     Note

     The vCPU count cannot be greater than the vCPU Maximum.

   • vCPU Maximum - The maximum number of virtual CPUs that can be configured for the VM. If this value is higher than the vCPU Count, additional vCPUs can be attached to the VM.
   • Sockets - The number of sockets to expose to the VM.
   • Cores per socket - The number of cores for each socket to expose to the VM.

   • Threads per core - The number of threads for each core to expose to the VM.

     Note that the Sockets, Cores per socket, and Threads per core options adjust the CPU topology of the VM. This may be beneficial for vCPU performance and may impact the functionality of certain software in the guest OS. If a different setting is not required by your deployment, keep the default values.

2. Click Apply.

   The virtual CPUs for the VM are configured.

   Note

   Changes to virtual CPU settings only take effect after the VM is restarted.

1. Optional: Use the numactl command to view the NUMA topology on the host:

   # numactl --hardware
   
   available: 2 nodes (0-1)
   node 0 size: 18156 MB
   node 0 free: 9053 MB
   node 1 size: 18180 MB
   node 1 free: 6853 MB
   node distances:
   node   0   1
     0:  10  20
     1:  20  10

2. Edit the XML configuration of a VM to assign CPU and memory resources to specific NUMA nodes. For example, the following configuration sets testguest6 to use vCPUs 0-7 on NUMA node 0 and vCPUS 8-15 on NUMA node 1. Both nodes are also assigned 16 GiB of VM’s memory.

   # virsh edit <testguest6>
   
   <domain type='kvm'>
     <name>testguest6</name>
     ...
     <vcpu placement='static'>16</vcpu>
     ...
     <cpu ...>
       <numa>
         <cell id='0' cpus='0-7' memory='16' unit='GiB'/>
         <cell id='1' cpus='8-15' memory='16' unit='GiB'/>
       </numa>
     ...
   </domain>

3. If the VM is running, restart it to apply the configuration.

Procedure

1. Check the CPU topology on the host:

   # lscpu -p=node,cpu
   
   Node,CPU
   0,0
   0,1
   0,2
   0,3
   0,4
   0,5
   0,6
   0,7
   1,0
   1,1
   1,2
   1,3
   1,4
   1,5
   1,6
   1,7

   In this example, the output contains NUMA nodes and the available physical CPU threads on the host.

2. Check the number of vCPU threads inside the VM:

   # lscpu -p=node,cpu
   
   Node,CPU
   0,0
   0,1
   0,2
   0,3

   In this example, the output contains NUMA nodes and the available vCPU threads inside the VM.

3. Pin specific vCPU threads from a VM to a specific host CPU or range of CPUs. This is recommended as a safe method of vCPU performance improvement.

   For example, the following commands pin vCPU threads 0 to 3 of the testguest6 VM to host CPUs 1, 3, 5, 7, respectively:

   # virsh vcpupin testguest6 0 1
   # virsh vcpupin testguest6 1 3
   # virsh vcpupin testguest6 2 5
   # virsh vcpupin testguest6 3 7

4. Optional: Verify whether the vCPU threads are successfully pinned to CPUs.

   # virsh vcpupin testguest6
   VCPU   CPU Affinity
   ----------------------
   0      1
   1      3
   2      5
   3      7

5. After pinning vCPU threads, you can also pin QEMU process threads associated with a specified VM to a specific host CPU or range of CPUs. This can further help the QEMU process to run more efficiently on the physical CPU.

   For example, the following commands pin the QEMU process thread of testguest6 to CPUs 2 and 4, and verify this was successful:

   # virsh emulatorpin testguest6 2,4
   # virsh emulatorpin testguest6
   emulator: CPU Affinity
   ----------------------------------
          *: 2,4

Procedure

1. View the current vCPU scheduling configuration on the host.

   # virsh schedinfo <vm_name>
   
   Scheduler      : posix
   cpu_shares     : 0
   vcpu_period : 0
   vcpu_quota : 0
   emulator_period: 0
   emulator_quota : 0
   global_period  : 0
   global_quota   : 0
   iothread_period: 0
   iothread_quota : 0

2. To configure an absolute vCPU cap for a VM, set the vcpu_period and vcpu_quota parameters. Both parameters use a numerical value that represents a time duration in microseconds.

   1. Set the vcpu_period parameter by using the virsh schedinfo command. For example:

      # virsh schedinfo <vm_name> --set vcpu_period=100000

      In this example, the vcpu_period is set to 100,000 microseconds, which means the scheduler enforces vCPU capping during this time interval.

      You can also use the --live --config options to configure a running VM without restarting it.

   2. Set the vcpu_quota parameter by using the virsh schedinfo command. For example:

      # virsh schedinfo <vm_name> --set vcpu_quota=50000

      In this example, the vcpu_quota is set to 50,000 microseconds, which specifies the maximum amount of CPU time that the VM can use during the vcpu_period time interval. In this case, vcpu_quota is set as the half of vcpu_period, so the VM can use up to 50% of the CPU time during that interval.

      You can also use the --live --config options to configure a running VM without restarting it.

Procedure

1. Check the current CPU weight of a VM.

   # virsh schedinfo <vm_name>
   
   Scheduler      : posix
   cpu_shares : 1024
   vcpu_period    : 0
   vcpu_quota     : 0
   emulator_period: 0
   emulator_quota : 0
   global_period  : 0
   global_quota   : 0
   iothread_period: 0
   iothread_quota : 0

2. Adjust the CPU weight to a preferred value.

   # virsh schedinfo <vm_name> --set cpu_shares=2048
   
   Scheduler      : posix
   cpu_shares : 2048
   vcpu_period    : 0
   vcpu_quota     : 0
   emulator_period: 0
   emulator_quota : 0
   global_period  : 0
   global_quota   : 0
   iothread_period: 0
   iothread_quota : 0

   In this example, cpu_shares is set to 2048. This means that if all other VMs have the value set to 1024, this VM gets approximately twice the amount of CPU time.

   You can also use the --live --config options to configure a running VM without restarting it.

Procedure

1. Enable KSM:

   Warning

   Enabling KSM increases CPU utilization and affects overall CPU performance.

   1. Install the ksmtuned service:

      # {PackageManagerCommand} install ksmtuned

   2. Start the service:

      • To enable KSM for a single session, use the systemctl utility to start the ksm and ksmtuned services.

        # systemctl start ksm
        # systemctl start ksmtuned

      • To enable KSM persistently, use the systemctl utility to enable the ksm and ksmtuned services.

        # systemctl enable ksm
        Created symlink /etc/systemd/system/multi-user.target.wants/ksm.service  /usr/lib/systemd/system/ksm.service
        
        # systemctl enable ksmtuned
        Created symlink /etc/systemd/system/multi-user.target.wants/ksmtuned.service  /usr/lib/systemd/system/ksmtuned.service

2. Monitor the performance and resource consumption of VMs on your host to evaluate the benefits of activating KSM. Specifically, ensure that the additional CPU usage by KSM does not offset the memory improvements and does not cause additional performance issues. In latency-sensitive workloads, also pay attention to cross-NUMA page merges.

3. Optional: If KSM has not improved your VM performance, disable it:

   • To disable KSM for a single session, use the systemctl utility to stop ksm and ksmtuned services.

     # systemctl stop ksm
     # systemctl stop ksmtuned

   • To disable KSM persistently, use the systemctl utility to disable ksm and ksmtuned services.

     # systemctl disable ksm
     Removed /etc/systemd/system/multi-user.target.wants/ksm.service.
     # systemctl disable ksmtuned
     Removed /etc/systemd/system/multi-user.target.wants/ksmtuned.service.