Chapter 12. Migrating virtual machines

• Storage located on a shared network
• The --copy-storage-all parameter for the virsh migrate command, which copies disk image contents from the source to the destination over the network.
• Storage area network (SAN) logical units (LUNs).
• Ceph storage clusters

• VMs that use certain features and configurations will not work correctly if migrated, or the migration will fail. Such features include:

  • Device passthrough
  • SR-IOV device assignment (With the exception of migrating a VM with an attached virtual function of a Mellanox networking device, which works correctly.)
  • Mediated devices, such as vGPUs (With the exception of migrating a VM with an attached NVIDIA vGPU, which works correctly.)
• A migration between hosts that use Non-Uniform Memory Access (NUMA) pinning works only if the hosts have similar topology. However, the performance on running workloads might be negatively affected by the migration.
• Both the source and destination hosts use specific RHEL versions that are supported for VM migration, see Supported hosts for virtual machine migration

• The physical CPUs, both on the source VM and the destination VM, must be identical, otherwise the migration might fail. Any differences between the VMs in the following CPU related areas can cause problems with the migration:

  • CPU model

    • Migrating between an Intel 64 host and an AMD64 host is unsupported, even though they share the x86-64 instruction set.
    • For steps to ensure that a VM will work correctly after migrating to a host with a different CPU model, see Verifying host CPU compatibility for virtual machine migration.
  • Physical machine firmware versions and settings

Prerequisites

• Hypervisor: The source host and the destination host both use the KVM hypervisor.
• Network connection: The source host and the destination host are able to reach each other over the network. Use the ping utility to verify this.

• Open ports: Ensure the following ports are open on the destination host.

  • Port 22 is needed for connecting to the destination host by using SSH.
  • Port 16509 is needed for connecting to the destination host by using TLS.
  • Port 16514 is needed for connecting to the destination host by using TCP.
  • Ports 49152-49215 are needed by QEMU for transferring the memory and disk migration data.
• Hosts: For the migration to be supportable by Red Hat, the source host and destination host must be using specific operating systems and machine types. To ensure this is the case, see Supported hosts for virtual machine migration.
• CPU: The VM must be compatible with the CPU features of the destination host. To ensure this is the case, see Verifying host CPU compatibility for virtual machine migration.

• Storage: The disk images of VMs that will be migrated are accessible to both the source host and the destination host. This is optional for offline migration, but required for migrating a running VM. To ensure storage accessibility for both hosts, one of the following must apply:

  • You are using storage area network (SAN) logical units (LUNs).
  • You are using a Ceph storage clusters.
  • You have created a disk image with the same format and size as the source VM disk and you will use the --copy-storage-all parameter when migrating the VM.
  • The disk image is located on a separate networked location. For instructions to set up such shared VM storage, see Sharing virtual machine disk images with other hosts.

• Network bandwidth: When migrating a running VM, your network bandwidth must be higher than the rate in which the VM generates dirty memory pages.

  To obtain the dirty page rate of your VM before you start the live migration, do the following:

  • Monitor the rate of dirty page generation of the VM for a short period of time.

    # virsh domdirtyrate-calc <example_VM> 30

  • After the monitoring finishes, obtain its results:

    # virsh domstats <example_VM> --dirtyrate
    Domain: 'example-VM'
      dirtyrate.calc_status=2
      dirtyrate.calc_start_time=200942
      dirtyrate.calc_period=30
      dirtyrate.megabytes_per_second=2

    In this example, the VM is generating 2 MB of dirty memory pages per second. Attempting to live-migrate such a VM on a network with a bandwidth of 2 MB/s or less will cause the live migration not to progress if you do not pause the VM or lower its workload.

    To ensure that the live migration finishes successfully, Red Hat recommends that your network bandwidth is significantly greater than the VM’s dirty page generation rate.

    Note

    The value of the calc_period option might differ based on the workload and dirty page rate. You can experiment with several calc_period values to determine the most suitable period that aligns with the dirty page rate in your environment.

• Bridge tap network specifics: When migrating an existing VM in a public bridge tap network, the source and destination hosts must be located on the same network. Otherwise, the VM network will not work after migration.

• Connection protocol: When performing a VM migration, the virsh client on the source host can use one of several protocols to connect to the libvirt daemon on the destination host. Examples in the following procedure use an SSH connection, but you can choose a different one.

  • If you want libvirt to use an SSH connection, ensure that the virtqemud socket is enabled and running on the destination host.

    # systemctl enable --now virtqemud.socket

  • If you want libvirt to use a TLS connection, ensure that the virtproxyd-tls socket is enabled and running on the destination host.

    # systemctl enable --now virtproxyd-tls.socket

  • If you want libvirt to use a TCP connection, ensure that the virtproxyd-tcp socket is enabled and running on the destination host.

    # systemctl enable --now virtproxyd-tcp.socket

Offline migration

• The following command migrates a shut-off example-VM VM from your local host to the system connection of the example-destination host by using an SSH tunnel.

  # virsh migrate --offline --persistent <example_VM> qemu+ssh://example-destination/system

Multi-FD live migration

• You can use multiple parallel connections to the destination host during the live migration. This is also known as multiple file descriptors (multi-FD) migration. With multi-FD migration, you can speed up the migration by utilizing all of the available network bandwidth for the migration process.

  # virsh migrate --live --persistent --parallel --parallel-connections 4 <example_VM> qemu+ssh://<example-destination>/system

  This example uses 4 multi-FD channels to migrate the <example_VM> VM. It is recommended to use one channel for each 10 Gbps of available network bandwidth. The default value is 2 channels.

Live migration with an increased downtime limit

• To improve the reliability of a live migration, you can set the maxdowntime parameter, which specifies the maximum amount of time, in milliseconds, the VM can be paused during live migration. Setting a larger downtime can help to ensure the migration completes successfully.

  # virsh migrate-setmaxdowntime <example_VM> <time_interval_in_milliseconds>

Post-copy migration

• If your VM has a large memory footprint, you can perform a post-copy migration, which transfers the source VM’s CPU state first and immediately starts the migrated VM on the destination host. The source VM’s memory pages are transferred after the migrated VM is already running on the destination host. Because of this, a post-copy migration can result in a smaller downtime of the migrated VM.

  However, the running VM on the destination host might try to access memory pages that have not yet been transferred, which causes a page fault. If too many page faults occur during the migration, the performance of the migrated VM can be severely degraded.

  Given the potential complications of a post-copy migration, it is recommended to use the following command that starts a standard live migration and switches to a post-copy migration if the live migration cannot be finished in a specified amount of time.

  # virsh migrate --live --persistent --postcopy --timeout <time_interval_in_seconds> --timeout-postcopy <example_VM> qemu+ssh://<example-destination>/system

Auto-converged live migration

• If your VM is under a heavy memory workload, you can use the --auto-converge option. This option automatically slows down the execution speed of the VM’s CPU. As a consequence, this CPU throttling can help to slow down memory writes, which means the live migration might succeed even in VMs with a heavy memory workload.

  However, the CPU throttling does not help to resolve workloads where memory writes are not directly related to CPU execution speed, and it can negatively impact the performance of the VM during a live migration.

  # virsh migrate --live --persistent --auto-converge <example_VM> qemu+ssh://<example-destination>/system

Verification

• For offline migration:

  • On the destination host, list the available VMs to verify that the VM was migrated successfully.

    # virsh list --all
    Id      Name             State
    ----------------------------------
    10    example-VM-1      shut off

• For live migration:

  • On the destination host, list the available VMs to verify the state of the destination VM:

    # virsh list --all
    Id      Name             State
    ----------------------------------
    10    example-VM-1      running

    If the state of the VM is listed as running, it means that the migration is finished. However, if the live migration is still in progress, the state of the destination VM will be listed as paused.

• For post-copy migration:

  1. On the source host, list the available VMs to verify the state of the source VM.

     # virsh list --all
     Id      Name             State
     ----------------------------------
     10    example-VM-1      shut off

  2. On the destination host, list the available VMs to verify the state of the destination VM.

     # virsh list --all
     Id      Name             State
     ----------------------------------
     10    example-VM-1      running

     If the state of the source VM is listed as shut off and the state of the destination VM is listed as running, it means that the migration is finished.

Prerequisites

• You have installed the RHEL 9 web console.

  For instructions, see Installing and enabling the web console.

• The web console VM plugin is installed on your system.
• Hypervisor: The source host and the destination host both use the KVM hypervisor.
• Hosts: The source and destination hosts are running.

• Open ports: Ensure the following ports are open on the destination host.

  • Port 22 is needed for connecting to the destination host by using SSH.
  • Port 16509 is needed for connecting to the destination host by using TLS.
  • Port 16514 is needed for connecting to the destination host by using TCP.
  • Ports 49152-49215 are needed by QEMU for transfering the memory and disk migration data.
• CPU: The VM must be compatible with the CPU features of the destination host. To ensure this is the case, see Verifying host CPU compatibility for virtual machine migration.

• Storage: The disk images of VMs that will be migrated are accessible to both the source host and the destination host. This is optional for offline migration, but required for migrating a running VM. To ensure storage accessibility for both hosts, one of the following must apply:

  • You are using storage area network (SAN) logical units (LUNs).
  • You are using a Ceph storage clusters.
  • You have created a disk image with the same format and size as the source VM disk and you will use the --copy-storage-all parameter when migrating the VM.
  • The disk image is located on a separate networked location. For instructions to set up such shared VM storage, see Sharing virtual machine disk images with other hosts.

• Network bandwidth: When migrating a running VM, your network bandwidth must be higher than the rate in which the VM generates dirty memory pages.

  To obtain the dirty page rate of your VM before you start the live migration, do the following in your command-line interface:

  1. Monitor the rate of dirty page generation of the VM for a short period of time.

     # virsh domdirtyrate-calc vm-name 30

  2. After the monitoring finishes, obtain its results:

     # virsh domstats vm-name --dirtyrate
     Domain: 'vm-name'
       dirtyrate.calc_status=2
       dirtyrate.calc_start_time=200942
       dirtyrate.calc_period=30
       dirtyrate.megabytes_per_second=2

     In this example, the VM is generating 2 MB of dirty memory pages per second. Attempting to live-migrate such a VM on a network with a bandwidth of 2 MB/s or less will cause the live migration not to progress if you do not pause the VM or lower its workload.

     To ensure that the live migration finishes successfully, Red Hat recommends that your network bandwidth is significantly greater than the VM’s dirty page generation rate.

     Note

     The value of the calc_period option might differ based on the workload and dirty page rate. You can experiment with several calc_period values to determine the most suitable period that aligns with the dirty page rate in your environment.

• Bridge tap network specifics: When migrating an existing VM in a public bridge tap network, the source and destination hosts must be located on the same network. Otherwise, the VM network will not work after migration.

• Confirm whether the VM appears in the list of VMs available on the destination host.
• Start the migrated VM and observe if it boots up.

• Calculating dirty memory page rate generation of the VM.

  Currently, when migrating a VM with an attached Mellanox VF, live migration data and statistics provided by virsh domjobinfo and virsh domdirtyrate-calc commands are inaccurate, because the calculations only count guest RAM without including the impact of the attached VF.

• Using a post-copy live migration.
• Using a virtual I/O Memory Management Unit (vIOMMU) device in the VM.

Prerequisites

• You have a Mellanox CX-7 networking device with a firmware version that is equal to or greater than 28.36.1010.

  Refer to Mellanox documentation for details about firmware versions.

• The mstflint package is installed on both the source and destination host:

  # dnf install mstflint

• The Mellanox CX-7 networking device has VF_MIGRATION_MODE set to MIGRATION_ENABLED:

  # mstconfig -d <device_pci_address> query | grep -i VF_migration
  
  VF_MIGRATION_MODE                           MIGRATION_ENABLED(2)

  • You can set VF_MIGRATION_MODE to MIGRATION_ENABLED by using the following command:

    # mstconfig -d <device_pci_address> set VF_MIGRATION_MODE=2

• The openvswitch package is installed on both the source and destination host:

  # dnf install openvswitch

• All of the general SR-IOV devices prerequisites. For details, see Attaching SR-IOV networking devices to virtual machines
• All of the general VM migration prerequisites. For details, see Migrating a virtual machine by using the command-line interface

Prerequisites

• You have an NVIDIA GPU with an NVIDIA Virtual GPU Software Driver version that supports this functionality. Refer to the relevant NVIDIA vGPU documentation for more details.

• You have a correctly configured NVIDIA vGPU assigned to a VM. For instructions, see: Setting up NVIDIA vGPU devices

  Note

  It is also possible to live migrate a VM with multiple vGPU devices attached.

• The host uses RHEL 9.4 or later as the operating system.
• All of the vGPU migration prerequisites that are documented by NVIDIA. Refer to the relevant NVIDIA vGPU documentation for more details.
• All of the general VM migration prerequisites. For details, see Migrating a virtual machine by using the command-line interface

Limitations

• Certain NVIDIA GPU features can disable the migration. For more information, see the specific NVIDIA documentation for your graphics card.
• Some GPU workloads are not compatible with the downtime that happens during a migration. As a consequence, the GPU workloads might stop or crash. It is recommended to test if your workloads are compatible with the downtime before attempting a vGPU live migration.
• Currently, vGPU live migration fails if the vGPU driver version differs on the source and destination hosts.

• Currently, some general virtualization features cannot be used when live migrating a VM with an attached vGPU:

  • Calculating dirty memory page rate generation of the VM.

    Currently, live migration data and statistics provided by virsh domjobinfo and virsh domdirtyrate-calc commands are inaccurate when migrating a VM with an attached vGPU, because the calculations only count guest RAM without including vRAM from the vGPU.

  • Using a post-copy live migration.
  • Using a virtual I/O Memory Management Unit (vIOMMU) device in the VM.

Procedure

• For instructions on how to proceed with the live migration, see: Migrating a virtual machine by using the command-line interface

Prerequisites

• The VM intended for migration is shut down.
• Optional: A host system is available for hosting the storage that is not the source or destination host, but both the source and the destination host can reach it through the network. This is the optimal solution for shared storage and is recommended by Red Hat.
• Make sure that NFS file locking is not used as it is not supported in KVM.
• The NFS protocol is installed and enabled on the source and destination hosts. See Deploying an NFS server.

• The virt_use_nfs SELinux boolean is set to on.

  # setsebool virt_use_nfs 1

Verification

• Start the VM on the source host and observe if it boots successfully.

Prerequisites

• Virtualization is installed and enabled on your system.
• You have administrator access to the source host and the destination host for the migration.

Next steps

• Sharing virtual machine disk images with other hosts
• Migrating a virtual machine by using the command-line interface
• Live-migrating a virtual machine by using the web console