Chapter 10. Migrating virtual machines

• Storage located on a shared network
• The --copy-storage-all parameter for the virsh migrate command, which copies disk image contents from the source to the destination over the network.
• Storage area network (SAN) logical units (LUNs).
• Ceph storage clusters

• Live storage migration cannot be performed on RHEL 8, but you can migrate storage while the VM is powered down. Note that live storage migration is available on Red Hat Virtualization.

• VMs that use certain features and configurations will not work correctly if migrated, or the migration will fail. Such features include:

  • Device passthrough
  • SR-IOV device assignment
  • Mediated devices, such as vGPUs
• A migration between hosts that use Non-Uniform Memory Access (NUMA) pinning works only if the hosts have similar topology. However, the performance on running workloads might be negatively affected by the migration.
• Both the source and destination hosts use specific RHEL versions that are supported for VM migration, see Supported hosts for virtual machine migration

• The physical CPUs, both on the source VM and the destination VM, must be identical, otherwise the migration might fail. Any differences between the VMs in the following CPU related areas can cause problems with the migration:

  • CPU model

    • Migrating between an Intel 64 host and an AMD64 host is unsupported, even though they share the x86-64 instruction set.
    • For steps to ensure that a VM will work correctly after migrating to a host with a different CPU model, see Verifying host CPU compatibility for virtual machine migration.
  • Physical machine firmware versions and settings

Prerequisites

• Hypervisor: The source host and the destination host both use the KVM hypervisor.
• Network connection: The source host and the destination host are able to reach each other over the network. Use the ping utility to verify this.

• Open ports: Ensure the following ports are open on the destination host.

  • Port 22 is needed for connecting to the destination host by using SSH.
  • Port 16509 is needed for connecting to the destination host by using TLS.
  • Port 16514 is needed for connecting to the destination host by using TCP.
  • Ports 49152-49215 are needed by QEMU for transferring the memory and disk migration data.
• Hosts: For the migration to be supportable by Red Hat, the source host and destination host must be using specific operating systems and machine types. To ensure this is the case, see Supported hosts for virtual machine migration.
• CPU: The VM must be compatible with the CPU features of the destination host. To ensure this is the case, see Verifying host CPU compatibility for virtual machine migration.

• Storage: The disk images of VMs that will be migrated are accessible to both the source host and the destination host. This is optional for offline migration, but required for migrating a running VM. To ensure storage accessibility for both hosts, one of the following must apply:

  • You are using storage area network (SAN) logical units (LUNs).
  • You are using a Ceph storage clusters.
  • You have created a disk image with the same format and size as the source VM disk and you will use the --copy-storage-all parameter when migrating the VM.
  • The disk image is located on a separate networked location. For instructions to set up such shared VM storage, see Sharing virtual machine disk images with other hosts.

• Network bandwidth: When migrating a running VM, your network bandwidth must be higher than the rate in which the VM generates dirty memory pages.

  To obtain the dirty page rate of your VM before you start the live migration, do the following:

  • Monitor the rate of dirty page generation of the VM for a short period of time.

    # virsh domdirtyrate-calc <example_VM> 30

  • After the monitoring finishes, obtain its results:

    # virsh domstats <example_VM> --dirtyrate
    Domain: 'example-VM'
      dirtyrate.calc_status=2
      dirtyrate.calc_start_time=200942
      dirtyrate.calc_period=30
      dirtyrate.megabytes_per_second=2

    In this example, the VM is generating 2 MB of dirty memory pages per second. Attempting to live-migrate such a VM on a network with a bandwidth of 2 MB/s or less will cause the live migration not to progress if you do not pause the VM or lower its workload.

    To ensure that the live migration finishes successfully, Red Hat recommends that your network bandwidth is significantly greater than the VM’s dirty page generation rate.

    Note

    The value of the calc_period option might differ based on the workload and dirty page rate. You can experiment with several calc_period values to determine the most suitable period that aligns with the dirty page rate in your environment.

• Bridge tap network specifics: When migrating an existing VM in a public bridge tap network, the source and destination hosts must be located on the same network. Otherwise, the VM network will not work after migration.

• libvirtd: Ensure that the libvirtd service is enabled and running.

  # systemctl enable --now libvirtd.service

Offline migration

• The following command migrates a shut-off example-VM VM from your local host to the system connection of the example-destination host by using an SSH tunnel.

  # virsh migrate --offline --persistent <example_VM> qemu+ssh://example-destination/system

Multi-FD live migration

• You can use multiple parallel connections to the destination host during the live migration. This is also known as multiple file descriptors (multi-FD) migration. With multi-FD migration, you can speed up the migration by utilizing all of the available network bandwidth for the migration process.

  # virsh migrate --live --persistent --parallel --parallel-connections 4 <example_VM> qemu+ssh://<example-destination>/system

  This example uses 4 multi-FD channels to migrate the <example_VM> VM. It is recommended to use one channel for each 10 Gbps of available network bandwidth. The default value is 2 channels.

Live migration with an increased downtime limit

• To improve the reliability of a live migration, you can set the maxdowntime parameter, which specifies the maximum amount of time, in milliseconds, the VM can be paused during live migration. Setting a larger downtime can help to ensure the migration completes successfully.

  # virsh migrate-setmaxdowntime <example_VM> <time_interval_in_milliseconds>

Post-copy migration

• If your VM has a large memory footprint, you can perform a post-copy migration, which transfers the source VM’s CPU state first and immediately starts the migrated VM on the destination host. The source VM’s memory pages are transferred after the migrated VM is already running on the destination host. Because of this, a post-copy migration can result in a smaller downtime of the migrated VM.

  However, the running VM on the destination host might try to access memory pages that have not yet been transferred, which causes a page fault. If too many page faults occur during the migration, the performance of the migrated VM can be severely degraded.

  Given the potential complications of a post-copy migration, it is recommended to use the following command that starts a standard live migration and switches to a post-copy migration if the live migration cannot be finished in a specified amount of time.

  # virsh migrate --live --persistent --postcopy --timeout <time_interval_in_seconds> --timeout-postcopy <example_VM> qemu+ssh://<example-destination>/system

Auto-converged live migration

• If your VM is under a heavy memory workload, you can use the --auto-converge option. This option automatically slows down the execution speed of the VM’s CPU. As a consequence, this CPU throttling can help to slow down memory writes, which means the live migration might succeed even in VMs with a heavy memory workload.

  However, the CPU throttling does not help to resolve workloads where memory writes are not directly related to CPU execution speed, and it can negatively impact the performance of the VM during a live migration.

  # virsh migrate --live --persistent --auto-converge <example_VM> qemu+ssh://<example-destination>/system

Verification

• For offline migration:

  • On the destination host, list the available VMs to verify that the VM was migrated successfully.

    # virsh list --all
    Id      Name             State
    ----------------------------------
    10    example-VM-1      shut off

• For live migration:

  • On the destination host, list the available VMs to verify the state of the destination VM:

    # virsh list --all
    Id      Name             State
    ----------------------------------
    10    example-VM-1      running

    If the state of the VM is listed as running, it means that the migration is finished. However, if the live migration is still in progress, the state of the destination VM will be listed as paused.

• For post-copy migration:

  1. On the source host, list the available VMs to verify the state of the source VM.

     # virsh list --all
     Id      Name             State
     ----------------------------------
     10    example-VM-1      shut off

  2. On the destination host, list the available VMs to verify the state of the destination VM.

     # virsh list --all
     Id      Name             State
     ----------------------------------
     10    example-VM-1      running

     If the state of the source VM is listed as shut off and the state of the destination VM is listed as running, it means that the migration is finished.

Prerequisites

• You have installed the RHEL 8 web console.

  For instructions, see Installing and enabling the web console.

• The web console VM plugin is installed on your system.
• Hypervisor: The source host and the destination host both use the KVM hypervisor.
• Hosts: The source and destination hosts are running.

• Open ports: Ensure the following ports are open on the destination host.

  • Port 22 is needed for connecting to the destination host by using SSH.
  • Port 16509 is needed for connecting to the destination host by using TLS.
  • Port 16514 is needed for connecting to the destination host by using TCP.
  • Ports 49152-49215 are needed by QEMU for transfering the memory and disk migration data.
• CPU: The VM must be compatible with the CPU features of the destination host. To ensure this is the case, see Verifying host CPU compatibility for virtual machine migration.

• Storage: The disk images of VMs that will be migrated are accessible to both the source host and the destination host. This is optional for offline migration, but required for migrating a running VM. To ensure storage accessibility for both hosts, one of the following must apply:

  • You are using storage area network (SAN) logical units (LUNs).
  • You are using a Ceph storage clusters.
  • You have created a disk image with the same format and size as the source VM disk and you will use the --copy-storage-all parameter when migrating the VM.
  • The disk image is located on a separate networked location. For instructions to set up such shared VM storage, see Sharing virtual machine disk images with other hosts.

• Network bandwidth: When migrating a running VM, your network bandwidth must be higher than the rate in which the VM generates dirty memory pages.

  To obtain the dirty page rate of your VM before you start the live migration, do the following in your command-line interface:

  1. Monitor the rate of dirty page generation of the VM for a short period of time.

     # virsh domdirtyrate-calc vm-name 30

  2. After the monitoring finishes, obtain its results:

     # virsh domstats vm-name --dirtyrate
     Domain: 'vm-name'
       dirtyrate.calc_status=2
       dirtyrate.calc_start_time=200942
       dirtyrate.calc_period=30
       dirtyrate.megabytes_per_second=2

     In this example, the VM is generating 2 MB of dirty memory pages per second. Attempting to live-migrate such a VM on a network with a bandwidth of 2 MB/s or less will cause the live migration not to progress if you do not pause the VM or lower its workload.

     To ensure that the live migration finishes successfully, Red Hat recommends that your network bandwidth is significantly greater than the VM’s dirty page generation rate.

     Note

     The value of the calc_period option might differ based on the workload and dirty page rate. You can experiment with several calc_period values to determine the most suitable period that aligns with the dirty page rate in your environment.

• Bridge tap network specifics: When migrating an existing VM in a public bridge tap network, the source and destination hosts must be located on the same network. Otherwise, the VM network will not work after migration.

• Confirm whether the VM appears in the list of VMs available on the destination host.
• Start the migrated VM and observe if it boots up.

Prerequisites

• The VM intended for migration is shut down.
• Optional: A host system is available for hosting the storage that is not the source or destination host, but both the source and the destination host can reach it through the network. This is the optimal solution for shared storage and is recommended by Red Hat.
• Make sure that NFS file locking is not used as it is not supported in KVM.
• The NFS protocol is installed and enabled on the source and destination hosts. See Deploying an NFS server.

• The virt_use_nfs SELinux boolean is set to on.

  # setsebool virt_use_nfs 1

Verification

• Start the VM on the source host and observe if it boots successfully.

Prerequisites

• Virtualization is installed and enabled on your system.
• You have administrator access to the source host and the destination host for the migration.

Next steps

• Sharing virtual machine disk images with other hosts
• Migrating a virtual machine by using the command-line interface
• Live-migrating a virtual machine by using the web console