Chapter 5. Migrating virtual machine instances between Compute nodes
You sometimes need to migrate instances from one Compute node to another Compute node in the overcloud, to perform maintenance, rebalance the workload, or replace a failed or failing node.
- Compute node maintenance
- If you need to temporarily take a Compute node out of service, for instance, to perform hardware maintenance or repair, kernel upgrades and software updates, you can migrate instances running on the Compute node to another Compute node.
- Failing Compute node
- If a Compute node is about to fail and you need to service it or replace it, you can migrate instances from the failing Compute node to a healthy Compute node.
- Failed Compute nodes
- If a Compute node has already failed, you can evacuate the instances. You can rebuild instances from the original image on another Compute node, using the same name, UUID, network addresses, and any other allocated resources the instance had before the Compute node failed.
- Workload rebalancing
- You can migrate one or more instances to another Compute node to rebalance the workload. For example, you can consolidate instances on a Compute node to conserve power, migrate instances to a Compute node that is physically closer to other networked resources to reduce latency, or distribute instances across Compute nodes to avoid hot spots and increase resiliency.
Director configures all Compute nodes to provide secure migration. All Compute nodes also require a shared SSH key to provide the users of each host with access to other Compute nodes during the migration process. Director creates this key using the OS::TripleO::Services::NovaCompute
composable service. This composable service is one of the main services included on all Compute roles by default. For more information, see Composable Services and Custom Roles in the Advanced Overcloud Customization guide.
If you have a functioning Compute node, and you want to make a copy of an instance for backup purposes, or to copy the instance to a different environment, follow the procedure in Importing virtual machines into the overcloud in the Director Installation and Usage guide.
5.1. Migration types
Red Hat OpenStack Platform (RHOSP) supports the following types of migration.
Cold migration
Cold migration, or non-live migration, involves shutting down a running instance before migrating it from the source Compute node to the destination Compute node.
Cold migration involves some downtime for the instance. The migrated instance maintains access to the same volumes and IP addresses.
Cold migration requires that both the source and destination Compute nodes are running.
Live migration
Live migration involves moving the instance from the source Compute node to the destination Compute node without shutting it down, and while maintaining state consistency.
Live migrating an instance involves little or no perceptible downtime. In some cases, instances cannot use live migration. For more information, see Migration Constraints.
Live migration requires that both the source and destination Compute nodes are running.
Evacuation
If you need to migrate instances because the source Compute node has already failed, you can evacuate the instances.
5.2. Migration constraints
In some cases, migrating instances involves additional constraints. Migration constraints typically arise with block migration, configuration disks, or when one or more instances access physical hardware on the Compute node.
CPU constraints
The source and destination Compute nodes must have the same CPU architecture. For example, Red Hat does not support migrating an instance from an x86_64
CPU to a ppc64le
CPU. In some cases, the CPU of the source and destination Compute node must match exactly, such as instances that use CPU host passthrough. In all cases, the CPU features of the destination node must be a superset of the CPU features on the source node. Using CPU pinning introduces additional constraints. For more information, see Live migration constraints.
Memory constraints
The destination Compute node must have sufficient available RAM. Memory oversubscription can cause migration to fail. Additionally, instances that use a NUMA topology must have sufficient available RAM on the same NUMA node on the destination Compute node.
Block migration constraints
Migrating instances that use disks that are stored locally on a Compute node takes significantly longer than migrating volume-backed instances that use shared storage, such as Red Hat Ceph Storage. This latency arises because OpenStack Compute (nova) migrates local disks block-by-block between the Compute nodes over the control plane network by default. By contrast, volume-backed instances that use shared storage, such as Red Hat Ceph Storage, do not have to migrate the volumes, because each Compute node already has access to the shared storage.
Network congestion in the control plane network caused by migrating local disks or instances that consume large amounts of RAM might impact the performance of other systems that use the control plane network, such as RabbitMQ.
Read-only drive migration constraints
Migrating a drive is supported only if the drive has both read and write capabilities. For example, OpenStack Compute (nova) cannot migrate a CD-ROM drive or a read-only config drive. However, OpenStack Compute (nova) can migrate a drive with both read and write capabilities, including a config drive with a drive format such as vfat
.
Live migration constraints
- Migration between RHEL minor versions
In general, you can live migrate an instance between RHEL minor versions when the instance machine type version on the source Compute node is equal to or less than that of the destination Compute node. For example, you cannot live migrate an instance with a RHEL-7.6 machine type running on a RHEL-7.6 Compute node to a RHEL-7.5 Compute node, because the RHEL-7.5 Compute node does not know of the RHEL-7.6 machine type.
However, if you do not set a machine type explicitly, the instance receives the default machine type and live migration can succeed across supported RHEL minor versions. For example, you can live migrate an instance that has the default machine type, RHEL-7.6, from a RHEL-7.8 Compute node to a RHEL-7.7 Compute node.
- No new operations during migration
- To achieve state consistency between the copies of the instance on the source and destination nodes, RHOSP must prevent new operations during live migration. Otherwise, live migration might take a long time or potentially never end if writes to memory occur faster than live migration can replicate the state of the memory.
- NUMA, CPU pinning, huge pages and DPDK
OpenStack Compute can live migrate an instance that uses NUMA, CPU pinning or DPDK when the environment meets the following conditions:
-
The destination Compute node must have sufficient capacity on the same NUMA node that the instance uses on the source Compute node. For example, if an instance uses
NUMA 0
onovercloud-compute-0
, to live migrate the instance toovercloud-compute-1
, you must ensure thatovercloud-compute-1
has sufficient capacity onNUMA 0
to support the instance. -
NovaEnableNUMALiveMigration
is set to "True" in the Compute configuration. This parameter is enabled by default only when the Compute host is configured for an OVS-DPDK deployment. -
The
NovaSchedulerDefaultFilters
parameter in the Compute configuration must include the valuesAggregateInstanceExtraSpecsFilter
andNUMATopologyFilter
. - CPU Pinning: When a flavor uses CPU pinning, the flavor implicitly introduces a NUMA topology to the instance and maps its CPUs and memory to specific host CPUs and memory. The difference between a simple NUMA topology and CPU pinning is that NUMA uses a range of CPU cores, whereas CPU pinning uses specific CPU cores. For more information, see Configuring CPU pinning with NUMA. To live migrate instances that use CPU pinning, the destination host must be empty and must have equivalent hardware.
-
Data Plane Development Kit (DPDK): When an instance uses DPDK, such as an instance running Open vSwitch with
dpdk-netdev
, the instance also uses huge pages. Huge pages impose a NUMA topology such that OpenStack Compute (nova) pins the instance to a NUMA node. When you migrate instances that use DPDK, the destination Compute node must have an identical hardware specification and configuration as the source Compute node. Additionally, there must not be any instances running on the destination Compute node to ensure that it preserves the NUMA topology of the source Compute node.
-
The destination Compute node must have sufficient capacity on the same NUMA node that the instance uses on the source Compute node. For example, if an instance uses
Constraints that preclude live migration
Live migration is not possible when the instance is configured for the following features:
- Single-root Input/Output Virtualization (SR-IOV): You can assign SR-IOV Virtual Functions (VFs) to instances. However, this prevents live migration. Unlike a regular network device, an SR-IOV VF network device does not have a permanent unique MAC address. The VF network device receives a new MAC address each time the Compute node reboots, or when the scheduler migrates the instance to a new Compute node. Consequently, OpenStack Compute cannot live migrate instances that use SR-IOV. You must cold migrate instances that use SR-IOV.
- PCI passthrough: QEMU/KVM hypervisors support attaching PCI devices on the Compute node to an instance. Use PCI passthrough to give an instance exclusive access to PCI devices, which appear and behave as if they are physically attached to the operating system of the instance. However, because PCI passthrough involves physical addresses, OpenStack Compute does not support live migration of instances using PCI passthrough.
5.3. Preparing to migrate
Before you migrate one or more instances, you need to determine the Compute node names and the IDs of the instances to migrate.
Procedure
Identify the source Compute node host name and the destination Compute node host name:
(undercloud)$ source ~/overcloudrc (overcloud)$ openstack compute service list
List the instances on the source Compute node and locate the ID of the instance or instances that you want to migrate:
(overcloud)$ openstack server list --host <source> --all-projects
Replace
<source>
with the name or ID of the source Compute node.Optional: To shut down the source Compute node for maintenance, disable the source Compute node from the undercloud to ensure that the scheduler does not attempt to assign new instances to the source Compute node during maintenance:
(overcloud)$ source ~/stackrc (undercloud)$ openstack compute service set <source> nova-compute --disable
Replace
<source>
with the name or ID of the source Compute node.
If you are not migrating NUMA, CPU-pinned or DPDK instances, you are now ready to perform the migration. Follow the required procedure detailed in Cold migrating an instance or Live migrating an instance.
If you are migrating NUMA, CPU-pinned or DPDK instances, you need to prepare the destination node. Complete the procedure detailed in Section 5.4, “Additional preparation for DPDK instances”.
5.4. Additional preparation for DPDK instances
If you are migrating NUMA, CPU-pinned or DPDK instances, you need to prepare the destination node.
Procedure
If the destination Compute node for NUMA, CPU-pinned or DPDK instances is not disabled, disable it to prevent the scheduler from assigning instances to the node:
(overcloud)$ openstack compute service set <dest> nova-compute --disable
Replace
<dest>
with the name or ID of the destination Compute node.Ensure that the destination Compute node has no instances, except for instances that you previously migrated from the source Compute node when you migrated multiple DPDK or NUMA instances:
(overcloud)$ openstack server list --host <dest> --all-projects
Replace
<dest>
with the name or ID of the destination Compute node.Ensure that the destination Compute node has sufficient resources to run the NUMA, CPU-pinned or DPDK instance:
(overcloud)$ openstack host show <dest> $ ssh <dest> $ numactl --hardware $ exit
Replace
<dest>
with the name or ID of the destination Compute node.To discover NUMA information about the source or destination Compute nodes, run the following commands:
$ ssh root@overcloud-compute-n # lscpu && lscpu | grep NUMA # virsh nodeinfo # virsh capabilities # exit
Use
ssh
to connect toovercloud-compute-n
whereovercloud-compute-n
is the source or destination Compute node.If you do not know if an instance uses NUMA, check the flavor of the instance:
(overcloud)$ openstack server list -c Name -c Flavor --name <vm> (overcloud)$ openstack flavor show <flavor>
-
Replace
<vm>
with the name or ID of the instance. Replace
<flavor>
with the name or ID of the flavor.-
If the
properties
field includeshw:mem_page_size
with a value other thanany
, such as2MB
,2048
or1GB
, the instance has a NUMA topology. -
If the
properties
field includesaggregate_instance_extra_specs:pinned='true'
, the instance uses CPU pinning. -
If the
properties
field includeshw:numa_nodes
, the OpenStack Compute (nova) service restricts the instance to a specific NUMA node.
-
If the
-
Replace
For each instance that uses NUMA, you can retrieve information about the NUMA topology from the underlying Compute node so that you can verify that the NUMA topology on the destination Compute node reflects the NUMA topology of the source Compute node after migration is complete. You can use the following commands to perform this check:
To view details about NUMA and CPU pinning, run the following command:
$ ssh root@overcloud-compute-n # virsh vcpuinfo <vm>
Replace
<vm>
with the name of the instance.To view details about which NUMA node the instance is using, run the following command:
$ ssh root@overcloud-compute-n # virsh numatune <vm>
Replace
<vm>
with the name of the instance.
5.5. Cold migrating an instance
Cold migrating an instance involves stopping the instance and moving it to another Compute node. Cold migration facilitates migration scenarios that live migrating cannot facilitate, such as migrating instances that use PCI passthrough or Single-Root Input/Output Virtualization (SR-IOV). The scheduler automatically selects the destination Compute node. For more information, see Migration Constraints.
During cold migrations, the Compute service rebuilds the migrated instances from scratch, and adjusts the machine type to the machine type of the destination Compute node. Therefore, if you cold migrate an instance with a RHEL-7.5 machine type running on a RHEL-7.5 Compute node, to a RHEL-7.6 Compute node, the migrated instance on the destination Compute node will have a RHEL-7.6 machine type.
Procedure
To cold migrate an instance, run the following command to power off and move the instance:
(overcloud)$ openstack server migrate <vm> --wait
-
Replace
<vm>
with the name or ID of the instance to migrate. -
Specify the
--block-migration
flag if migrating a locally stored volume.
-
Replace
- Wait for migration to complete. See Checking migration status to check the status of the migration.
Check the status of the instance:
(overcloud)$ openstack server list --all-projects
A status of "VERIFY_RESIZE" indicates you need to confirm or revert the migration:
If the migration worked as expected, confirm it:
(overcloud)$ openstack server resize --confirm <vm>`
Replace
<vm>
with the name or ID of the instance to migrate. A status of "ACTIVE" indicates that the instance is ready to use.If the migration did not work as expected, revert it:
(overcloud)$ openstack server resize --revert <vm>`
Replace
<vm>
with the name or ID of the instance.
Restart the instance:
(overcloud)$ openstack server start <vm>
Replace
<vm>
with the name or ID of the instance.
When you finish migrating the instances, proceed to Completing the migration.
5.6. Live migrating an instance
Live migration moves an instance from a source Compute node to a destination Compute node with a minimal amount of downtime. Live migration might not be appropriate for all instances. For more information, see Section 5.2, “Migration constraints”.
Live migrations preserve the instance machine type on the destination Compute node. Therefore, if you live migrate an instance with a RHEL-7.5 machine type running on a RHEL-7.5 Compute node, to a RHEL-7.6 Compute node, the migrated instance on the destination Compute node retains the RHEL-7.5 machine type. To change the machine type, you must set the image metadata property hw_machine_type
, or set the NovaHWMachineType
parameter on each Compute node.
Procedure
To live migrate an instance, specify the instance and the destination Compute node:
(overcloud)$ openstack server migrate <vm> --live <dest> --wait
-
Replace
<vm>
with the name or ID of the instance. Replace
<dest>
with the name or ID of the destination Compute node.NoteThe
openstack server migrate
command covers migrating instances with shared storage, which is the default. Specify the--block-migration
flag to migrate a locally stored volume:(overcloud)$ openstack server migrate <vm> --live <dest> --wait --block-migration
-
Replace
Confirm that the instance is migrating:
$ openstack server show <vm> +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | ... | ... | | status | MIGRATING | | ... | ... | +----------------------+--------------------------------------+
- Wait for migration to complete. See Checking migration status to check the status of the migration.
Check the status of the instance to confirm if the migration was successful:
(overcloud)$ openstack server list --host <dest> --all-projects
Replace
<dest>
with the name or ID of the destination Compute node.Optional: For instances that use NUMA, CPU-pinning, or DPDK, retrieve information about the NUMA topology from a Compute node to compare it with the NUMA topology that you retrieved during the preparing to migrate procedure. Comparing the NUMA topologies of the source and destination Compute nodes ensures that the source and destination Compute nodes use the same NUMA topology.
To view details about NUMA and CPU pinning, run the following command:
$ ssh root@overcloud-compute-n # virsh vcpuinfo <vm>
-
Replace
overcloud-compute-n
with the host name of the Compute node. -
Replace
<vm>
with the name of the instance.
-
Replace
To view details about which NUMA node the instance is using, run the following command:
$ ssh root@overcloud-compute-n # virsh numatune <vm>
-
Replace
overcloud-compute-n
with the host name of the Compute node. -
Replace
<vm>
with the name or ID of the instance.
-
Replace
When you finish migrating the instances, proceed to Completing the migration.
5.7. Checking migration status
Migration involves several state transitions before migration is complete. During a healthy migration, the migration state typically transitions as follows:
- Queued: The Compute service has accepted the request to migrate an instance, and migration is pending.
- Preparing: The Compute service is preparing to migrate the instance.
- Running: The Compute service is migrating the instance.
- Post-migrating: The Compute service has built the instance on the destination Compute node and is releasing resources on the source Compute node.
- Completed: The Compute service has completed migrating the instance and finished releasing resources on the source Compute node.
Procedure
Retrieve the list of migration IDs for the instance:
$ nova server-migration-list <vm> +----+-------------+----------- (...) | Id | Source Node | Dest Node | (...) +----+-------------+-----------+ (...) | 2 | - | - | (...) +----+-------------+-----------+ (...)
Replace
<vm>
with the name or ID of the instance.Show the status of the migration:
$ nova server-migration-show <vm> <migration-id>
-
Replace
<vm>
with the name or ID of the instance. Replace
<migration-id>
with the ID of the migration.Running the
nova server-migration-show
command returns the following example output:+------------------------+--------------------------------------+ | Property | Value | +------------------------+--------------------------------------+ | created_at | 2017-03-08T02:53:06.000000 | | dest_compute | controller | | dest_host | - | | dest_node | - | | disk_processed_bytes | 0 | | disk_remaining_bytes | 0 | | disk_total_bytes | 0 | | id | 2 | | memory_processed_bytes | 65502513 | | memory_remaining_bytes | 786427904 | | memory_total_bytes | 1091379200 | | server_uuid | d1df1b5a-70c4-4fed-98b7-423362f2c47c | | source_compute | compute2 | | source_node | - | | status | running | | updated_at | 2017-03-08T02:53:47.000000 | +------------------------+--------------------------------------+
TipThe OpenStack Compute service measures progress of the migration by the number of remaining memory bytes to copy. If this number does not decrease over time, the migration might be unable to complete, and the Compute service might abort it.
-
Replace
Sometimes instance migration can take a long time or encounter errors. For more information, see Section 5.10, “Troubleshooting migration”.
5.8. Completing the migration
After you migrate one or more instances, you need to re-enable the source Compute nodes from the undercloud to ensure that the scheduler can assign new instances to the source Compute node. For migrated instances that use DPDK, you must also re-enable the destination Compute node from the undercloud.
Procedure
Re-enable the source Compute node:
(overcloud)$ source ~/stackrc (undercloud)$ openstack compute service set <source> nova-compute --enable
Replace
<source>
with the host name of the source Compute node.Optional: For instances that use DPDK, re-enable the destination Compute node from the undercloud:
(undercloud)$ openstack compute service set <dest> nova-compute --enable
Replace
<dest>
with the host name of the destination Compute node.
5.9. Evacuating an instance
If you want to move an instance from a dead or shut-down Compute node to a new host in the same environment, you can evacuate it. The evacuate process rebuilds the instance on another Compute node. If the instance uses shared storage, the instance root disk is not rebuilt during the evacuate process, as the disk remains accessible by the destination Compute node. If the instance does not use shared storage, then the instance root disk is also rebuilt on the destination Compute node.
-
You can only perform an evacuation when the Compute node is fenced, and the API reports that the state of the Compute node is "down" or "forced-down". If the Compute node is not reported as "down" or "forced-down", the
evacuate
command fails. - To perform an evacuation, you must be a cloud administrator.
- During evacuations, the Compute service rebuilds the evacuated instances from scratch, and adjusts the machine type to the machine type of the destination Compute node. Therefore, if you evacuate an instance with a RHEL-7.5 machine type running on a RHEL-7.5 Compute node, to a RHEL-7.6 Compute node, the migrated instance on the destination Compute node will have a RHEL-7.6 machine type.
5.9.1. Evacuating one instance
You can evacuate instances one at a time.
Procedure
- Log onto the failed Compute node as an administrator.
Disable the Compute node:
(overcloud)[stack@director ~]$ openstack compute service set \ <host> <service> --disable
-
Replace
<host>
with the name of the Compute node to evacuate the instance from. -
Replace
<service>
with the name of the service to disable, for examplenova-compute
.
-
Replace
To evacuate an instance, run the following command:
(overcloud)[stack@director ~]$ nova evacuate [--password <pass>] <vm> [dest]
-
Replace
<pass>
with the admin password to set for the evacuated instance. If a password is not specified, a random password is generated and output when the evacuation is complete. -
Replace
<vm>
with the name or ID of the instance to evacuate. Replace
[dest]
with the name of the Compute node to evacuate the instance to. If you do not specify the destination Compute node, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:(overcloud)[stack@director ~]$ openstack hypervisor list
-
Replace
5.9.2. Evacuating all instances on a host
You can evacuate all instances on a specified Compute node.
Procedure
- Log onto the failed Compute node as an administrator.
Disable the Compute node:
(overcloud)[stack@director ~]$ openstack compute service set \ <host> <service> --disable
-
Replace
<host>
with the name of the Compute node to evacuate the instances from. -
Replace
<service>
with the name of the service to disable, for examplenova-compute
.
-
Replace
Evacuate all instances on a specified Compute node:
(overcloud)[stack@director ~]$ nova host-evacuate [--target_host <dest>] [--force] <host>
Replace
<dest>
with the name of the destination Compute node to evacuate the instances to. If you do not specify the destination, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:(overcloud)[stack@director ~]$ openstack hypervisor list
-
Replace
<host>
with the name of the Compute node to evacuate the instances from.
5.10. Troubleshooting migration
The following issues can arise during instance migration:
- The migration process encounters errors.
- The migration process never ends.
- Performance of the instance degrades after migration.
5.10.1. Errors during migration
The following issues can send the migration operation into an error
state:
- Running a cluster with different versions of Red Hat OpenStack Platform (RHOSP).
- Specifying an instance ID that cannot be found.
-
The instance you are trying to migrate is in an
error
state. - The Compute service is shutting down.
- A race condition occurs.
-
Live migration enters a
failed
state.
When live migration enters a failed
state, it is typically followed by an error
state. The following common issues can cause a failed
state:
- A destination Compute host is not available.
- A scheduler exception occurs.
- The rebuild process fails due to insufficient computing resources.
- A server group check fails.
- The instance on the source Compute node gets deleted before migration to the destination Compute node is complete.
5.10.2. Never-ending live migration
Live migration can fail to complete, which leaves migration in a perpetual running
state. A common reason for a live migration that never completes is that client requests to the instance running on the source Compute node create changes that occur faster than the Compute service can replicate them to the destination Compute node.
Use one of the following methods to address this situation:
- Abort the live migration.
- Force the live migration to complete.
Aborting live migration
If the instance state changes faster than the migration procedure can copy it to the destination node, and you do not want to temporarily suspend the instance operations, you can abort the live migration.
Procedure
Retrieve the list of migrations for the instance:
$ nova server-migration-list <vm>
Replace
<vm>
with the name or ID of the instance.Abort the live migration:
$ nova live-migration-abort <vm> <migration-id>
-
Replace
<vm>
with the name or ID of the instance. -
Replace
<migration-id>
with the ID of the migration.
-
Replace
Forcing live migration to complete
If the instance state changes faster than the migration procedure can copy it to the destination node, and you want to temporarily suspend the instance operations to force migration to complete, you can force the live migration procedure to complete.
Forcing live migration to complete might lead to perceptible downtime.
Procedure
Retrieve the list of migrations for the instance:
$ nova server-migration-list <vm>
Replace
<vm>
with the name or ID of the instance.Force the live migration to complete:
$ nova live-migration-force-complete <vm> <migration-id>
-
Replace
<vm>
with the name or ID of the instance. -
Replace
<migration-id>
with the ID of the migration.
-
Replace
5.10.3. Instance performance degrades after migration
For instances that use a NUMA topology, the source and destination Compute nodes must have the same NUMA topology and configuration. The NUMA topology of the destination Compute node must have sufficient resources available. If the NUMA configuration between the source and destination Compute nodes is not the same, it is possible that live migration succeeds while the instance performance degrades. For example, if the source Compute node maps NIC 1 to NUMA node 0, but the destination Compute node maps NIC 1 to NUMA node 5, after migration the instance might route network traffic from a first CPU across the bus to a second CPU with NUMA node 5 to route traffic to NIC 1. This can result in expected behavior, but degraded performance. Similarly, if NUMA node 0 on the source Compute node has sufficient available CPU and RAM, but NUMA node 0 on the destination Compute node already has instances using some of the resources, the instance might run correctly but suffer performance degradation. For more information, see Section 5.2, “Migration constraints”.