Chapter 5. Migrating virtual machine instances between Compute nodes

Procedure

1. Identify the source Compute node host name and the destination Compute node host name:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack compute service list

2. List the instances on the source Compute node and locate the ID of the instance or instances that you want to migrate:

   (overcloud)$ openstack server list --host <source> --all-projects

   Replace <source> with the name or ID of the source Compute node.

3. Optional: To shut down the source Compute node for maintenance, disable the source Compute node from the undercloud to ensure that the scheduler does not attempt to assign new instances to the source Compute node during maintenance:

   (overcloud)$ source ~/stackrc
   (undercloud)$ openstack compute service set <source> nova-compute --disable

   Replace <source> with the name or ID of the source Compute node.

Procedure

1. If the destination Compute node for NUMA, CPU-pinned or DPDK instances is not disabled, disable it to prevent the scheduler from assigning instances to the node:

   (overcloud)$ openstack compute service set <dest> nova-compute --disable

   Replace <dest> with the name or ID of the destination Compute node.

2. Ensure that the destination Compute node has no instances, except for instances that you previously migrated from the source Compute node when you migrated multiple DPDK or NUMA instances:

   (overcloud)$ openstack server list --host <dest> --all-projects

   Replace <dest> with the name or ID of the destination Compute node.

3. Ensure that the destination Compute node has sufficient resources to run the NUMA, CPU-pinned or DPDK instance:

   (overcloud)$ openstack host show <dest>
   $ ssh <dest>
   $ numactl --hardware
   $ exit

   Replace <dest> with the name or ID of the destination Compute node.

4. To discover NUMA information about the source or destination Compute nodes, run the following commands:

   $ ssh root@overcloud-compute-n
   # lscpu && lscpu | grep NUMA
   # virsh nodeinfo
   # virsh capabilities
   # exit

   Use ssh to connect to overcloud-compute-n where overcloud-compute-n is the source or destination Compute node.

5. If you do not know if an instance uses NUMA, check the flavor of the instance:

   (overcloud)$ openstack server list -c Name -c Flavor --name <vm>
   (overcloud)$ openstack flavor show <flavor>

   • Replace <vm> with the name or ID of the instance.

   • Replace <flavor> with the name or ID of the flavor.

     • If the properties field includes hw:mem_page_size with a value other than any, such as 2MB, 2048 or 1GB, the instance has a NUMA topology.
     • If the properties field includes aggregate_instance_extra_specs:pinned='true', the instance uses CPU pinning.
     • If the properties field includes hw:numa_nodes, the OpenStack Compute (nova) service restricts the instance to a specific NUMA node.

6. For each instance that uses NUMA, you can retrieve information about the NUMA topology from the underlying Compute node so that you can verify that the NUMA topology on the destination Compute node reflects the NUMA topology of the source Compute node after migration is complete. You can use the following commands to perform this check:

   • To view details about NUMA and CPU pinning, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh vcpuinfo <vm>

     Replace <vm> with the name of the instance.

   • To view details about which NUMA node the instance is using, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh numatune <vm>

     Replace <vm> with the name of the instance.

Procedure

1. To cold migrate an instance, run the following command to power off and move the instance:

   (overcloud)$ openstack server migrate <vm> --wait

   • Replace <vm> with the name or ID of the instance to migrate.
   • Specify the --block-migration flag if migrating a locally stored volume.
2. Wait for migration to complete. See Checking migration status to check the status of the migration.

3. Check the status of the instance:

   (overcloud)$ openstack server list --all-projects

   A status of "VERIFY_RESIZE" indicates you need to confirm or revert the migration:

   • If the migration worked as expected, confirm it:

     (overcloud)$ openstack server resize --confirm <vm>`

     Replace <vm> with the name or ID of the instance to migrate. A status of "ACTIVE" indicates that the instance is ready to use.

   • If the migration did not work as expected, revert it:

     (overcloud)$ openstack server resize --revert <vm>`

     Replace <vm> with the name or ID of the instance.

4. Restart the instance:

   (overcloud)$ openstack server start <vm>

   Replace <vm> with the name or ID of the instance.

Procedure

1. To live migrate an instance, specify the instance and the destination Compute node:

   (overcloud)$ openstack server migrate <vm> --live <dest> --wait

   • Replace <vm> with the name or ID of the instance.

   • Replace <dest> with the name or ID of the destination Compute node.

     Note

     The openstack server migrate command covers migrating instances with shared storage, which is the default. Specify the --block-migration flag to migrate a locally stored volume:

     (overcloud)$ openstack server migrate <vm> --live <dest> --wait --block-migration

2. Confirm that the instance is migrating:

   $ openstack server show <vm>
   
   +----------------------+--------------------------------------+
   | Field                | Value                                |
   +----------------------+--------------------------------------+
   | ...                  | ...                                  |
   | status               | MIGRATING                            |
   | ...                  | ...                                  |
   +----------------------+--------------------------------------+

3. Wait for migration to complete. See Checking migration status to check the status of the migration.

4. Check the status of the instance to confirm if the migration was successful:

   (overcloud)$ openstack server list --host <dest> --all-projects

   Replace <dest> with the name or ID of the destination Compute node.

5. Optional: For instances that use NUMA, CPU-pinning, or DPDK, retrieve information about the NUMA topology from a Compute node to compare it with the NUMA topology that you retrieved during the preparing to migrate procedure. Comparing the NUMA topologies of the source and destination Compute nodes ensures that the source and destination Compute nodes use the same NUMA topology.

   • To view details about NUMA and CPU pinning, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh vcpuinfo <vm>

     • Replace overcloud-compute-n with the host name of the Compute node.
     • Replace <vm> with the name of the instance.

   • To view details about which NUMA node the instance is using, run the following command:

     $ ssh root@overcloud-compute-n
     # virsh numatune <vm>

     • Replace overcloud-compute-n with the host name of the Compute node.
     • Replace <vm> with the name or ID of the instance.

1. Queued: The Compute service has accepted the request to migrate an instance, and migration is pending.
2. Preparing: The Compute service is preparing to migrate the instance.
3. Running: The Compute service is migrating the instance.
4. Post-migrating: The Compute service has built the instance on the destination Compute node and is releasing resources on the source Compute node.
5. Completed: The Compute service has completed migrating the instance and finished releasing resources on the source Compute node.

Procedure

1. Retrieve the list of migration IDs for the instance:

   $ nova server-migration-list <vm>
   
   +----+-------------+-----------  (...)
   | Id | Source Node | Dest Node | (...)
   +----+-------------+-----------+ (...)
   | 2  | -           | -         | (...)
   +----+-------------+-----------+ (...)

   Replace <vm> with the name or ID of the instance.

2. Show the status of the migration:

   $ nova server-migration-show <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.

   • Replace <migration-id> with the ID of the migration.

     Running the nova server-migration-show command returns the following example output:

     +------------------------+--------------------------------------+
     | Property               | Value                                |
     +------------------------+--------------------------------------+
     | created_at             | 2017-03-08T02:53:06.000000           |
     | dest_compute           | controller                           |
     | dest_host              | -                                    |
     | dest_node              | -                                    |
     | disk_processed_bytes   | 0                                    |
     | disk_remaining_bytes   | 0                                    |
     | disk_total_bytes       | 0                                    |
     | id                     | 2                                    |
     | memory_processed_bytes | 65502513                             |
     | memory_remaining_bytes | 786427904                            |
     | memory_total_bytes     | 1091379200                           |
     | server_uuid            | d1df1b5a-70c4-4fed-98b7-423362f2c47c |
     | source_compute         | compute2                             |
     | source_node            | -                                    |
     | status                 | running                              |
     | updated_at             | 2017-03-08T02:53:47.000000           |
     +------------------------+--------------------------------------+

     Tip

     The OpenStack Compute service measures progress of the migration by the number of remaining memory bytes to copy. If this number does not decrease over time, the migration might be unable to complete, and the Compute service might abort it.

Procedure

1. Re-enable the source Compute node:

   (overcloud)$ source ~/stackrc
   (undercloud)$ openstack compute service set <source> nova-compute --enable

   Replace <source> with the host name of the source Compute node.

2. Optional: For instances that use DPDK, re-enable the destination Compute node from the undercloud:

   (undercloud)$ openstack compute service set <dest> nova-compute --enable

   Replace <dest> with the host name of the destination Compute node.