Chapter 13. Performance metrics and recommendations
MTV provides performance metrics for monitoring migrations and recommendations for optimizing network and storage, cold and warm migrations, and migration concurrency.
Performance metrics are available for all source providers. Performance recommendations and test data are based on VMware vSphere migrations.
The data that forms the basis of the following recommendations was collected from testing in Red Hat labs and is provided for reference only. Overall, these numbers should be considered to show the best-case scenarios. The observed performance of migration can differ from these results and depends on several factors. For procedures to configure performance optimizations, see the "Additional resources" section.
13.2. Performance metrics
MTV provides performance metrics to help you make informed decisions about system configuration and performance tuning. Some metrics, such as network and storage throughput, are displayed in the MTV user interface during active migrations. All metrics are available for local monitoring and alerts, for example, in a custom dashboard.
Network and storage throughput metrics are Technology Preview features only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- mtv_plans_status
Type: Gauge
Description: Reflects the current status of all migration plans.
Labels:
-
plan_status: Migration plan status (Succeeded,Failed,Canceled,Executing,Running,Pending,Blocked,Deleted) -
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) target: Destination cluster (local,remote)Example:
Executing, vsphere, warm, local = 2indicates 2 warm migrations from vSphere to a local cluster are currently executing.
-
- mtv_migration_duration_seconds
Type: Gauge
Description: Measures the duration of completed migration plans in seconds.
Labels:
-
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) -
target: Destination cluster (local,remote) plan_id: Migration plan identifier (Kubernetes UUID)Example:
vsphere, warm, local, abcdefab-1234-... = 1234indicates a warm migration from vSphere to a local cluster, with plan IDabcdefab-1234-..., completed in 1234 seconds.
-
- mtv_migration_data_transferred_bytes
Type: Gauge
Description: Tracks the total data transferred for each successful migration plan in bytes.
Labels:
-
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) -
target: Destination cluster (local,remote) plan_id: Migration plan identifier (Kubernetes UUID)Example:
vsphere, warm, local, abcdefab-1234-... = 12345678901indicates a successful warm migration plan from vSphere to a local cluster, with plan IDabcdefab-1234-..., transferred approximately 12 GB of data.
-
- mtv_workload_migrations_status_total
Type: Counter
Description: Provides a total count for every combination of labels for different migrations, including the plan ID. This metric serves as a mapping between migration status and specific plan IDs.
Labels:
-
migration_status: Migration status (Succeeded,Failed,Canceled) -
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) -
target: Destination cluster (local,remote) plan_id: Migration plan identifier (Kubernetes UUID)Example:
Succeeded, vsphere, cold, local, abcdef... = 1indicates 1 successful cold migration from vSphere to a local cluster for planabcdef....
-
- mtv_migrations_duration_seconds
Type: Histogram
Description: Counts the number of migration plans that finished within specific time intervals (for example, 1-2h, 2-5h, 5-10h, 10-24h, 24-48h).
Labels:
-
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) target: Destination cluster (local,remote)Example:
vsphere, warm, local = 2indicates 2 warm migrations from vSphere to a local cluster finished in 1 to 2 hours.
-
- mtv_plan_alert_status
Type: Gauge
Description: Contains information about migration failures, including the migration mode, source provider, plan name, and the phase where the migration failed.
Labels:
-
mode: Migration type (cold,warm) -
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
plan_name: Migration plan name phase: Migration phase where failure occurredThis metric helps identify where in the migration process failures are occurring.
-
- mtv_migration_net_throughput
Type: Gauge
Description: Exposes current network throughput for each running migration plan.
Labels:
-
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) -
target: Destination cluster (local,remote) plan_id: Migration plan identifier (Kubernetes UUID)This metric provides real-time insights into network performance during active migrations.
-
- mtv_migration_storage_throughput
Type: Gauge
Description: Exposes current storage throughput for each running migration plan.
Labels:
-
provider: Source platform (vsphere,ova,openstack,ovirt,openshift) -
mode: Migration type (cold,warm) -
target: Destination cluster (local,remote) plan_id: Migration plan identifier (Kubernetes UUID)This metric provides real-time insights into storage performance during active migrations.
-
13.3. Infrastructure and network optimization
Optimizing your infrastructure and network configuration is essential for achieving fast migration performance. Ensure fast storage and network speeds, both for VMware and Red Hat OpenShift environments.
To perform fast migrations, VMware must have fast read access to datastores. Networking between VMware ESXi hosts should be fast. Ensure a 10 GiB network connection and avoid network bottlenecks.
- Extend the VMware network to the Red Hat OpenShift Workers Interface network environment.
- It is important to ensure that the VMware network offers high throughput (10 Gigabit Ethernet) and rapid networking to guarantee that the reception rates align with the read rate of the ESXi datastore.
- Be aware that the migration process uses significant network bandwidth and that the migration network is utilized. If other services use that network, it might have an impact on those services and their migration rates.
-
For example, 200-325 MiB per second was the average network transfer rate from the
vmnicfor each ESXi host associated with transferring data to the Red Hat OpenShift interface.
13.3.1. Ensure fast datastore read speeds
Datastores read rates impact the total transfer times, so it is essential to ensure fast reads are possible from the ESXi datastore to the ESXi host.
Example in numbers: 200-300 MiB per second was the average read rate for both vSphere and ESXi endpoints for a single ESXi host. When multiple ESXi hosts are used, higher datastore read rates are possible.
13.3.2. Endpoint types
Migration Toolkit for Virtualization (MTV) allows for the following vSphere provider options:
- ESXi endpoint (inventory and disk transfers from ESXi).
- vCenter Server endpoint; no networks for the ESXi host (inventory and disk transfers from vCenter).
- vCenter endpoint and ESXi networks are available (inventory from vCenter, disk transfers from ESXi).
When transferring many VMs that are registered to multiple ESXi hosts, using the vCenter endpoint and ESXi network is suggested.
As of vSphere 7.0, ESXi hosts can label which network to use for Network Block Device (NBD) transport. This is accomplished by tagging the desired virtual network interface controller (NIC) with the appropriate vSphereBackupNFC label. When this is done, MTV is able to use the ESXi interface for network transfer to OpenShift provided that the worker and ESXi host interfaces are reachable. This is especially useful when migration users might not have access to the ESXi credentials yet want to be able to control which ESXi interface is used for migration.
For more details, see: (MTV-1230)
You can use the following ESXi command, which designates interface vmk2 for NBD backup:
$ esxcli network ip interface tag add -t vSphereBackupNFC -i vmk213.3.3. ESXi performance
ESXi performance can be measured for a single ESXI host or for multiple ESXi hosts.
Where possible, ensure that hosts used to perform migrations are set with BIOS profiles related to maximum performance. Hosts which use Host Power Management controlled within vSphere should check that High Performance is set.
Testing showed that when transferring more than 10 VMs with both BIOS and host power management set accordingly, migrations had an increase of 15 MiB in the average datastore read rate.
Single ESXi host performance
Test migrations by using a single ESXi host.
In each iteration, the total VMs increase to display the impact of concurrent migration on the duration.
The results show that migration time is linear when increasing the total VMs (50 GiB disk, Utilization 70%).
The optimal number of VMs per ESXi is 10.
| Test Case Description | MTV | VDDK | max_vm inflight | Migration Type | Total Duration |
|---|---|---|---|---|---|
| Cold migration, 10 VMs, Single ESXi, Private Network (non-management network) | 2.6 | 7.0.3 | 100 | cold | 0:21:39 |
| cold migration, 20 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 0:41:16 |
| Cold migration, 30 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 1:00:59 |
| Cold migration, 40 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 1:23:02 |
| Cold migration, 50 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 1:46:24 |
| Cold migration, 80 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 2:42:49 |
| Cold migration, 100 VMs, Single ESXi, Private Network | 2.6 | 7.0.3 | 100 | cold | 3:25:15 |
Multiple ESXi hosts and a single data store
In each iteration, the number of ESXi hosts was increased to show that increasing the number of ESXi hosts improves the migration time (50 GiB disk, Utilization 70%).
| Test Case Description | MTV | VDDK | max_vm inflight | Migration Type | Total Duration |
|---|---|---|---|---|---|
| Cold migration, 100 VMs, Single ESXi, Private Network (non-management network) | 2.6 | 7.0.3 | 100 | cold | 3:25:15 |
| Cold migration, 100 VMs, 4 ESXs (25 VMs per ESX), Private Network | 2.6 | 7.0.3 | 100 | cold | 1:22:27 |
| Cold migration, 100 VMs, 5 ESXs (20 VMs per ESX), Private Network, 1 Data Store | 2.6 | 7.0.3 | 100 | cold | 1:04:57 |
13.3.4. Performance using different migration networks
In each test, the Migration Network was changed, by using the Provider, to find the fastest network for migration.
The results indicate that there is no degradation by using management compared to non-management networks when all interfaces and network speeds are the same.
| Test Case Description | MTV | VDDK | max_vm inflight | Migration Type | Total Duration |
|---|---|---|---|---|---|
| Cold migration, 10 VMs, Single ESXi, MGMT Network | 2.6 | 7.0.3 | 100 | cold | 0:21:30 |
| Cold migration, 10 VMs, Single ESXi, Private Network (non-management network) | 2.6 | 7.0.3 | 20 | cold | 0:21:20 |
| Cold migration, 10 VMs, Single ESXi, Default Network | 2.6.2 | 7.0.3 | 20 | cold | 0:21:30 |
13.3.5. Avoid additional network load on VMware networks
You can reduce the network load on VMware networks by selecting the migration network when using the ESXi endpoint.
By incorporating a virtualization provider, MTV enables the selection of a specific network, which is accessible on the ESXi hosts, for the purpose of migrating virtual machines to OpenShift. Selecting this migration network from the ESXi host in the MTV UI ensures that the transfer is performed using the selected network as an ESXi endpoint.
It is imperative to ensure that the network selected has connectivity to the OCP interface, has adequate bandwidth for migrations, and that the network interface is not saturated.
In environments with fast networks, such as 10GbE networks, migration network impacts can be expected to match the rate of ESXi datastore reads.
13.4. Concurrent migration settings
Configuring concurrent migration settings can significantly improve migration performance by allowing multiple VMs and hosts to migrate simultaneously. Set the MAX_VM_INFLIGHT MTV variable to control the maximum number of concurrent VM transfers allowed for the ESXi host.
Migration Toolkit for Virtualization (MTV) allows for concurrency to be controlled by using this variable; by default, it is set to 20.
When setting MAX_VM_INFLIGHT, consider the number of maximum concurrent VM transfers that are required for ESXi hosts. It is important to consider the type of migration to be transferred concurrently.
Warm migrations use snapshots to compare and migrate only the differences between previous snapshots of the disk. The migration of the differences between snapshots happens over specific intervals before a final cut-over of the running VM to OpenShift occurs.
In MTV, MAX_VM_INFLIGHT reserves one transfer slot per VM, regardless of current migration activity for a specific snapshot or the number of disks that belong to a single VM. The total set by MAX_VM_INFLIGHT is used to indicate how many concurrent VM transfers per ESXi host is allowed.
Example:
MAX_VM_INFLIGHT = 20 and 2 ESXi hosts defined in the provider mean each host can transfer 20 VMs.
13.4.1. Migrations are completed faster when migrating multiple VMs concurrently
When multiple VMs from a specific ESXi host are to be migrated by using Migration Toolkit for Virtualization (MTV), starting concurrent migrations for multiple VMs leads to faster migration times.
Testing demonstrated that migrating 10 VMs (each containing 35 GiB of data, with a total size of 50 GiB) from a single host is significantly faster than migrating the same number of VMs sequentially, one after another.
It is possible to increase concurrent migration to more than 10 virtual machines from a single host, but it does not show a significant improvement.
Examples:
- 1 single disk VMs took 6 minutes, with migration rate of 100 MiB/s
- 10 single disk VMs took 22 minutes, with migration rate of 272 MiB/s
- 20 single disk VMs took 42 minutes, with migration rate of 284 MiB/s
Based on these examples, migrating 10 virtual machines simultaneously is three times faster than migrating the same virtual machines sequentially.
The migration rate was almost the same when moving 10 or 20 virtual machines simultaneously.
13.4.2. Migrations complete faster using multiple hosts
Using multiple hosts with registered VMs equally distributed among the ESXi hosts used for migrations leads to faster migration times.
Testing showed that when transferring more than 10 single disk VMs, each containing 35 GiB of data out of a total of 50 GiB, using additional hosts can reduce migration time.
Examples:
- 80 single disk VMs, containing 35 GiB of data each, using a single host took 2 hours and 43 minutes, with a migration rate of 294 MiB/s.
- 80 single disk VMs, containing 35 GiB of data each, using 8 ESXi hosts took 41 minutes, with a migration rate of 1,173 MiB/s.
Based on these examples, migrating 80 VMs from 8 ESXi hosts, 10 from each host, concurrently is four times faster than migrating the same VMs from a single ESXi host.
Migrating a larger number of VMs from more than 8 ESXi hosts concurrently could potentially show increased performance. However, this was not tested and is therefore not recommended.
13.4.3. Multiple migration plans compared to a single large migration plan
The maximum number of disks that can be referenced by a single migration plan is 500. For more details, see (MTV-1203).
When attempting to migrate many VMs in a single migration plan, it can take some time for all migrations to start. By breaking up one migration plan into several migration plans, it is possible to start them at the same time.
Comparing migrations of:
-
500 VMs using 8 ESXi hosts in 1 plan,
max_vm_inflight=100, took 5 hours and 10 minutes. -
800 VMs using 8 ESXi hosts with 8 plans,
max_vm_inflight=100, took 57 minutes.
Testing showed that by breaking one single large plan into multiple moderately sized plans, for example, by using 100 VMs per plan, the total migration time can be reduced.
13.5. Maximum values tested for cold migrations
The following maximum values were tested for cold migrations:
- Maximum number of ESXi hosts tested: 8
- Maximum number of VMs in a single migration plan: 500
- Maximum number of VMs migrated in a single test: 5000
- Maximum number of migration plans performed concurrently: 40
- Maximum single disk size migrated: 6 TB disk, which contained 3 TB of data
- Maximum number of disks on a single VM migrated: 50
- Highest observed single datastore read rate from a single ESXi host: 312 MiB/s
- Highest observed multi-datastore read rate using eight ESXi hosts and two datastores: 1,242 MiB/s
- Highest observed virtual NIC transfer rate to an OpenShift worker: 327 MiB/s
- Maximum migration transfer rate of a single disk: 162 MiB/s (rate observed when transferring nonconcurrent migration of 1.5 TB utilized data)
- Maximum cold migration transfer rate of the multiple VMs (single disk) from a single ESXi host: 294 MiB/s (concurrent migration of 30 VMs, 35/50 GiB used, from Single ESXi)
- Maximum cold migration transfer rate of the multiple VMs (single disk) from multiple ESXi hosts: 1,173 MB/s (concurrent migration of 80 VMs, 35/50 GiB used, from 8 ESXi hosts, 10 VMs from each ESXi)
13.6. Warm migration recommendations
The following recommendations are specific to warm migrations:
- Migrate up to 400 disks in parallel
- Testing involved migrating 200 VMs in parallel, with 2 disks each using 8 ESXi hosts, for a total of 400 disks. No tests were run on migration plans migrating over 400 disks in parallel, so it is not recommended to migrate over this number of disks in parallel.
- Migrate up to 200 disks in parallel for the fastest rate
Testing was successfully performed on parallel disk migrations with 200, 300, and 400 disks. There was a decrease in the precopy migration rate, approximately 25%, between the tests migrating 200 disks and those migrating 300 and 400 disks.
Therefore, it is recommended to perform parallel disk migrations in groups of 200 or fewer, instead of 300 to 400 disks, unless a decline of 25% in precopy speed does not affect your cutover planning.
- When possible, set cutover time to be immediately after a migration plan starts
- To reduce the overall time of warm migrations, it is recommended to set the cutover to occur immediately after the migration plan is started. This causes MTV to run only one precopy per VM. This recommendation is valid, no matter how many VMs are in the migration plan.
- Increase precopy intervals between snapshots
-
If you are creating many migration plans with a single VM and have enough time between the migration start and the cutover, increase the value of the
controller_precopy_intervalparameter to between 120 and 240 minutes, inclusive. The longer setting will reduce the total number of snapshots and disk transfers per VM before the cutover.
13.6.1. Maximum values tested for warm migrations
The following maximum values were tested for warm migrations:
- Maximum number of ESXi hosts tested: 8
- Maximum number of worker nodes: 12
- Maximum number of VMs in a single migration plan: 200
- Maximum number of total parallel disk transfers: 400, with 200 VMs, 6 ESXis, and a transfer rate of 667 MB/s
- Maximum single disk size migrated: 6 TB disk, which contained 3 TB of data
- Maximum number of disks on a single VM migrated: 3
- Maximum number of parallel disk transfers per ESXi host: 68
- Maximum transfer rate observed of a single disk with no concurrent migrations: 76.5 MB/s
- Maximum transfer rate observed of multiple disks from a single ESXi host: 253 MB/s (concurrent migration of 10 VMs, 1 disk each, 35/50 GiB used per disk)
- Total transfer rate observed of multiple disks (210) from 8 ESXi hosts: 802 MB/s (concurrent migration of 70 VMs, 3 disks each, 35/50 GiB used per disk)
13.7. Large disk migrations
Migrating VMs with large disks requires special considerations for optimal performance and minimal downtime.
The following recommendations are suggested for VMs with data on disk totaling to 1 TB or greater for each individual disk:
- Schedule appropriate maintenance windows for migrating large disk virtual machines (VMs). Such migrations are sensitive operations and might require careful planning of maintenance windows and downtime, especially during periods of lower storage and network activity.
- Check that no other migration activities or other heavy network or storage activities are run during those large virtual machine (VM) migrations. During those migrations, prioritize MTV activities. Plan to migrate those VMs to a time when there are fewer activities on those VMs and related datastore.
-
For large VMs with a high churn rate, which means data is frequently changed in amounts of 100 GB or more between snapshots, consider reducing the warm migration
controller_precopy_intervalfrom the default, which is 60 minutes. It is important to ensure that this process is started at least 24 hours before the scheduled cutover to allow for multiple successful precopy snapshots to complete. When scheduling the cutover, ensure that the maintenance window allows for enough time for the last snapshot of changes to be copied over and that the cutover process begins at the beginning of that maintenance window. - In cases of particularly large single-disk VMs, where some downtime is possible, select cold migrations rather than warm migrations, especially in the case of large VM snapshots.
- Consider splitting data on particularly large disks to multiple disks, which enables parallel disk migration with MTV when warm migration is used.
- If you have large database disks with continuous writes of large amounts of data, where downtime and VM snapshots are not possible, it might be necessary to consider database vendor-specific replication options of the database data to target these specific migrations outside MTV. Consult the vendor-specific options of your database if this case applies.
13.8. Increasing AIO sizes and buffer counts for NBD transport mode
You can change Network Block Device (NBD) transport network file copy (NFC) parameters to increase migration performance when you use Asynchronous Input/Output (AIO) buffering with the Migration Toolkit for Virtualization (MTV).
For information about configuring AIO buffering, see Advanced migration options.
Using AIO buffering is only suitable for cold migration use cases.
Remove AIO buffer configuration before initializing warm migrations. For more details, see Removing AIO buffer configuration.
13.8.1. Key findings
The best migration performance was achieved by migrating multiple (10) virtual machines (VMs) on a single ESXi host with the following values:
-
VixDiskLib.nfcAio.Session.BufSizeIn64KB=16 -
vixDiskLib.nfcAio.Session.BufCount=4
-
The following improvements were noted when using AIO buffer settings (asynchronous buffer counts):
- Migration time was reduced by 31.1%, from 0:24:32 to 0:16:54.
- Read rate was increased from 347.83 MB/s to 504.93 MB/s.
- There was no significant improvement observed when using AIO buffer settings with a single VM.
- There was no significant improvement observed when using AIO buffer settings with multiple VMs from multiple hosts.
13.8.2. Key requirements for support for AIO sizes and buffer counts
Support is based upon tests performed using the following versions:
- vSphere 7.0.3
- VDDK 7.0.3
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}