Chapter 9. Troubleshooting migration
Troubleshoot migration issues, navigate custom resources (CRs), services, and workflows, and download logs and CRs for troubleshooting information.
9.1. Error messages Copy linkLink copied to clipboard!
This section describes error messages and how to resolve them.
9.1.1. warm import retry limit reached Copy linkLink copied to clipboard!
The warm import retry limit reached error message is displayed during a warm migration if a VMware virtual machine (VM) has reached the maximum number (28) of changed block tracking (CBT) snapshots during the precopy stage.
To resolve this problem, delete some of the CBT snapshots from the VM and restart the migration plan.
9.1.2. Unable to resize disk image to required size Copy linkLink copied to clipboard!
The Unable to resize disk image to required size error message is displayed when migration fails because a virtual machine on the target provider uses persistent volumes with an EXT4 file system on block storage. The problem occurs because the default overhead that is assumed by CDI does not completely include the reserved place for the root partition.
To resolve this problem, increase the file system overhead in CDI to more than 10%.
9.2. Troubleshooting storage copy offload Copy linkLink copied to clipboard!
This section describes problems that are unique to storage copy offload and how you can resolve them.
9.2.1. vSphere-ESXi connectivity issues Copy linkLink copied to clipboard!
- Remote ESXi connection fails with a SOAP error
Description: Sometimes a remote ESXi execution fails, returning a SOAP error with no apparent root cause message.
Explanation: Because vSphere invokes some SOAP/REST endpoints on the ESXi, a connection can fail because of standard error reasons that vanish after the next try.
Solution: If the populator fails, the migration can be restarted. Try to restart or retry the populator, or restart the migration.
- VIB issues returned with a CLI error
Description: Migration Toolkit for Virtualization (MTV) returns the following error:
CLI Fault: The object or item referred to could not be found. <obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="5.0" xsi:type="LocalizedMethodFault"><fault xsi:type="NotFound"></fault><localizedMessage>The object or item referred to could not be found.</localizedMessage></obj>Explanation: If the VIB is installed, but
/etc/init.d/hostddid not restart, then thevmkfstoolsnamespace inesxcliis either not updated or does not exist. If that namespace does not exist, it means that this is the first usage, probably right after the first use.Solution: Use SSH to log in to the ESXi and run
/etc/init.d/hostd restart. Wait for a few seconds until the ESXi renews the connection with vSphere.
9.2.2. SSH error messages Copy linkLink copied to clipboard!
- Manual SSH key connection errors
Description: MTV returns one of the following errors:
Manual SSH key configuration required,Failed to connect via SSH, orSSH connection timeout.Explanation: The SSH connection is not running on the ESXi host for one of the following reasons:
The SSH is disabled.
Solution: Manually enable an SSH connection on the ESXi host by using the commands in Setting up storage copy offload using manually generated SSH keys.
There is a problem with the network connectivity.
Solution: Verify that the ESXi management network is accessible from the migration pods.
Timeout issue (least likely issue)
Solution: Increase the value of
SSH_TIMEOUT_SECONDSin the providerSecret.Verification steps for the preceding solutions:
To verify that the SSH service is running on an ESXi host, run the following command:
$ vim-cmd hostsvc/get_ssh_statusTo manually test SSH connectivity from a migration pod, run the following command:
$ ssh -i /path/to/<private_key> root@<ESXI_host_Ip>
9.2.3. NetApp Error Copy linkLink copied to clipboard!
Description: MTV returns the following error:
Cannot derive SVM to use; please specify SVM in config file
Explanation: ONTAP is not configured correctly.
Solution: Configure your default ONTAP Storage Virtual Machine (SVM) by running the following commands:
Show the current configuration for the SVM by running the following command on the ONTAP server:
$ vserver show -vserver ${NAME_OF_SVM}-
Set a management interface for the SVM and enter its
hostnamein theSTORAGE_HOSTNAMEby following the instructions in the NetApp Knowledge Base article, link: Trident fails to access ONTAP on SVM level and on Cluster level. The link requires you to log in.
9.3. Using the must-gather tool Copy linkLink copied to clipboard!
You can collect logs and information about MTV custom resources (CRs) by using the must-gather tool. You must attach a must-gather data file to all customer cases.
You can gather data for a specific namespace, migration plan, or virtual machine (VM) by using the filtering options.
If you specify a non-existent resource in the filtered must-gather command, no archive file is created.
Prerequisites
-
You must be logged in to the OpenShift Virtualization cluster as a user with the
cluster-adminrole. - You must have the Red Hat OpenShift CLI (oc) installed.
Procedure
-
Navigate to the directory where you want to store the
must-gatherdata. Run the
oc adm must-gathercommand:$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7The data is saved as
/must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.Optional: Run the
oc adm must-gathercommand with the following options to gather filtered data:Namespace:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- NS=<namespace> /usr/bin/targetedMigration plan:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- PLAN=<migration_plan> /usr/bin/targetedVirtual machine:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- VM=<vm_id> NS=<namespace> /usr/bin/targetedwhere:
- <vm_id>
-
Specifies the VM ID as it appears in the
PlanCR.
9.4. MTV custom resources and services Copy linkLink copied to clipboard!
The MTV Operator is provided as an Red Hat OpenShift Operator. It creates and manages the following custom resources (CRs) and services.
9.4.1. MTV custom resources Copy linkLink copied to clipboard!
-
ProviderCR stores attributes that enable MTV to connect to and interact with the source and target providers. -
NetworkMappingCR maps the networks of the source and target providers. -
StorageMappingCR maps the storage of the source and target providers. -
PlanCR contains a list of VMs with the same migration parameters and associated network and storage mappings. MigrationCR runs a migration plan.Only one
MigrationCR per migration plan can run at a given time. You can create multipleMigrationCRs for a singlePlanCR.
9.4.2. MTV services Copy linkLink copied to clipboard!
The
Inventoryservice performs the following actions:- Connects to the source and target providers.
- Maintains a local inventory for mappings and plans.
- Stores VM configurations.
-
Runs the
Validationservice if a VM configuration change is detected.
-
The
Validationservice checks the suitability of a VM for migration by applying rules. The
Migration Controllerservice orchestrates migrations.When you create a migration plan, the
Migration Controllerservice validates the plan and adds a status label. If the plan fails validation, the plan status isNot readyand the plan cannot be used to perform a migration. If the plan passes validation, the plan status isReadyand it can be used to perform a migration. After a successful migration, theMigration Controllerservice changes the plan status toCompleted.-
The
Populator Controllerservice orchestrates disk transfers using Volume Populators. -
The
Kubevirt ControllerandContainerized Data Import (CDI) Controllerservices handle most technical operations.
9.5. High-level migration workflow Copy linkLink copied to clipboard!
The high-level workflow shows the migration process from the point of view of the user:
- You create a source provider, a target provider, a network mapping, and a storage mapping.
You create a
Plancustom resource (CR) that includes the following resources:- Source provider
- Target provider, if MTV is not installed on the target cluster
- Network mapping
- Storage mapping
- One or more virtual machines (VMs)
You run a migration plan by creating a
MigrationCR that references thePlanCR.If you cannot migrate all the VMs for any reason, you can create multiple
MigrationCRs for the samePlanCR until all VMs are migrated.-
For each VM in the
PlanCR, theMigration Controllerservice records the VM migration progress in theMigrationCR. Once the data transfer for each VM in the
PlanCR completes, theMigration Controllerservice creates aVirtualMachineCR.When all VMs have been migrated, the
Migration Controllerservice updates the status of thePlanCR toCompleted. The power state of each source VM is maintained after migration.
9.5.1. Detailed migration workflow Copy linkLink copied to clipboard!
You can use the detailed migration workflow to troubleshoot a failed migration.
The workflow describes the following steps:
Warm Migration or migration to a remote OpenShift cluster:
When you create the
Migrationcustom resource (CR) to run a migration plan, theMigration Controllerservice creates aDataVolumeCR for each source VM disk.For each VM disk:
-
The
Containerized Data Importer (CDI) Controllerservice creates a persistent volume claim (PVC) based on the parameters specified in theDataVolumeCR. -
If the
StorageClasshas a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by theStorageClassprovisioner. -
The
CDI Controllerservice creates animporterpod. The
importerpod streams the VM disk to the PV.After the VM disks are transferred:
The
Migration Controllerservice creates aconversionpod with the PVCs attached to it when importing from VMware.The
conversionpod runsvirt-v2v, which installs and configures device drivers on the PVCs of the target VM.-
The
Migration Controllerservice creates aVirtualMachineCR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the
Migration Controllerpowers on the VM, theKubeVirt Controllerservice creates avirt-launcherpod and aVirtualMachineInstanceCR.The
virt-launcherpod runsQEMU-KVMwith the PVCs attached as VM disks.
Cold migration from Red Hat Virtualization or OpenStack to the local OpenShift cluster:
When you create a
Migrationcustom resource (CR) to run a migration plan, theMigration Controllerservice creates for each source VM disk aPersistentVolumeClaimCR, and anOvirtVolumePopulatorwhen the source is RHV, or anOpenstackVolumePopulatorCR when the source is OpenStack.For each VM disk:
-
The
Populator Controllerservice creates a temporarily persistent volume claim (PVC). If the
StorageClasshas a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by theStorageClassprovisioner.-
The
Migration Controllerservice creates a dummy pod to bind all PVCs. The name of the pod containspvcinit.
-
The
-
The
Populator Controllerservice creates apopulatorpod. The
populatorpod transfers the disk data to the PV.After the VM disks are transferred:
- The temporary PVC is deleted, and the initial PVC points to the PV with the data.
-
The
Migration Controllerservice creates aVirtualMachineCR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the
Migration Controllerpowers on the VM, theKubeVirt Controllerservice creates avirt-launcherpod and aVirtualMachineInstanceCR.The
virt-launcherpod runsQEMU-KVMwith the PVCs attached as VM disks.
Cold migration from VMware to the local OpenShift cluster:
When you create a
Migrationcustom resource (CR) to run a migration plan, theMigration Controllerservice creates aDataVolumeCR for each source VM disk.For each VM disk:
-
The
Containerized Data Importer (CDI) Controllerservice creates a blank persistent volume claim (PVC) based on the parameters specified in theDataVolumeCR. -
If the
StorageClasshas a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by theStorageClassprovisioner.
For all VM disks:
-
The
Migration Controllerservice creates a dummy pod to bind all PVCs. The name of the pod containspvcinit. -
The
Migration Controllerservice creates aconversionpod for all PVCs. The
conversionpod runsvirt-v2v, which converts the VM to the KVM hypervisor and transfers the disks' data to their corresponding PVs.After the VM disks are transferred:
-
The
Migration Controllerservice creates aVirtualMachineCR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the
Migration Controllerpowers on the VM, theKubeVirt Controllerservice creates avirt-launcherpod and aVirtualMachineInstanceCR.The
virt-launcherpod runsQEMU-KVMwith the PVCs attached as VM disks.
9.5.2. How MTV uses the virt-v2v tool Copy linkLink copied to clipboard!
The Migration Toolkit for Virtualization (MTV) uses the virt-v2v tool to convert the disk image of a virtual machine (VM) into a format compatible with OpenShift Virtualization. The tool makes migrations easier because it automatically performs the tasks needed to make your VMs work with OpenShift Virtualization. For example, enabling paravirtualized VirtIO drivers in the converted VM, if possible, and installing the QEMU guest agent.
virt-v2v is included in Red Hat Enterprise Linux (RHEL) versions 7 and later.
9.5.2.1. Main functions of virt-v2v in MTV migrations Copy linkLink copied to clipboard!
During migration, MTV uses virt-v2v to collect metadata about VMs, make necessary changes to VM disks, and copy the disks containing the VMs to OpenShift Virtualization.
virt-v2v makes the following changes to VM disks to prepare them for migration:
Additions:
- Injection of VirtIO drivers, for example, network or disk drivers.
- Preparation of hypervisor-specific tools or agents, for example, a QEMU guest agent installation.
- Modification of boot configuration, for example, updated boot loader or boot entries.
Removals:
- Unnecessary or former hypervisor-specific files, for example, VMware tools or VirtualBox additions.
- Old network driver configurations, for example, removing VMware-specific NIC drivers.
- Configuration settings that are incompatible with the target system, for example, old boot settings.
If you are migrating from VMware or from Open Virtual Appliances (OVA) files, virt-v2v also sets their IP addresses either during the migration or during the first reboot of the VMs after migration. s
You can also run predefined Ansible hooks before or after a migration using MTV. For more information, see About hooks for MTV migration plans.
These hooks do not necessarily use virt-v2v.
9.5.2.2. Customizing, removing, and installing files Copy linkLink copied to clipboard!
MTV uses virt-v2v to perform additional guest customizations during the conversion, such as the following actions:
- Customization to preserve IP addresses
- Customization to preserve drive letters
For Red Hat Enterprise Linux (RHEL)-based guests, virt-v2v attempts to install the guest agent from the Red Hat registry. If the migration is run in a detached environment, the installation program fails, and you must use hooks or other automation to install the guest agent.
For more information, see the man reference pages:
9.5.2.3. Permissions and virt-v2v Copy linkLink copied to clipboard!
virt-v2v does not require permissions or access credentials for the guest operating system itself because virt-v2v is not run against a running VM, but only against the disks of a VM.
9.5.3. Raw copy mode Copy linkLink copied to clipboard!
In regular cold and warm migrations, Migration Toolkit for Virtualization (MTV) uses a program called virt-v2v to prepare virtual machines (VMs) for migration to OpenShift Virtualization after the VMs have been copied from their source provider.
The main function of virt-v2v is to convert the disk image of a VM into a format compatible with OpenShift Virtualization.
This program is described in detail in How MTV uses the virt-v2v tool.
What is important to note here is that although virt-v2v is compatible with major operating systems such as recent versions of Red Hat Enterprise Linux, Windows, and Windows Server, it is not compatible with macOS and some other operating systems.
For a list of the operating systems that virt-v2v supports, see Converting virtual machines from other hypervisors to KVM with virt-v2v in RHEL 7, RHEL 8, RHEL 9, and RHEL 10.
As a workaround for migrating VMs that use an operating system that virt-v2v does not support, MTV includes a feature called raw copy mode. Raw copy mode copies VMs without applying any tool to convert them for use with OpenShift Virtualization. The migrated VMs use emulated devices.
This enables more robust migrations, enabling a wider range of operating systems and configurations. Examples are VMs with uncommon file systems, VMs with uncommon encryption technologies or without access to keys.
However, VMs migrated using raw copy mode might not boot on OpenShift Virtualization or perform as well as VMs migrated in the regular way. VMs migrated using raw copy mode might not boot on OpenShift Virtualization or perform as well as VMs migrated in the regular way.
Therefore, using raw copy mode is a tradeoff between being a more versatile migration option compared to increasing the risk of problems following migration.
Because of this risk, users are asked to request that Red Hat support perform raw copy mode migrations.
- For information about configuring raw copy mode by using the MTV user interface, see Configuring VMware migration plan settings in Planning your migration to Red Hat OpenShift Virtualization.
-
For information about configuring raw copy mode by using the command-line interface, see the
skipGuestConversionanduseCompatibilityModeparameters in Running a VMware vSphere migration from the command-line. - For information about device compatibility mode for raw copy migrations, see Device compatibility mode for raw copy migrations.
9.5.4. Device compatibility mode for raw copy migrations Copy linkLink copied to clipboard!
When you use raw copy mode, you can configure which device types MTV uses for migrated VMs through the Use compatibility mode setting.
- Compatibility devices (default)
By default, when you enable raw copy mode, MTV uses compatibility devices to ensure VMs can boot on OpenShift Virtualization:
- SATA bus for disk controllers
- E1000E NICs for network interfaces
USB controllers
These emulated devices work without requiring additional drivers in the guest operating system, ensuring maximum bootability for migrated VMs.
- VirtIO devices
For source VMs that already have VirtIO drivers installed, you can disable compatibility mode to use VirtIO devices instead. VirtIO devices provide better performance:
- VirtIO disk bus provides higher I/O throughput than SATA
VirtIO network interface provides better network performance than E1000E
ImportantThe Use compatibility mode setting only applies when you enable raw copy mode. When you disable raw copy mode (standard V2V conversion), MTV always uses VirtIO devices regardless of this setting.
WarningBefore you disable compatibility mode, verify that VirtIO drivers are installed in the source VM. VMs without VirtIO drivers do not boot on OpenShift Virtualization if you disable compatibility mode.
- When to disable compatibility mode
Consider disabling compatibility mode when:
- Your source VMs are running modern operating systems with VirtIO drivers pre-installed, for example, recent versions of Red Hat Enterprise Linux or Windows with VirtIO drivers
- Maximum performance is required for your workload
- You have verified VirtIO driver presence in the source VM before migration
- Configuring compatibility mode
- You can configure the compatibility mode setting when you create a migration plan. For more information, see Configuring VMware migration plan settings.
9.6. Collected logs and custom resource information Copy linkLink copied to clipboard!
You can download logs and custom resource (CR) YAML files for the following targets by using the Red Hat OpenShift web console or the command-line interface (CLI):
- Migration plan: Web console or CLI.
- Virtual machine: Web console or CLI.
- Namespace: CLI only.
The must-gather tool collects the following logs and CR files in an archive file:
CRs:
-
DataVolumeCR: Represents a disk mounted on a migrated VM. -
VirtualMachineCR: Represents a migrated VM. -
PlanCR: Defines the VMs and storage and network mapping. -
JobCR: Optional: Represents a pre-migration hook, a post-migration hook, or both.
-
Logs:
-
importerpod: Disk-to-data-volume conversion log. Theimporterpod naming convention isimporter-<migration_plan>-<vm_id><5_char_id>, for example,importer-mig-plan-ed90dfc6-9a17-4a8btnfh, whereed90dfc6-9a17-4a8is a truncated RHV VM ID andbtnfhis the generated 5-character ID. -
conversionpod: VM conversion log. Theconversionpod runsvirt-v2v, which installs and configures device drivers on the PVCs of the VM. Theconversionpod naming convention is<migration_plan>-<vm_id><5_char_id>. -
virt-launcherpod: VM launcher log. When a migrated VM is powered on, thevirt-launcherpod runsQEMU-KVMwith the PVCs attached as VM disks. -
forklift-controllerpod: The log is filtered for the migration plan, virtual machine, or namespace specified by themust-gathercommand. -
forklift-must-gather-apipod: The log is filtered for the migration plan, virtual machine, or namespace specified by themust-gathercommand. hook-jobpod: The log is filtered for hook jobs. Thehook-jobnaming convention is<migration_plan>-<vm_id><5_char_id>, for example,plan2j-vm-3696-posthook-4mx85orplan2j-vm-3696-prehook-mwqnl.NoteEmpty or excluded log files are not included in the
must-gatherarchive file.
-
The following schematic drawing shows the must-gather archive structure for an example VMware migration plan:
must-gather
└── namespaces
├── target-vm-ns
│ ├── crs
│ │ ├── datavolume
│ │ │ ├── mig-plan-vm-7595-tkhdz.yaml
│ │ │ ├── mig-plan-vm-7595-5qvqp.yaml
│ │ │ └── mig-plan-vm-8325-xccfw.yaml
│ │ └── virtualmachine
│ │ ├── test-test-rhel8-2disks2nics.yaml
│ │ └── test-x2019.yaml
│ └── logs
│ ├── importer-mig-plan-vm-7595-tkhdz
│ │ └── current.log
│ ├── importer-mig-plan-vm-7595-5qvqp
│ │ └── current.log
│ ├── importer-mig-plan-vm-8325-xccfw
│ │ └── current.log
│ ├── mig-plan-vm-7595-4glzd
│ │ └── current.log
│ └── mig-plan-vm-8325-4zw49
│ └── current.log
└── openshift-mtv
├── crs
│ └── plan
│ └── mig-plan-cold.yaml
└── logs
├── forklift-controller-67656d574-w74md
│ └── current.log
└── forklift-must-gather-api-89fc7f4b6-hlwb6
└── current.log
9.6.1. Downloading logs and custom resource information from the web console Copy linkLink copied to clipboard!
You can download logs and information about custom resources (CRs) for a completed, failed, or canceled migration plan or for migrated virtual machines (VMs) from the Red Hat OpenShift web console.
Procedure
- In the Red Hat OpenShift web console, click Migration for Virtualization > Migration plans.
- Click Get logs beside a migration plan name.
In the Get logs window, click Get logs.
The logs are collected. A
Log collection completemessage is displayed.- Click Download logs to download the archive file.
- To download logs for a migrated VM, click a migration plan name and then click Get logs beside the VM.
9.6.2. Accessing logs and custom resource information from the command line Copy linkLink copied to clipboard!
You can access logs and information about custom resources (CRs) from the command line by using the must-gather tool. You must attach a must-gather data file to all customer cases.
You can gather data for a specific namespace, a completed, failed, or canceled migration plan, or a migrated virtual machine (VM) by using the filtering options.
If you specify a non-existent resource in the filtered must-gather command, no archive file is created.
Prerequisites
-
You must be logged in to the OpenShift Virtualization cluster as a user with the
cluster-adminrole. - You must have the Red Hat OpenShift CLI (oc) installed.
Procedure
-
Navigate to the directory where you want to store the
must-gatherdata. Run the
oc adm must-gathercommand:$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7The data is saved as
/must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.Optional: Run the
oc adm must-gathercommand with the following options to gather filtered data:Namespace:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- NS=<namespace> /usr/bin/targetedMigration plan:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- PLAN=<migration_plan> /usr/bin/targetedVirtual machine:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.10.7 \ -- VM=<vm_name> NS=<namespace> /usr/bin/targetedwhere:
<vm_name>-
Specifies the VM name, not the VM ID, as it appears in the
PlanCR.