Chapter 9. Troubleshooting
This section provides information for troubleshooting common migration issues.
9.1. Error messages
This section describes error messages and how to resolve them.
warm import retry limit reached
The warm import retry limit reached
error message is displayed during a warm migration if a VMware virtual machine (VM) has reached the maximum number (28) of changed block tracking (CBT) snapshots during the precopy stage.
To resolve this problem, delete some of the CBT snapshots from the VM and restart the migration plan.
Unable to resize disk image to required size
The Unable to resize disk image to required size
error message is displayed when migration fails because a virtual machine on the target provider uses persistent volumes with an EXT4 file system on block storage. The problem occurs because the default overhead that is assumed by CDI does not completely include the reserved place for the root partition.
To resolve this problem, increase the file system overhead in CDI to be more than 10%.
9.2. Using the must-gather tool
You can collect logs and information about MTV custom resources (CRs) by using the must-gather
tool. You must attach a must-gather
data file to all customer cases.
You can gather data for a specific namespace, migration plan, or virtual machine (VM) by using the filtering options.
If you specify a non-existent resource in the filtered must-gather
command, no archive file is created.
Prerequisites
-
You must be logged in to the OpenShift Virtualization cluster as a user with the
cluster-admin
role. -
You must have the OpenShift Container Platform CLI (
oc
) installed.
Collecting logs and CR information
-
Navigate to the directory where you want to store the
must-gather
data. Run the
oc adm must-gather
command:$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0
The data is saved as
/must-gather/must-gather.tar.gz
. You can upload this file to a support case on the Red Hat Customer Portal.Optional: Run the
oc adm must-gather
command with the following options to gather filtered data:Namespace:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- NS=<namespace> /usr/bin/targeted
Migration plan:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- PLAN=<migration_plan> /usr/bin/targeted
Virtual machine:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- VM=<vm_id> NS=<namespace> /usr/bin/targeted 1
9.3. Architecture
This section describes MTV custom resources, services, and workflows.
9.3.1. MTV custom resources and services
The Migration Toolkit for Virtualization (MTV) is provided as an OpenShift Container Platform Operator. It creates and manages the following custom resources (CRs) and services.
MTV custom resources
-
Provider
CR stores attributes that enable MTV to connect to and interact with the source and target providers. -
NetworkMapping
CR maps the networks of the source and target providers. -
StorageMapping
CR maps the storage of the source and target providers. -
Provisioner
CR stores the configuration of the storage provisioners, such as supported volume and access modes. -
Plan
CR contains a list of VMs with the same migration parameters and associated network and storage mappings. Migration
CR runs a migration plan.Only one
Migration
CR per migration plan can run at a given time. You can create multipleMigration
CRs for a singlePlan
CR.
MTV services
The
Inventory
service performs the following actions:- Connects to the source and target providers.
- Maintains a local inventory for mappings and plans.
- Stores VM configurations.
-
Runs the
Validation
service if a VM configuration change is detected.
-
The
Validation
service checks the suitability of a VM for migration by applying rules. The
User Interface
service performs the following actions:- Enables you to create and configure MTV CRs.
- Displays the status of the CRs and the progress of a migration.
The
Migration Controller
service orchestrates migrations.When you create a migration plan, the
Migration Controller
service validates the plan and adds a status label. If the plan fails validation, the plan status isNot ready
and the plan cannot be used to perform a migration. If the plan passes validation, the plan status isReady
and it can be used to perform a migration. After a successful migration, theMigration Controller
service changes the plan status toCompleted
.-
The
Kubevirt Controller
andContainerized Data Import (CDI) Controller
services handle most technical operations.
9.3.2. High-level migration workflow
The high-level workflow shows the migration process from the point of view of the user:
- You create a source provider, a target provider, a network mapping, and a storage mapping.
You create a
Plan
custom resource (CR) that includes the following resources:- Source provider
- Target provider, if MTV is not installed on the target cluster
- Network mapping
- Storage mapping
- One or more virtual machines (VMs)
You run a migration plan by creating a
Migration
CR that references thePlan
CR.If you cannot migrate all the VMs for any reason, you can create multiple
Migration
CRs for the samePlan
CR until all VMs are migrated.For each VM in the
Plan
CR, theMigration Controller
service creates aVirtualMachine
CR and records the VM migration progress in theMigration
CR.When all VMs have been migrated, the
Migration Controller
service updates the status of thePlan
CR toCompleted
. The power state of each source VM is maintained after migration.
9.3.3. Detailed migration workflow
You can use the detailed migration workflow to troubleshoot a failed migration.
The workflow describes the following steps:
When you create a
Migration
custom resource (CR) to run a migration plan, theMigration Controller
service creates aVirtualMachine
CR for each source virtual machine (VM) and aDataVolume
CR for each source VM disk.For each VM disk:
-
The
Containerized Data Importer (CDI) Controller
service creates a persistent volume claim (PVC) based on the parameters specified in theDataVolume
CR. -
If the
StorageClass
has a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by theStorageClass
provisioner. -
The
CDI Controller
service creates animporter
pod. The
importer
pod streams the VM disk to the PV.After the VM disks are transferred:
The
Migration Controller
service creates aconversion
pod with the PVCs attached to it.The
conversion
pod runsvirt-v2v
, which installs and configures device drivers on the PVCs of the target VM.When the target VM is powered on, the
KubeVirt Controller
service creates avirt-launcher
pod and aVirtualMachineInstance
CR.The
virt-launcher
pod runsQEMU-KVM
with the PVCs attached as VM disks.
9.4. Logs and custom resources
You can download logs and custom resource (CR) information for troubleshooting. For more information, see the detailed migration workflow.
9.4.1. Collected logs and custom resource information
You can download logs and custom resource (CR) yaml
files for the following targets by using the MTV web console or the command line interface (CLI):
- Migration plan: Web console or CLI.
- Virtual machine: Web console or CLI.
- Namespace: CLI only.
The must-gather
tool collects the following logs and CR files in an archive file:
CRs:
-
DataVolume
CR: Represents a disk mounted on a migrated VM. -
VirtualMachine
CR: Represents a migrated VM. -
Plan
CR: Defines the VMs and storage and network mapping. -
Job
CR: Optional: Represents a pre-migration hook, a post-migration hook, or both.
-
Logs:
-
importer
pod: Disk-to-data-volume conversion log. Theimporter
pod naming convention isimporter-<migration_plan>-<vm_id><5_char_id>
, for example,importer-mig-plan-ed90dfc6-9a17-4a8btnfh
, whereed90dfc6-9a17-4a8
is a truncated RHV VM ID andbtnfh
is the generated 5-character ID. -
conversion
pod: VM conversion log. Theconversion
pod runsvirt-v2v
, which installs and configures device drivers on the PVCs of the VM. Theconversion
pod naming convention is<migration_plan>-<vm_id><5_char_id>
. -
virt-launcher
pod: VM launcher log. When a migrated VM is powered on, thevirt-launcher
pod runsQEMU-KVM
with the PVCs attached as VM disks. -
forklift-controller
pod: The log is filtered for the migration plan, virtual machine, or namespace specified by themust-gather
command. -
forklift-must-gather-api
pod: The log is filtered for the migration plan, virtual machine, or namespace specified by themust-gather
command. hook-job
pod: The log is filtered for hook jobs. Thehook-job
naming convention is`<migration_plan>-<vm_id><5_char_id>, for example, `plan2j-vm-3696-posthook-4mx85
orplan2j-vm-3696-prehook-mwqnl
.NoteEmpty or excluded log files are not included in the
must-gather
archive file.
-
Example must-gather archive structure for a VMware migration plan
must-gather └── namespaces ├── target-vm-ns │ ├── crs │ │ ├── datavolume │ │ │ ├── mig-plan-vm-7595-tkhdz.yaml │ │ │ ├── mig-plan-vm-7595-5qvqp.yaml │ │ │ └── mig-plan-vm-8325-xccfw.yaml │ │ └── virtualmachine │ │ ├── test-test-rhel8-2disks2nics.yaml │ │ └── test-x2019.yaml │ └── logs │ ├── importer-mig-plan-vm-7595-tkhdz │ │ └── current.log │ ├── importer-mig-plan-vm-7595-5qvqp │ │ └── current.log │ ├── importer-mig-plan-vm-8325-xccfw │ │ └── current.log │ ├── mig-plan-vm-7595-4glzd │ │ └── current.log │ └── mig-plan-vm-8325-4zw49 │ └── current.log └── openshift-mtv ├── crs │ └── plan │ └── mig-plan-cold.yaml └── logs ├── forklift-controller-67656d574-w74md │ └── current.log └── forklift-must-gather-api-89fc7f4b6-hlwb6 └── current.log
9.4.2. Downloading logs and custom resource information from the web console
You can download logs and information about custom resources (CRs) for a completed, failed, or canceled migration plan or for migrated virtual machines (VMs) by using the MTV web console.
Procedure
- In the web console, click Migration plans.
- Click Get logs beside a migration plan name.
In the Get logs window, click Get logs.
The logs are collected. A
Log collection complete
message is displayed.- Click Download logs to download the archive file.
- To download logs for a migrated VM, click a migration plan name and then click Get logs beside the VM.
9.4.3. Accessing logs and custom resource information from the command line interface
You can access logs and information about custom resources (CRs) from the command line interface by using the must-gather
tool. You must attach a must-gather
data file to all customer cases.
You can gather data for a specific namespace, a completed, failed, or canceled migration plan, or a migrated virtual machine (VM) by using the filtering options.
If you specify a non-existent resource in the filtered must-gather
command, no archive file is created.
Prerequisites
-
You must be logged in to the OpenShift Virtualization cluster as a user with the
cluster-admin
role. -
You must have the OpenShift Container Platform CLI (
oc
) installed.
Procedure
-
Navigate to the directory where you want to store the
must-gather
data. Run the
oc adm must-gather
command:$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0
The data is saved as
/must-gather/must-gather.tar.gz
. You can upload this file to a support case on the Red Hat Customer Portal.Optional: Run the
oc adm must-gather
command with the following options to gather filtered data:Namespace:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- NS=<namespace> /usr/bin/targeted
Migration plan:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- PLAN=<migration_plan> /usr/bin/targeted
Virtual machine:
$ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.3.0 \ -- VM=<vm_name> NS=<namespace> /usr/bin/targeted 1
- 1
- You must specify the VM name, not the VM ID, as it appears in the
Plan
CR.