3.5. Troubleshooting

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   Copy to Clipboard Toggle word wrap

   $ oc get migmigration -n openshift-migration

   Example output

   Copy to Clipboard Toggle word wrap

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. Inspect the MigMigration CR:

   Copy to Clipboard Toggle word wrap

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Get the mig-log-reader pod:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-migration get pods | grep log

2. Enter the following command to display a single migration log:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 

   1

   1

   The -c plain option displays the log without colors.

Procedure

1. In the MTC console, click Migration plans to view the list of migration plans.
2. Click the Options menu of a specific migration plan and select Logs.

3. Click Download Logs to download the logs of the MigrationController, Velero, and Restic pods for all clusters.

   You can download a single log by selecting the cluster, log source, and pod source, and then clicking Download Selected.

   You can access a pod log from the CLI by using the oc logs command:

   Copy to Clipboard Toggle word wrap

   $ oc logs <pod-name> -f -n openshift-migration 

   1

   1

   Specify the pod name.

Procedure

1. Run the migration plan.

2. View the MigPlan custom resource (CR):

   Copy to Clipboard Toggle word wrap

   $ oc describe migplan <migplan_name> -n openshift-migration 

   1

   1

   Specify the name of the MigPlan CR.

   The output is similar to the following:

   Copy to Clipboard Toggle word wrap

   metadata:
     ...
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a 

   1

   
   status:
     ...
     conditions:
     - category: Warn
       lastTransitionTime: 2020-04-30T17:16:23Z
       message: 'Some namespaces contain GVKs incompatible with destination cluster.
         See: `incompatibleNamespaces` for details'
       status: "True"
       type: GVKsIncompatible
     incompatibleNamespaces:
     - gvks: 

   2

   
       - group: batch
         kind: cronjobs
         version: v2alpha1
       - group: batch
         kind: scheduledjobs
         version: v2alpha1

   1

   Record the MigPlan CR UID.

   2

   Record the deprecated APIs listed in the gvks section.

3. Get the MigMigration name associated with the MigPlan UID:

   Copy to Clipboard Toggle word wrap

   $ oc get migmigration -o json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=="<migplan_uid>") | .metadata.name' 

   1

   1

   Specify the MigPlan CR UID.

4. Get the MigMigration UID associated with the MigMigration name:

   Copy to Clipboard Toggle word wrap

   $ oc get migmigration <migmigration_name> -o jsonpath='{.metadata.uid}' 

   1

   1

   Specify the MigMigration name.

5. Get the Velero Backup name associated with the MigMigration UID:

   Copy to Clipboard Toggle word wrap

   $ oc get backup.velero.io --selector migration-initial-backup="<migmigration_uid>" -o jsonpath={.items[*].metadata.name} 

   1

   1

   Specify the MigMigration UID.

6. Download the contents of the Velero Backup to your local machine by running the command for your storage provider:

   • AWS S3:

     Copy to Clipboard Toggle word wrap

     $ aws s3 cp s3://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 

     1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • GCP:

     Copy to Clipboard Toggle word wrap

     $ gsutil cp gs://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 

     1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • Azure:

     Copy to Clipboard Toggle word wrap

     $ azcopy copy 'https://velerobackups.blob.core.windows.net/velero/backups/<backup_name>' '<backup_local_dir>' --recursive 

     1

     1

     Specify the backup name and your local backup directory name.

7. Extract the Velero Backup archive file:

   Copy to Clipboard Toggle word wrap

   $ tar -xfv <backup_local_dir>/<backup_name>.tar.gz -C <backup_local_dir>

8. Run oc convert in offline mode on each deprecated API:

   Copy to Clipboard Toggle word wrap

   $ oc convert -f <backup_local_dir>/resources/<gvk>.json

9. Create the converted API on the target cluster:

   Copy to Clipboard Toggle word wrap

   $ oc create -f <gvk>.json

Procedure

1. Check the status of the MigMigration CR:

   Copy to Clipboard Toggle word wrap

   $ oc describe migmigration <pod_name> -n openshift-migration

   The output includes the following status message:

   Example output

   Copy to Clipboard Toggle word wrap

   ...
   Some or all transfer pods are not running for more than 10 mins on destination cluster
   ...

2. On the source cluster, obtain the details of a migrated namespace:

   Copy to Clipboard Toggle word wrap

   $ oc get namespace <namespace> -o yaml 

   1

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   Copy to Clipboard Toggle word wrap

   $ oc edit namespace <namespace>

4. Add missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   Copy to Clipboard Toggle word wrap

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

5. Run the migration plan again.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the must-gather command:

   Copy to Clipboard Toggle word wrap

   $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.4

3. Remove authentication keys and other sensitive information.

4. Create an archive file containing the contents of the must-gather data directory:

   Copy to Clipboard Toggle word wrap

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

5. Upload the compressed file as an attachment to your customer support case.