1.7. Troubleshooting

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   $ oc get migmigration -n openshift-migration

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

   Copy to Clipboard Toggle word wrap

2. Inspect the MigMigration CR:

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   Copy to Clipboard Toggle word wrap

   The output is similar to the following examples.

Procedure

1. Get the mig-log-reader pod:

   $ oc -n openshift-migration get pods | grep log

   Copy to Clipboard Toggle word wrap

2. Enter the following command to display a single migration log:

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 

   1

   Copy to Clipboard Toggle word wrap

   1

   The -c plain option displays the log without colors.

Procedure

1. In the MTC console, click Migration plans to view the list of migration plans.
2. Click the Options menu of a specific migration plan and select Logs.

3. Click Download Logs to download the logs of the MigrationController, Velero, and Restic pods for all clusters.

   You can download a single log by selecting the cluster, log source, and pod source, and then clicking Download Selected.

   You can access a pod log from the CLI by using the oc logs command:

   $ oc logs <pod-name> -f -n openshift-migration 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the pod name.

Procedure

1. Run the migration plan.

2. View the MigPlan custom resource (CR):

   $ oc describe migplan <migplan_name> -n openshift-migration 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the MigPlan CR.

   The output is similar to the following:

   metadata:
     ...
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a 

   1

   
   status:
     ...
     conditions:
     - category: Warn
       lastTransitionTime: 2020-04-30T17:16:23Z
       message: 'Some namespaces contain GVKs incompatible with destination cluster.
         See: `incompatibleNamespaces` for details'
       status: "True"
       type: GVKsIncompatible
     incompatibleNamespaces:
     - gvks: 

   2

   
       - group: batch
         kind: cronjobs
         version: v2alpha1
       - group: batch
         kind: scheduledjobs
         version: v2alpha1

   Copy to Clipboard Toggle word wrap

   1

   Record the MigPlan CR UID.

   2

   Record the deprecated APIs listed in the gvks section.

3. Get the MigMigration name associated with the MigPlan UID:

   $ oc get migmigration -o json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=="<migplan_uid>") | .metadata.name' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the MigPlan CR UID.

4. Get the MigMigration UID associated with the MigMigration name:

   $ oc get migmigration <migmigration_name> -o jsonpath='{.metadata.uid}' 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the MigMigration name.

5. Get the Velero Backup name associated with the MigMigration UID:

   $ oc get backup.velero.io --selector migration-initial-backup="<migmigration_uid>" -o jsonpath={.items[*].metadata.name} 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the MigMigration UID.

6. Download the contents of the Velero Backup to your local machine by running the command for your storage provider:

   • AWS S3:

     $ aws s3 cp s3://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 

     1

     Copy to Clipboard Toggle word wrap

     1

     Specify the bucket, backup name, and your local backup directory name.

   • GCP:

     $ gsutil cp gs://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 

     1

     Copy to Clipboard Toggle word wrap

     1

     Specify the bucket, backup name, and your local backup directory name.

   • Azure:

     $ azcopy copy 'https://velerobackups.blob.core.windows.net/velero/backups/<backup_name>' '<backup_local_dir>' --recursive 

     1

     Copy to Clipboard Toggle word wrap

     1

     Specify the backup name and your local backup directory name.

7. Extract the Velero Backup archive file:

   $ tar -xfv <backup_local_dir>/<backup_name>.tar.gz -C <backup_local_dir>

   Copy to Clipboard Toggle word wrap

8. Run oc convert in offline mode on each deprecated API:

   $ oc convert -f <backup_local_dir>/resources/<gvk>.json

   Copy to Clipboard Toggle word wrap

9. Create the converted API on the target cluster:

   $ oc create -f <gvk>.json

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the status of the MigMigration CR:

   $ oc describe migmigration <pod_name> -n openshift-migration

   Copy to Clipboard Toggle word wrap

   The output includes the following status message:

   Example output

   ...
   Some or all transfer pods are not running for more than 10 mins on destination cluster
   ...

   Copy to Clipboard Toggle word wrap

2. On the source cluster, obtain the details of a migrated namespace:

   $ oc get namespace <namespace> -o yaml 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   $ oc edit namespace <namespace>

   Copy to Clipboard Toggle word wrap

4. Add missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

   Copy to Clipboard Toggle word wrap
5. Run the migration plan again.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.4

   Copy to Clipboard Toggle word wrap
3. Remove authentication keys and other sensitive information.

4. Create an archive file containing the contents of the must-gather data directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

   Copy to Clipboard Toggle word wrap
5. Upload the compressed file as an attachment to your customer support case.