1.7. Troubleshooting

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   $ oc get migmigration -n openshift-migration

   Example output

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. Inspect the MigMigration CR:

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Get the mig-log-reader pod:

   $ oc -n openshift-migration get pods | grep log

2. Enter the following command to display a single migration log:

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 1

   1

   The -c plain option displays the log without colors.

Procedure

1. In the MTC console, click Migration plans to view the list of migration plans.
2. Click the Options menu of a specific migration plan and select Logs.

3. Click Download Logs to download the logs of the MigrationController, Velero, and Restic pods for all clusters.

   You can download a single log by selecting the cluster, log source, and pod source, and then clicking Download Selected.

   You can access a pod log from the CLI by using the oc logs command:

   $ oc logs <pod-name> -f -n openshift-migration 1

   1

   Specify the pod name.

Procedure

1. Run the migration plan.

2. View the MigPlan custom resource (CR):

   $ oc describe migplan <migplan_name> -n openshift-migration 1

   1

   Specify the name of the MigPlan CR.

   The output is similar to the following:

   metadata:
     ...
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a 1
   status:
     ...
     conditions:
     - category: Warn
       lastTransitionTime: 2020-04-30T17:16:23Z
       message: 'Some namespaces contain GVKs incompatible with destination cluster.
         See: `incompatibleNamespaces` for details'
       status: "True"
       type: GVKsIncompatible
     incompatibleNamespaces:
     - gvks: 2
       - group: batch
         kind: cronjobs
         version: v2alpha1
       - group: batch
         kind: scheduledjobs
         version: v2alpha1

   1

   Record the MigPlan CR UID.

   2

   Record the deprecated APIs listed in the gvks section.

3. Get the MigMigration name associated with the MigPlan UID:

   $ oc get migmigration -o json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=="<migplan_uid>") | .metadata.name' 1

   1

   Specify the MigPlan CR UID.

4. Get the MigMigration UID associated with the MigMigration name:

   $ oc get migmigration <migmigration_name> -o jsonpath='{.metadata.uid}' 1

   1

   Specify the MigMigration name.

5. Get the Velero Backup name associated with the MigMigration UID:

   $ oc get backup.velero.io --selector migration-initial-backup="<migmigration_uid>" -o jsonpath={.items[*].metadata.name} 1

   1

   Specify the MigMigration UID.

6. Download the contents of the Velero Backup to your local machine by running the command for your storage provider:

   • AWS S3:

     $ aws s3 cp s3://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • GCP:

     $ gsutil cp gs://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • Azure:

     $ azcopy copy 'https://velerobackups.blob.core.windows.net/velero/backups/<backup_name>' '<backup_local_dir>' --recursive 1

     1

     Specify the backup name and your local backup directory name.

7. Extract the Velero Backup archive file:

   $ tar -xfv <backup_local_dir>/<backup_name>.tar.gz -C <backup_local_dir>

8. Run oc convert in offline mode on each deprecated API:

   $ oc convert -f <backup_local_dir>/resources/<gvk>.json

9. Create the converted API on the target cluster:

   $ oc create -f <gvk>.json

Procedure

1. Check the status of the MigMigration CR:

   $ oc describe migmigration <pod_name> -n openshift-migration

   The output includes the following status message:

   Example output

   ...
   Some or all transfer pods are not running for more than 10 mins on destination cluster
   ...

2. On the source cluster, obtain the details of a migrated namespace:

   $ oc get namespace <namespace> -o yaml 1

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   $ oc edit namespace <namespace>

4. Add missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

5. Run the migration plan again.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.4

3. Remove authentication keys and other sensitive information.

4. Create an archive file containing the contents of the must-gather data directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

5. Upload the compressed file as an attachment to your customer support case.