Chapter 10. Troubleshooting

Procedure

1. In the MTC web console, click Migration Plans.

2. Click the Migrations number next to a migration plan to view the Migrations page.

   The Migrations page displays the migration types associated with the migration plan, for example, Stage, Migration, or Rollback.

3. Click the Type link to view the Migration details page.

4. Expand Migration resources to view the migration resources and their status.

   Note

   To troubleshoot a failed migration, start with a high-level resource that has failed and then work down the resource tree towards the lower-level resources.

5. Click the Options menu next to a resource and select one of the following options:

   • Copy oc describe command copies the command to your clipboard.

     • Log in to the relevant cluster and then run the command.

       The conditions and events of the resource are displayed in YAML format.

   • Copy oc logs command copies the command to your clipboard.

     • Log in to the relevant cluster and then run the command.

       If the resource supports log filtering, a filtered log is displayed.

   • View JSON displays the resource data in JSON format in a web browser.

     The data is the same as the output for the oc get <resource> command.

Procedure

1. In the MTC web console, click Migration Plans.

2. Click the Migrations number next to a migration plan to view the Migrations page.

   The Migrations page displays the migration types associated with the migration plan, for example, Stage or Cutover for warm migration.

3. Click View logs.
4. Click the Copy icon to copy the oc logs command to your clipboard.

5. Log in to the relevant cluster and enter the command on the CLI.

   The aggregated log for the migration plan is displayed.

Procedure

1. Get the mig-log-reader pod:

   $ oc -n openshift-migration get pods | grep log

   Copy to Clipboard Toggle word wrap

2. Enter the following command to display a single migration log:

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 

   1

   Copy to Clipboard Toggle word wrap

   1

   The -c plain option displays the log without colors.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   • To gather data for the past hour:

     $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.4

     Copy to Clipboard Toggle word wrap

     The data is saved as /must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.

   • To gather data for the past 24 hours:

     $ oc adm must-gather --image= \
       registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8: \
       v1.4 -- /usr/bin/gather_metrics_dump

     Copy to Clipboard Toggle word wrap

     This operation can take a long time. The data is saved as /must-gather/metrics/prom_data.tar.gz. You can view this file with the Prometheus console.

To view data with the Prometheus console

1. Create a local Prometheus instance:

   $ make prometheus-run

   Copy to Clipboard Toggle word wrap

   The command outputs the Prometheus URL:

   Output

   Started Prometheus on http://localhost:9090

   Copy to Clipboard Toggle word wrap

2. Launch a web browser and navigate to the URL to view the data by using the Prometheus web console.

3. After you have viewed the data, delete the Prometheus instance and data:

   $ make prometheus-cleanup

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the status of a MigMigration CR:

   $ oc get migmigration <migmigration> -o yaml

   Copy to Clipboard Toggle word wrap

   Example output

   status:
     conditions:
     - category: Warn
       durable: true
       lastTransitionTime: "2021-01-26T20:48:40Z"
       message: 'Final Restore openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf: partially failed on destination cluster'
       status: "True"
       type: VeleroFinalRestorePartiallyFailed
     - category: Advisory
       durable: true
       lastTransitionTime: "2021-01-26T20:48:42Z"
       message: The migration has completed with warnings, please look at `Warn` conditions.
       reason: Completed
       status: "True"
       type: SucceededWithWarnings

   Copy to Clipboard Toggle word wrap

2. Check the status of the Restore CR by using the Velero describe command:

   $ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore describe <restore>

   Copy to Clipboard Toggle word wrap

   Example output

   Phase:  PartiallyFailed (run 'velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf' for more information)
   
   Errors:
     Velero:     <none>
     Cluster:    <none>
     Namespaces:
       migration-example:  error restoring example.com/migration-example/migration-example: the server could not find the requested resource

   Copy to Clipboard Toggle word wrap

3. Check the Restore CR logs by using the Velero logs command:

   $ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore logs <restore>

   Copy to Clipboard Toggle word wrap

   Example output

   time="2021-01-26T20:48:37Z" level=info msg="Attempting to restore migration-example: migration-example" logSource="pkg/restore/restore.go:1107" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
   time="2021-01-26T20:48:37Z" level=info msg="error restoring migration-example: the server could not find the requested resource" logSource="pkg/restore/restore.go:1170" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf

   Copy to Clipboard Toggle word wrap

   The Restore CR log error message, the server could not find the requested resource, indicates the cause of the partially failed migration.

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   $ oc get migmigration -n openshift-migration

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

   Copy to Clipboard Toggle word wrap

2. Inspect the MigMigration CR:

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   Copy to Clipboard Toggle word wrap

   The output is similar to the following examples.

Procedure

1. Ensure that the internal registries are exposed on the OpenShift Container Platform 3 and 4 clusters.

   The internal registry is exposed by default on OpenShift Container Platform 4.

2. If you are using insecure registries, add your registry host values to the [registries.insecure] section of /etc/container/registries.conf to ensure that podman does not encounter a TLS verification error.

3. Log in to the OpenShift Container Platform 3 registry:

   $ podman login -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false <registry_url>:<port>

   Copy to Clipboard Toggle word wrap

4. Log in to the OpenShift Container Platform 4 registry:

   $ podman login -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false <registry_url>:<port>

   Copy to Clipboard Toggle word wrap

5. Pull the OpenShift Container Platform 3 image:

   $ podman pull <registry_url>:<port>/openshift/<image>

   Copy to Clipboard Toggle word wrap

6. Tag the OpenShift Container Platform 3 image for the OpenShift Container Platform 4 registry:

   $ podman tag <registry_url>:<port>/openshift/<image> \ 

   1

   
     <registry_url>:<port>/openshift/<image> 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the registry URL and port for the OpenShift Container Platform 3 cluster.

   2

   Specify the registry URL and port for the OpenShift Container Platform 4 cluster.

7. Push the image to the OpenShift Container Platform 4 registry:

   $ podman push <registry_url>:<port>/openshift/<image> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the OpenShift Container Platform 4 cluster.

8. Verify that the image has a valid image stream:

   $ oc get imagestream -n openshift | grep <image>

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      IMAGE REPOSITORY                                                      TAGS    UPDATED
   my_image  image-registry.openshift-image-registry.svc:5000/openshift/my_image  latest  32 seconds ago

   Copy to Clipboard Toggle word wrap

Procedure

1. Check the status of the MigMigration CR:

   $ oc describe migmigration <pod> -n openshift-migration

   Copy to Clipboard Toggle word wrap

   The output includes the following status message:

   Example output

   Some or all transfer pods are not running for more than 10 mins on destination cluster

   Copy to Clipboard Toggle word wrap

2. On the source cluster, obtain the details of a migrated namespace:

   $ oc get namespace <namespace> -o yaml 

   1

   Copy to Clipboard Toggle word wrap

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   $ oc edit namespace <namespace>

   Copy to Clipboard Toggle word wrap

4. Add the missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

   Copy to Clipboard Toggle word wrap
5. Run the migration plan again.

Procedure

1. In the MTC web console, click Migration plans.
2. Click the Options menu beside a migration plan and select Rollback.

3. Click Rollback and wait for rollback to complete.

   In the migration plan details, Rollback succeeded is displayed.

4. Verify that rollback was successful in the OpenShift Container Platform web console of the source cluster:

   1. Click Home Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads Pods to verify that the pods are running in the migrated namespace.
   5. Click Storage Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. Create a MigMigration CR based on the following example:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigMigration
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: <migmigration>
     namespace: openshift-migration
   spec:
   ...
     rollback: true
   ...
     migPlanRef:
       name: <migplan> 

   1

   
       namespace: openshift-migration
   EOF

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the associated MigPlan CR.

2. In the MTC web console, verify that the migrated project resources have been removed from the target cluster.
3. Verify that the migrated project resources are present in the source cluster and that the application is running.