Migrating from version 3 to 4

Procedure

1. Place an exterior network component, such as an application load balancer or a reverse proxy, between the clients and the target cluster.
2. Update the application FQDN on the source cluster in the DNS server to return the IP address of the exterior network component.
3. Configure the network component to send requests received for the application in the source domain to the load balancer in the target cluster domain.
4. Create a wildcard DNS record for the *.apps.source.example.com domain that points to the IP address of the load balancer of the source cluster.
5. Create a DNS record for each application that points to the IP address of the exterior network component in front of the target cluster. A specific DNS record has higher priority than a wildcard record, so no conflict arises when the application FQDN is resolved.

1. Create a route in the target cluster’s project that is configured to accept requests addressed to the application’s FQDN in the source cluster:

   $ oc expose svc <app1-svc> --hostname <app1.apps.source.example.com> \
    -n <app1-namespace>

   With this new route in place, the server accepts any request for that FQDN and sends it to the corresponding application pods. In addition, when you migrate the application, another route is created in the target cluster domain. Requests reach the migrated application using either of these hostnames.

2. Create a DNS record with your DNS provider that points the application’s FQDN in the source cluster to the IP address of the default load balancer of the target cluster. This will redirect traffic away from your source cluster to your target cluster.

   The FQDN of the application resolves to the load balancer of the target cluster. The default ingress controller router accept requests for that FQDN because a route for that hostname is exposed.

1. Replace the x509 certificate of the default ingress controller created during the installation process with a custom certificate.

2. Configure this certificate to include the wildcard DNS domains for both the source and target clusters in the subjectAltName field.

   The new certificate is valid for securing connections made using either DNS domain.

Procedure

1. Get the MigrationController CR manifest:

   $ oc get migrationcontroller <migration_controller> -n openshift-migration

2. Update the proxy parameters:

   apiVersion: migration.openshift.io/v1alpha1
   kind: MigrationController
   metadata:
     name: <migration_controller>
     namespace: openshift-migration
   ...
   spec:
     stunnel_tcp_proxy: http://<username>:<password>@<ip>:<port> 1
     noProxy: example.com 2

   1

   Stunnel proxy URL for direct volume migration.

   2

   Comma-separated list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying.

   Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

   This field is ignored if neither the httpProxy nor the httpsProxy field is set.

3. Save the manifest as migration-controller.yaml.

4. Apply the updated manifest:

   $ oc replace -f migration-controller.yaml -n openshift-migration

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

   You use these credentials to add MCG as a replication repository.

Procedure

1. Set the BUCKET variable:

   $ BUCKET=<your_bucket>

2. Set the REGION variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION 1

   1

   us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero 1

   1

   If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a velero-policy.json file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the velero user the necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the velero user:

   $ aws iam create-access-key --user-name velero

   Example output

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID. You use the credentials to add AWS as a replication repository.

Procedure

1. Log in to GCP:

   $ gcloud auth login

2. Set the BUCKET variable:

   $ BUCKET=<bucket> 1

   1

   Specify your bucket name.

3. Create the storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero service account"

6. List your service accounts:

   $ gcloud iam service-accounts list

7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:Velero service account" \
       --format 'value(email)')

8. Attach the policies to give the velero user the necessary permissions:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )

9. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server

11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to add GCP as a replication repository.

Procedure

1. Log in to Azure:

   $ az login

2. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

3. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location CentralUS 1

   1

   Specify your location.

4. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

5. Create an Azure storage account:

   $ az storage account create \
       --name $AZURE_STORAGE_ACCOUNT_ID \
       --resource-group $AZURE_RESOURCE_GROUP \
       --sku Standard_GRS \
       --encryption-services blob \
       --https-only true \
       --kind BlobStorage \
       --access-tier Hot

6. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

7. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

8. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv` \
     AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv` \
     AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" \
     --role "Contributor" --query 'password' -o tsv` \
     AZURE_CLIENT_ID=`az ad sp list --display-name "velero" \
     --query '[0].appId' -o tsv`

9. Save the service principal credentials in the credentials-velero file:

   $ cat << EOF > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

   You use the credentials-velero file to add Azure as a replication repository.

Procedure

1. Get the MigrationController CR manifest:

   $ oc get migrationcontroller <migration_controller> -n openshift-migration

2. Update the proxy parameters:

   apiVersion: migration.openshift.io/v1alpha1
   kind: MigrationController
   metadata:
     name: <migration_controller>
     namespace: openshift-migration
   ...
   spec:
     stunnel_tcp_proxy: http://<username>:<password>@<ip>:<port> 1
     noProxy: example.com 2

   1

   Stunnel proxy URL for direct volume migration.

   2

   Comma-separated list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying.

   Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

   This field is ignored if neither the httpProxy nor the httpsProxy field is set.

3. Save the manifest as migration-controller.yaml.

4. Apply the updated manifest:

   $ oc replace -f migration-controller.yaml -n openshift-migration

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed MTC.

2. Obtain the MTC web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the MTC web console.

   Note

   If you try to access the MTC web console immediately after installing the Migration Toolkit for Containers Operator, the console might not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. Log in to the cluster.

2. Obtain the migration-controller service account token:

   $ oc sa get-token migration-controller -n openshift-migration

   Example output

   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. In the MTC web console, click Clusters.
4. Click Add cluster.

5. Fill in the following fields:

   • Cluster name: The cluster name can contain lower-case letters (a-z) and numbers (0-9). It must not contain spaces or international characters.
   • URL: Specify the API server URL, for example, https://<www.example.com>:8443.
   • Service account token: Paste the migration-controller service account token.

   • Exposed route host to image registry: If you are using direct image migration, specify the exposed route to the image registry of the source cluster, for example, www.example.apps.cluster.com.

     You can specify a port. The default port is 5000.

   • Azure cluster: You must select this option if you use Azure snapshots to copy your data.
   • Azure resource group: This field is displayed if Azure cluster is selected. Specify the Azure resource group.
   • Require SSL verification: Optional: Select this option to verify SSL connections to the cluster.
   • CA bundle file: This field is displayed if Require SSL verification is selected. If you created a custom CA certificate bundle file for self-signed certificates, click Browse, select the CA bundle file, and upload it.

6. Click Add cluster.

   The cluster appears in the Clusters list.

Procedure

1. In the MTC web console, click Replication repositories.
2. Click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for S3 providers, including AWS and MCG:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • S3 bucket name: Specify the name of the S3 bucket.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for some S3 providers. Check the product documentation of your S3 provider for expected values.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG and other S3 providers.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG and other S3 providers.
     • Require SSL verification: Clear this checkbox if you are using a generic S3 provider.
     • If you created a custom CA certificate bundle for self-signed certificates, click Browse and browse to the Base64-encoded file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories list.

Procedure

1. In the MTC web console, click Migration plans.
2. Click Add migration plan.

3. Enter the Plan name.

   The migration plan name must not exceed 253 lower-case alphanumeric characters (a-z, 0-9) and must not contain spaces or underscores (_).

4. Select a Source cluster, a Target cluster, and a Repository, and click Next.
5. On the Namespaces page, select the projects to be migrated and click Next.

6. On the Persistent volumes page, click a Migration type for each PV:

   • The Copy option copies the data from the PV of a source cluster to the replication repository and then restores the data on a newly created PV, with similar characteristics, in the target cluster.
   • The Move option unmounts a remote volume, for example, NFS, from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using.
7. Click Next.

8. On the Copy options page, select a Copy method for each PV:

   • Snapshot copy backs up and restores data using the cloud provider’s snapshot functionality. It is significantly faster than Filesystem copy.

   • Filesystem copy backs up the files on the source cluster and restores them on the target cluster.

     The file system copy method is required for direct volume migration.

9. You can select Verify copy to verify data migrated with Filesystem copy. Data is verified by generating a checksum for each source file and checking the checksum after restoration. Data verification significantly reduces performance.

10. Select a Target storage class.

    If you selected Filesystem copy, you can change the target storage class.

11. Click Next.

12. On the Migration options page, the Direct image migration option is selected if you specified an exposed image registry route for the source cluster. The Direct PV migration option is selected if you are migrating data with Filesystem copy.

    The direct migration options copy images and files directly from the source cluster to the target cluster. This option is much faster than copying images and files from the source cluster to the replication repository and then from the replication repository to the target cluster.

13. Click Next.

14. Optional: On the Hooks page, click Add Hook to add a hook to the migration plan.

    A hook runs custom code. You can add up to four hooks to a single migration plan. Each hook runs during a different migration step.

    1. Enter the name of the hook to display in the web console.
    2. If the hook is an Ansible playbook, select Ansible playbook and click Browse to upload the playbook or paste the contents of the playbook in the field.
    3. Optional: Specify an Ansible runtime image if you are not using the default hook image.

    4. If the hook is not an Ansible playbook, select Custom container image and specify the image name and path.

       A custom container image can include Ansible playbooks.

    5. Select Source cluster or Target cluster.
    6. Enter the Service account name and the Service account namespace.

    7. Select the migration step for the hook:

       • preBackup: Before the application workload is backed up on the source cluster
       • postBackup: After the application workload is backed up on the source cluster
       • preRestore: Before the application workload is restored on the target cluster
       • postRestore: After the application workload is restored on the target cluster
    8. Click Add.

15. Click Finish.

    The migration plan is displayed in the Migration plans list.

Procedure

1. Log in to the MTC web console and click Migration plans.

2. Click the Options menu next to a migration plan and select one of the following options under Migration:

   • Stage copies data from the source cluster to the target cluster without stopping the application.

   • Cutover stops the transactions on the source cluster and moves the resources to the target cluster.

     Optional: In the Cutover migration dialog, you can clear the Halt transactions on the source cluster during migration checkbox.

   • State copies selected persistent volume claims (PVCs).

     Important

     Do not use state migration to migrate a namespace between clusters. Use stage or cutover migration instead.

     • Select one or more PVCs in the State migration dialog and click Migrate.

3. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volumes are correctly provisioned.

Procedure

1. Create a MigCluster CR manifest for the host cluster:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigCluster
   metadata:
     name: <host_cluster>
     namespace: openshift-migration
   spec:
     isHostCluster: true
   EOF

2. Create a Secret object manifest for each remote cluster:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: <cluster_secret>
     namespace: openshift-config
   type: Opaque
   data:
     saToken: <sa_token> 1
   EOF

   1

   Specify the base64-encoded migration-controller service account (SA) token of the remote cluster. You can obtain the token by running the following command:

   $ oc sa get-token migration-controller -n openshift-migration | base64 -w 0

3. Create a MigCluster CR manifest for each remote cluster:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigCluster
   metadata:
     name: <remote_cluster> 1
     namespace: openshift-migration
   spec:
     exposedRegistryPath: <exposed_registry_route> 2
     insecure: false 3
     isHostCluster: false
     serviceAccountSecretRef:
       name: <remote_cluster_secret> 4
       namespace: openshift-config
     url: <remote_cluster_url> 5
   EOF

   1

   Specify the Cluster CR of the remote cluster.

   2

   Optional: For direct image migration, specify the exposed registry route.

   3

   SSL verification is enabled if false. CA certificates are not required or checked if true.

   4

   Specify the Secret object of the remote cluster.

   5

   Specify the URL of the remote cluster.

4. Verify that all clusters are in a Ready state:

   $ oc describe cluster <cluster>

5. Create a Secret object manifest for the replication repository:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     namespace: openshift-config
     name: <migstorage_creds>
   type: Opaque
   data:
     aws-access-key-id: <key_id_base64> 1
     aws-secret-access-key: <secret_key_base64> 2
   EOF

   1

   Specify the key ID in base64 format.

   2

   Specify the secret key in base64 format.

   AWS credentials are base64-encoded by default. For other storage providers, you must encode your credentials by running the following command with each key:

   $ echo -n "<key>" | base64 -w 0 1

   1

   Specify the key ID or the secret key. Both keys must be base64-encoded.

6. Create a MigStorage CR manifest for the replication repository:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigStorage
   metadata:
     name: <migstorage>
     namespace: openshift-migration
   spec:
     backupStorageConfig:
       awsBucketName: <bucket> 1
       credsSecretRef:
         name: <storage_secret> 2
         namespace: openshift-config
     backupStorageProvider: <storage_provider> 3
     volumeSnapshotConfig:
       credsSecretRef:
         name: <storage_secret> 4
         namespace: openshift-config
     volumeSnapshotProvider: <storage_provider> 5
   EOF

   1

   Specify the bucket name.

   2

   Specify the Secrets CR of the object storage. You must ensure that the credentials stored in the Secrets CR of the object storage are correct.

   3

   Specify the storage provider.

   4

   Optional: If you are copying data by using snapshots, specify the Secrets CR of the object storage. You must ensure that the credentials stored in the Secrets CR of the object storage are correct.

   5

   Optional: If you are copying data by using snapshots, specify the storage provider.

7. Verify that the MigStorage CR is in a Ready state:

   $ oc describe migstorage <migstorage>

8. Create a MigPlan CR manifest:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigPlan
   metadata:
     name: <migplan>
     namespace: openshift-migration
   spec:
     destMigClusterRef:
       name: <host_cluster>
       namespace: openshift-migration
     indirectImageMigration: true 1
     indirectVolumeMigration: true 2
     migStorageRef:
       name: <migstorage> 3
       namespace: openshift-migration
     namespaces:
       - <application_namespace> 4
     srcMigClusterRef:
       name: <remote_cluster> 5
       namespace: openshift-migration
   EOF

   1

   Direct image migration is enabled if false.

   2

   Direct volume migration is enabled if false.

   3

   Specify the name of the MigStorage CR instance.

   4

   Specify one or more source namespaces. By default, the destination namespace has the same name.

   5

   Specify the name of the source cluster MigCluster instance.

9. Verify that the MigPlan instance is in a Ready state:

   $ oc describe migplan <migplan> -n openshift-migration

10. Create a MigMigration CR manifest to start the migration defined in the MigPlan instance:

    $ cat << EOF | oc apply -f -
    apiVersion: migration.openshift.io/v1alpha1
    kind: MigMigration
    metadata:
      name: <migmigration>
      namespace: openshift-migration
    spec:
      migPlanRef:
        name: <migplan> 1
        namespace: openshift-migration
      quiescePods: true 2
      stage: false 3
      rollback: false 4
    EOF

    1

    Specify the MigPlan CR name.

    2

    The pods on the source cluster are stopped before migration if true.

    3

    A stage migration, which copies most of the data without stopping the application, is performed if true.

    4

    A completed migration is rolled back if true.

11. Verify the migration by watching the MigMigration CR progress:

    $ oc watch migmigration <migmigration> -n openshift-migration

    The output resembles the following:

    Example output

    Name:         c8b034c0-6567-11eb-9a4f-0bc004db0fbc
    Namespace:    openshift-migration
    Labels:       migration.openshift.io/migplan-name=django
    Annotations:  openshift.io/touch: e99f9083-6567-11eb-8420-0a580a81020c
    API Version:  migration.openshift.io/v1alpha1
    Kind:         MigMigration
    ...
    Spec:
      Mig Plan Ref:
        Name:       migplan
        Namespace:  openshift-migration
      Stage:        false
    Status:
      Conditions:
        Category:              Advisory
        Last Transition Time:  2021-02-02T15:04:09Z
        Message:               Step: 19/47
        Reason:                InitialBackupCreated
        Status:                True
        Type:                  Running
        Category:              Required
        Last Transition Time:  2021-02-02T15:03:19Z
        Message:               The migration is ready.
        Status:                True
        Type:                  Ready
        Category:              Required
        Durable:               true
        Last Transition Time:  2021-02-02T15:04:05Z
        Message:               The migration registries are healthy.
        Status:                True
        Type:                  RegistriesHealthy
      Itinerary:               Final
      Observed Digest:         7fae9d21f15979c71ddc7dd075cb97061895caac5b936d92fae967019ab616d5
      Phase:                   InitialBackupCreated
      Pipeline:
        Completed:  2021-02-02T15:04:07Z
        Message:    Completed
        Name:       Prepare
        Started:    2021-02-02T15:03:18Z
        Message:    Waiting for initial Velero backup to complete.
        Name:       Backup
        Phase:      InitialBackupCreated
        Progress:
          Backup openshift-migration/c8b034c0-6567-11eb-9a4f-0bc004db0fbc-wpc44: 0 out of estimated total of 0 objects backed up (5s)
        Started:        2021-02-02T15:04:07Z
        Message:        Not started
        Name:           StageBackup
        Message:        Not started
        Name:           StageRestore
        Message:        Not started
        Name:           DirectImage
        Message:        Not started
        Name:           DirectVolume
        Message:        Not started
        Name:           Restore
        Message:        Not started
        Name:           Cleanup
      Start Timestamp:  2021-02-02T15:03:18Z
    Events:
      Type    Reason   Age                 From                     Message
      ----    ------   ----                ----                     -------
      Normal  Running  57s                 migmigration_controller  Step: 2/47
      Normal  Running  57s                 migmigration_controller  Step: 3/47
      Normal  Running  57s (x3 over 57s)   migmigration_controller  Step: 4/47
      Normal  Running  54s                 migmigration_controller  Step: 5/47
      Normal  Running  54s                 migmigration_controller  Step: 6/47
      Normal  Running  52s (x2 over 53s)   migmigration_controller  Step: 7/47
      Normal  Running  51s (x2 over 51s)   migmigration_controller  Step: 8/47
      Normal  Ready    50s (x12 over 57s)  migmigration_controller  The migration is ready.
      Normal  Running  50s                 migmigration_controller  Step: 9/47
      Normal  Running  50s                 migmigration_controller  Step: 10/47

Procedure

1. Migrate persistent volume data from the source to the target cluster.

   You can perform this step as many times as needed. The source application continues running.

2. Quiesce the source application.

   You can do this by setting the replicas of workload resources to 0, either directly on the source cluster or by updating the manifests in GitHub and re-syncing the Argo CD application.

3. Clone application manifests to the target cluster.

   You can use Argo CD to clone the application manifests to the target cluster.

4. Migrate the remaining volume data from the source to the target cluster.

   Migrate any new data created by the application during the state migration process by performing a final data migration.

5. If the cloned application is in a quiesced state, unquiesce it.
6. Switch the DNS record to the target cluster to re-direct user traffic to the migrated application.

Procedure

1. Migrate persistent volume data from the source to the target cluster.

   You can perform this step as many times as needed. The source application continues running.

2. Quiesce the source application.

   You can do this by setting the replicas of workload resources to 0, either directly on the source cluster or by updating the manifests in GitHub and re-syncing the Argo CD application.

3. Clone application manifests to the target cluster.

   You can use Argo CD to clone the application manifests to the target cluster.

4. Migrate the remaining volume data from the source to the target cluster.

   Migrate any new data created by the application during the state migration process by performing a final data migration.

5. If the cloned application is in a quiesced state, unquiesce it.
6. Switch the DNS record to the target cluster to re-direct user traffic to the migrated application.

Procedure

1. Edit the MigrationController custom resource manifest:

   $ oc edit migrationcontroller <migration_controller> -n openshift-migration

2. Update the spec section by adding a parameter to exclude specific resources or by adding a resource to the excluded_resources parameter if it does not have its own exclusion parameter:

   apiVersion: migration.openshift.io/v1alpha1
   kind: MigrationController
   metadata:
     name: migration-controller
     namespace: openshift-migration
   spec:
     disable_image_migration: true 1
     disable_pv_migration: true 2
     ...
     excluded_resources: 3
     - imagetags
     - templateinstances
     - clusterserviceversions
     - packagemanifests
     - subscriptions
     - servicebrokers
     - servicebindings
     - serviceclasses
     - serviceinstances
     - serviceplans
     - operatorgroups
     - events
     - events.events.k8s.io

   1

   Add disable_image_migration: true to exclude image streams from the migration. Do not edit the excluded_resources parameter. imagestreams is added to excluded_resources when the MigrationController pod restarts.

   2

   Add disable_pv_migration: true to exclude PVs from the migration plan. Do not edit the excluded_resources parameter. persistentvolumes and persistentvolumeclaims are added to excluded_resources when the MigrationController pod restarts. Disabling PV migration also disables PV discovery when you create the migration plan.

   3

   You can add OpenShift Container Platform resources to the excluded_resources list. Do not delete the default excluded resources. These resources are problematic to migrate and must be excluded.

3. Wait two minutes for the MigrationController pod to restart so that the changes are applied.

4. Verify that the resource is excluded:

   $ oc get deployment -n openshift-migration migration-controller -o yaml | grep EXCLUDED_RESOURCES -A1

   The output contains the excluded resources:

   Example output

       - name: EXCLUDED_RESOURCES
         value:
         imagetags,templateinstances,clusterserviceversions,packagemanifests,subscriptions,servicebrokers,servicebindings,serviceclasses,serviceinstances,serviceplans,imagestreams,persistentvolumes,persistentvolumeclaims

Procedure

1. Update the MigPlan CR to include Kubernetes resources and, optionally, to filter the included resources by adding the labelSelector parameter:

   1. To update the MigPlan CR to include Kubernetes resources:

      apiVersion: migration.openshift.io/v1alpha1
      kind: MigPlan
      metadata:
        name: <migplan>
        namespace: openshift-migration
      spec:
        includedResources:
        - kind: <kind> 1
          group: ""
        - kind: <kind>
          group: ""

      1

      Specify the Kubernetes object, for example, Secret or ConfigMap.

   2. Optional: To filter the included resources by adding the labelSelector parameter:

      apiVersion: migration.openshift.io/v1alpha1
      kind: MigPlan
      metadata:
        name: <migplan>
        namespace: openshift-migration
      spec:
        includedResources:
        - kind: <kind> 1
          group: ""
        - kind: <kind>
          group: ""
      ...
        labelSelector:
          matchLabels:
            <label> 2

      1

      Specify the Kubernetes object, for example, Secret or ConfigMap.

      2

      Specify the label of the resources to migrate, for example, app: frontend.

2. Create a MigMigration CR to migrate the selected Kubernetes resources. Verify that the correct MigPlan is referenced in migPlanRef:

   apiVersion: migration.openshift.io/v1alpha1
   kind: MigMigration
   metadata:
     generateName: <migplan>
     namespace: openshift-migration
   spec:
     migPlanRef:
       name: <migplan>
       namespace: openshift-migration
     stage: false

Procedure

1. Edit the MigrationController custom resource (CR) manifest:

   $ oc edit migrationcontroller -n openshift-migration

2. Update the following parameters:

   ...
   mig_controller_limits_cpu: "1" 1
   mig_controller_limits_memory: "10Gi" 2
   ...
   mig_controller_requests_cpu: "100m" 3
   mig_controller_requests_memory: "350Mi" 4
   ...
   mig_pv_limit: 100 5
   mig_pod_limit: 100 6
   mig_namespace_limit: 10 7
   ...

   1

   Specifies the number of CPUs available to the MigrationController CR.

   2

   Specifies the amount of memory available to the MigrationController CR.

   3

   Specifies the number of CPU units available for MigrationController CR requests. 100m represents 0.1 CPU units (100 * 1e-3).

   4

   Specifies the amount of memory available for MigrationController CR requests.

   5

   Specifies the number of persistent volumes that can be migrated.

   6

   Specifies the number of pods that can be migrated.

   7

   Specifies the number of namespaces that can be migrated.

3. Create a migration plan that uses the updated parameters to verify the changes.

   If your migration plan exceeds the MigrationController CR limits, the MTC console displays a warning message when you save the migration plan.

Procedure

1. Log in to the host cluster.

2. Enable PV resizing by patching the MigrationController CR:

   $ oc patch migrationcontroller migration-controller -p '{"spec":{"enable_dvm_pv_resizing":true}}' \ 1
     --type='merge' -n openshift-migration

   1

   Set the value to false to disable PV resizing.

3. Optional: Update the pv_resizing_threshold parameter to increase the threshold:

   $ oc patch migrationcontroller migration-controller -p '{"spec":{"pv_resizing_threshold":41}}' \ 1
     --type='merge' -n openshift-migration

   1

   The default value is 3.

   When the threshold is exceeded, the following status message is displayed in the MigPlan CR status:

   status:
     conditions:
   ...
     - category: Warn
       durable: true
       lastTransitionTime: "2021-06-17T08:57:01Z"
       message: 'Capacity of the following volumes will be automatically adjusted to avoid disk capacity issues in the target cluster:  [pvc-b800eb7b-cf3b-11eb-a3f7-0eae3e0555f3]'
       reason: Done
       status: "False"
       type: PvCapacityAdjustmentRequired

   Note

   For AWS gp2 storage, this message does not appear unless the pv_resizing_threshold is 42% or greater because of the way gp2 calculates volume usage and size. (BZ#1973148)

Procedure

1. Enable cached clients by running the following command:

   $ oc -n openshift-migration patch migrationcontroller migration-controller --type=json --patch \
     '[{ "op": "replace", "path": "/spec/mig_controller_enable_cache", "value": true}]'

2. Optional: Increase the MigrationController CR memory limits by running the following command:

   $ oc -n openshift-migration patch migrationcontroller migration-controller --type=json --patch \
     '[{ "op": "replace", "path": "/spec/mig_controller_limits_memory", "value": <10Gi>}]'

3. Optional: Increase the MigrationController CR memory requests by running the following command:

   $ oc -n openshift-migration patch migrationcontroller migration-controller --type=json --patch \
     '[{ "op": "replace", "path": "/spec/mig_controller_requests_memory", "value": <350Mi>}]'

Procedure

1. In the MTC web console, click Migration Plans.
2. Click the Migrations number next to a migration plan to view the Migrations page.
3. Click a migration to view the Migration details.

4. Expand Migration resources to view the migration resources and their status in a tree view.

   Note

   To troubleshoot a failed migration, start with a high-level resource that has failed and then work down the resource tree towards the lower-level resources.

5. Click the Options menu next to a resource and select one of the following options:

   • Copy oc describe command copies the command to your clipboard.

     • Log in to the relevant cluster and then run the command.

       The conditions and events of the resource are displayed in YAML format.

   • Copy oc logs command copies the command to your clipboard.

     • Log in to the relevant cluster and then run the command.

       If the resource supports log filtering, a filtered log is displayed.

   • View JSON displays the resource data in JSON format in a web browser.

     The data is the same as the output for the oc get <resource> command.

Procedure

1. In the MTC web console, click Migration Plans.
2. Click the Migrations number next to a migration plan.
3. Click View logs.
4. Click the Copy icon to copy the oc logs command to your clipboard.

5. Log in to the relevant cluster and enter the command on the CLI.

   The aggregated log for the migration plan is displayed.

Procedure

1. Get the mig-log-reader pod:

   $ oc -n openshift-migration get pods | grep log

2. Enter the following command to display a single migration log:

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 1

   1

   The -c plain option displays the log without colors.

Procedure

1. In the OpenShift Container Platform web console, click Monitoring → Metrics.

2. Enter a PromQL query, select a time window to display, and click Run Queries.

   If your web browser does not display all the results, use the Prometheus console.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command for one of the following data collection options:

   • To collect data for the past hour:

     $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.7

     The data is saved as must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.

   • To collect data for the past 24 hours:

     $ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v1.7 \
       -- /usr/bin/gather_metrics_dump

     This operation can take a long time. The data is saved as must-gather/metrics/prom_data.tar.gz.

Procedure

1. Decompress the prom_data.tar.gz file:

   $ tar -xvzf must-gather/metrics/prom_data.tar.gz

2. Create a local Prometheus instance:

   $ make prometheus-run

   The command outputs the Prometheus URL.

   Output

   Started Prometheus on http://localhost:9090

3. Launch a web browser and navigate to the URL to view the data by using the Prometheus web console.

4. After you have viewed the data, delete the Prometheus instance and data:

   $ make prometheus-cleanup

Procedure

1. Check the status of a MigMigration CR:

   $ oc get migmigration <migmigration> -o yaml

   Example output

   status:
     conditions:
     - category: Warn
       durable: true
       lastTransitionTime: "2021-01-26T20:48:40Z"
       message: 'Final Restore openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf: partially failed on destination cluster'
       status: "True"
       type: VeleroFinalRestorePartiallyFailed
     - category: Advisory
       durable: true
       lastTransitionTime: "2021-01-26T20:48:42Z"
       message: The migration has completed with warnings, please look at `Warn` conditions.
       reason: Completed
       status: "True"
       type: SucceededWithWarnings

2. Check the status of the Restore CR by using the Velero describe command:

   $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
     restore describe <restore>

   Example output

   Phase:  PartiallyFailed (run 'velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf' for more information)
   
   Errors:
     Velero:     <none>
     Cluster:    <none>
     Namespaces:
       migration-example:  error restoring example.com/migration-example/migration-example: the server could not find the requested resource

3. Check the Restore CR logs by using the Velero logs command:

   $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
     restore logs <restore>

   Example output

   time="2021-01-26T20:48:37Z" level=info msg="Attempting to restore migration-example: migration-example" logSource="pkg/restore/restore.go:1107" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
   time="2021-01-26T20:48:37Z" level=info msg="error restoring migration-example: the server could not find the requested resource" logSource="pkg/restore/restore.go:1170" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf

   The Restore CR log error message, the server could not find the requested resource, indicates the cause of the partially failed migration.

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   $ oc get migmigration -n openshift-migration

   Example output

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. Inspect the MigMigration CR:

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Ensure that the internal registries are exposed on the OpenShift Container Platform 3 and 4 clusters.

   The internal registry is exposed by default on OpenShift Container Platform 4.

2. If you are using insecure registries, add your registry host values to the [registries.insecure] section of /etc/container/registries.conf to ensure that podman does not encounter a TLS verification error.

3. Log in to the OpenShift Container Platform 3 registry:

   $ podman login -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false <registry_url>:<port>

4. Log in to the OpenShift Container Platform 4 registry:

   $ podman login -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false <registry_url>:<port>

5. Pull the OpenShift Container Platform 3 image:

   $ podman pull <registry_url>:<port>/openshift/<image>

6. Tag the OpenShift Container Platform 3 image for the OpenShift Container Platform 4 registry:

   $ podman tag <registry_url>:<port>/openshift/<image> \ 1
     <registry_url>:<port>/openshift/<image> 2

   1

   Specify the registry URL and port for the OpenShift Container Platform 3 cluster.

   2

   Specify the registry URL and port for the OpenShift Container Platform 4 cluster.

7. Push the image to the OpenShift Container Platform 4 registry:

   $ podman push <registry_url>:<port>/openshift/<image> 1

   1

   Specify the OpenShift Container Platform 4 cluster.

8. Verify that the image has a valid image stream:

   $ oc get imagestream -n openshift | grep <image>

   Example output

   NAME      IMAGE REPOSITORY                                                      TAGS    UPDATED
   my_image  image-registry.openshift-image-registry.svc:5000/openshift/my_image  latest  32 seconds ago

Procedure

1. Check the status of the MigMigration CR:

   $ oc describe migmigration <pod> -n openshift-migration

   The output includes the following status message:

   Example output

   Some or all transfer pods are not running for more than 10 mins on destination cluster

2. On the source cluster, obtain the details of a migrated namespace:

   $ oc get namespace <namespace> -o yaml 1

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   $ oc edit namespace <namespace>

4. Add the missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

5. Run the migration plan again.

Procedure

1. In the MTC web console, click Migration plans.
2. Click the Options menu beside a migration plan and select Rollback under Migration.

3. Click Rollback and wait for rollback to complete.

   In the migration plan details, Rollback succeeded is displayed.

4. Verify that rollback was successful in the OpenShift Container Platform web console of the source cluster:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. Create a MigMigration CR based on the following example:

   $ cat << EOF | oc apply -f -
   apiVersion: migration.openshift.io/v1alpha1
   kind: MigMigration
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: <migmigration>
     namespace: openshift-migration
   spec:
   ...
     rollback: true
   ...
     migPlanRef:
       name: <migplan> 1
       namespace: openshift-migration
   EOF

   1

   Specify the name of the associated MigPlan CR.

2. In the MTC web console, verify that the migrated project resources have been removed from the target cluster.
3. Verify that the migrated project resources are present in the source cluster and that the application is running.

Procedure

1. Delete the stage pods on all clusters:

   $ oc delete $(oc get pods -l migration.openshift.io/is-stage-pod -n <namespace>) 1

   1

   Namespaces specified in the MigPlan CR.

2. Unquiesce the application on the source cluster by scaling the replicas to their premigration number:

   $ oc scale deployment <deployment> --replicas=<premigration_replicas>

   The migration.openshift.io/preQuiesceReplicas annotation in the Deployment CR displays the premigration number of replicas:

   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     annotations:
       deployment.kubernetes.io/revision: "1"
       migration.openshift.io/preQuiesceReplicas: "1"

3. Verify that the application pods are running on the source cluster:

   $ oc get pod -n <namespace>