Chapter 3. Migrating from OpenShift Container Platform 4.2 and later

1. Install the Cluster Application Migration Operator on all clusters.

   You can install the Cluster Application Migration Operator in a restricted environment with limited or no internet access. The source and target clusters must have network access to each other and to a mirror registry.

2. Configure the replication repository, an intermediate object storage that the CAM tool uses to migrate data.

   The source and target clusters must have network access to the replication repository during migration. In a restricted environment, you can use an internally hosted S3 storage repository. If you use a proxy server, you must ensure that replication repository is whitelisted.

3. Add the source cluster to the CAM web console.
4. Add the replication repository to the CAM web console.

5. Create a migration plan, with one of the following data migration options:

   • Copy: The CAM tool copies the data from the source cluster to the replication repository, and from the replication repository to the target cluster.

     

   • Move: The CAM tool unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.

     Note

     Although the replication repository does not appear in this diagram, it is required for the actual migration.

     

6. Run the migration plan, with one of the following options:

   • Stage (optional) copies data to the target cluster without stopping the application.

     Staging can be run multiple times so that most of the data is copied to the target before migration. This minimizes the actual migration time and application downtime.

   • Migrate stops the application on the source cluster and recreates its resources on the target cluster. Optionally, you can migrate the workload without stopping the application.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed the CAM tool.

2. Obtain the CAM web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the CAM web console.

   Note

   If you try to access the CAM web console immediately after installing the Cluster Application Migration Operator, the console may not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster’s API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket <bucket_name> \ 1
       --region <bucket_region> 2

   1

   Specify your S3 bucket name.

   2

   Specify your S3 bucket region, for example, us-east-1.

2. Create the IAM user velero:

   $ aws iam create-user --user-name velero

3. Create an EC2 EBS snapshot policy:

   $ cat > velero-ec2-snapshot-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           }
       ]
   }
   EOF

4. Create an AWS S3 access policy for one or for all S3 buckets:

   $ cat > velero-s3-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>/*" 1
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>" 2
               ]
           }
       ]
   }
   EOF

   1 2

   To grant access to a single S3 bucket, specify the bucket name. To grant access to all AWS S3 buckets, specify * instead of a bucket name:

   "Resource": [
       "arn:aws:s3:::*"

5. Attach the EC2 EBS policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-ebs \
     --policy-document file://velero-ec2-snapshot-policy.json

6. Attach the AWS S3 policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-s3 \
     --policy-document file://velero-s3-policy.json

7. Create an access key for velero:

   $ aws iam create-access-key --user-name velero
   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1
           "AccessKeyId": <AWS_ACCESS_KEY_ID> 2
       }
   }

   1 2

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID for adding the AWS repository to the CAM web console.

Procedure

1. Run gsutil init to log in:

   $ gsutil init
   Welcome! This command will take you through the configuration of gcloud.
   
   Your current configuration has been set to: [default]
   
   To continue, you must login. Would you like to login (Y/n)?

2. Set the BUCKET variable:

   $ BUCKET=<bucket_name> 1

   1

   Specify your bucket name.

3. Create a storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a velero service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero Storage"

6. Set the SERVICE_ACCOUNT_EMAIL variable to the service account’s email address:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
     --filter="displayName:Velero Storage" \
     --format 'value(email)')

7. Grant permissions to the service account:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )
   
   gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
   
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
       --role projects/$PROJECT_ID/roles/velero.server
   
   gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

8. Save the service account’s keys to the credentials-velero file in the current directory:

   $ gcloud iam service-accounts keys create credentials-velero \
     --iam-account $SERVICE_ACCOUNT_EMAIL

Procedure

1. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

2. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1

   1

   Specify your location.

3. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID=velerobackups

4. Create an Azure storage account:

   $ az storage account create \
     --name $AZURE_STORAGE_ACCOUNT_ID \
     --resource-group $AZURE_RESOURCE_GROUP \
     --sku Standard_GRS \
     --encryption-services blob \
     --https-only true \
     --kind BlobStorage \
     --access-tier Hot

5. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

6. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

7. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
   $ AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
   $ AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv`
   $ AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

8. Save the service principal’s credentials in the credentials-velero file:

   $ cat << EOF  > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

Procedure

1. Log in to the cluster.

2. Obtain the service account token:

   $ oc sa get-token mig -n openshift-migration
   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. Log in to the CAM web console.
4. In the Clusters section, click Add cluster.

5. Fill in the following fields:

   • Cluster name: May contain lower-case letters (a-z) and numbers (0-9). Must not contain spaces or international characters.
   • Url: URL of the cluster’s API server, for example, https://<master1.example.com>:8443.
   • Service account token: String that you obtained from the source cluster.
   • Azure cluster: Optional. Select it if you are using Azure snapshots to copy your data.
   • Azure resource group: This field appears if Azure cluster is checked.
   • If you use a custom CA bundle, click Browse and browse to the CA bundle file.

6. Click Add cluster.

   The cluster appears in the Clusters section.

Procedure

1. Log in to the CAM web console.
2. In the Replication repositories section, click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for AWS S3, MCG, and generic S3 providers:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • S3 bucket name: Specify the name of the S3 bucket you created.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for other S3 providers.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG.
     • Require SSL verification: Clear this check box if you are using a generic S3 provider.
     • If you use a custom CA bundle, click Browse and browse to the Base64-encoded CA bundle file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories section.

Procedure

1. Edit the Migration controller CR:

   $ oc get migrationcontroller -n openshift-migration
   NAME AGE
   migration-controller 5d19h
   
   $ oc edit migrationcontroller -n openshift-migration

2. Update the following parameters:

   ...
   migration_controller: true
   
   # This configuration is loaded into mig-controller, and should be set on the
   # cluster where `migration_controller: true`
   mig_pv_limit: 100
   mig_pod_limit: 100
   mig_namespace_limit: 10
   ...

Procedure

1. Log in to the CAM web console.
2. In the Plans section, click Add plan.

3. Enter the Plan name and click Next.

   The Plan name can contain up to 253 lower-case alphanumeric characters (a-z, 0-9). It must not contain spaces or underscores (_).

4. Select a Source cluster.
5. Select a Target cluster.
6. Select a Replication repository.
7. Select the projects to be migrated and click Next.

8. Select Copy or Move for the PVs:

   • Copy copies the data in a source cluster’s PV to the replication repository and then restores it on a newly created PV, with similar characteristics, in the target cluster.

     Optional: You can verify data copied with the filesystem method by selecting Verify copy. This option, which generates a checksum for each source file and checks it after restoration, significantly reduces performance.

   • Move unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.
9. Click Next.

10. Select a Copy method for the PVs:

    • Snapshot backs up and restores the disk using the cloud provider’s snapshot functionality. It is significantly faster than Filesystem.

      Note

      The storage and clusters must be in the same region and the storage class must be compatible.

    • Filesystem copies the data files from the source disk to a newly created target disk.

11. Select a Storage class for the PVs.

    If you selected the Filesystem copy method, you can change the storage class during migration, for example, from Red Hat Gluster Storage or NFS storage to Red Hat Ceph Storage.

12. Click Next.

13. If you want to add a migration hook, click Add Hook and perform the following steps:

    1. Specify the name of the hook.
    2. Select Ansible playbook to use your own playbook or Custom container image for a hook written in another language.
    3. Click Browse to upload the playbook.
    4. Optional: If you are not using the default Ansible runtime image, specify your custom Ansible image.
    5. Specify the cluster on which you want the hook to run.
    6. Specify the service account name.
    7. Specify the namespace.

    8. Select the migration step at which you want the hook to run:

       • PreBackup: Before backup tasks are started on the source cluster
       • PostBackup: After backup tasks are complete on the source cluster
       • PreRestore: Before restore tasks are started on the target cluster
       • PostRestore: After restore tasks are complete on the target cluster

14. Click Add.

    You can add up to four hooks to a migration plan, assigning each hook to a different migration step.

15. Click Finish.

16. Click Close.

    The migration plan appears in the Plans section.

Procedure

1. Log in to the CAM web console on the target cluster.
2. Select a migration plan.

3. Click Stage to copy data from the source cluster to the target cluster without stopping the application.

   You can run Stage multiple times to reduce the actual migration time.

4. When you are ready to migrate the application workload, click Migrate.

   Migrate stops the application workload on the source cluster and recreates its resources on the target cluster.

5. Optional: In the Migrate window, you can select Do not stop applications on the source cluster during migration.
6. Click Migrate.
7. Optional: To stop a migration in progress, click the Options menu and select Cancel.

8. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads Pods to verify that the Pods are running in the migrated namespace.
   5. Click Storage Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. Get the CR name:

   $ oc get <migration_cr> -n openshift-migration 1

   1

   Specify the migration CR, for example, migmigration.

   The output is similar to the following:

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. View the CR:

   $ oc describe <migration_cr> <88435fe0-c9f8-11e9-85e6-5d593ce65e10> -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Log in to the CAM console.
2. Click Plans to view the list of migration plans.
3. Click the Options menu of a specific migration plan and select Logs.
4. Click Download Logs to download the logs of the Migration controller, Velero, and Restic for all clusters.

5. To download a specific log:

   1. Specify the log options:

      • Cluster: Select the source, target, or CAM host cluster.
      • Log source: Select Velero, Restic, or Controller.

      • Pod source: Select the Pod name, for example, controller-manager-78c469849c-v6wcf

        The selected log is displayed.

        You can clear the log selection settings by changing your selection.

   2. Click Download Selected to download the selected log.

Procedure

1. On the target cluster, switch to the migrated project:

   $ oc project <project>

2. Get the deployed resources:

   $ oc get all

3. Delete the deployed resources to ensure that the application is not running on the target cluster and accessing data on the PVC:

   $ oc delete <resource_type>

4. To stop a DaemonSet without deleting it, update the nodeSelector in the YAML file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset
     template:
       metadata:
         labels:
           name: hello-daemonset
       spec:
         nodeSelector:
           role: worker 1

   1

   Specify a nodeSelector value that does not exist on any node.

5. Update each PV’s reclaim policy so that unnecessary data is removed. During migration, the reclaim policy for bound PVs is Retain, to ensure that data is not lost when an application is removed from the source cluster. You can remove these PVs during rollback.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain 1
     ...
   status:
     ...

   1

   Specify Recycle or Delete.

6. On the source cluster, switch to your migrated project:

   $ oc project <project_name>

7. Obtain the project’s deployed resources:

   $ oc get all

8. Start one or more replicas of each deployed resource:

   $ oc scale --replicas=1 <resource_type>/<resource_name>

9. Update the nodeSelector of a DaemonSet to its original value, if you changed it during the procedure.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8

   The must-gather tool collects the cluster information and stores it in a must-gather.local.<uid> directory.

3. Remove authentication keys and other sensitive information from the must-gather data.

4. Create an archive file containing the contents of the must-gather.local.<uid> directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

   You can attach the compressed file to your customer support case on the Red Hat Customer Portal.