Migration

1. Install the Cluster Application Migration Operator on all clusters.

   You can install the Cluster Application Migration Operator in a restricted environment with limited or no internet access. The source and target clusters must have network access to each other and to a mirror registry.

2. Configure the replication repository, an intermediate object storage that the CAM tool uses to migrate data.

   The source and target clusters must have network access to the replication repository during migration. In a restricted environment, you can use an internally hosted S3 storage repository. If you use a proxy server, you must ensure that replication repository is whitelisted.

3. Add the source cluster to the CAM web console.
4. Add the replication repository to the CAM web console.

5. Create a migration plan, with one of the following data migration options:

   • Copy: The CAM tool copies the data from the source cluster to the replication repository, and from the replication repository to the target cluster.

     

   • Move: The CAM tool unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.

     Note

     Although the replication repository does not appear in this diagram, it is required for the actual migration.

     

6. Run the migration plan, with one of the following options:

   • Stage (optional) copies data to the target cluster without stopping the application.

     Staging can be run multiple times so that most of the data is copied to the target before migration. This minimizes the actual migration time and application downtime.

   • Migrate stops the application on the source cluster and recreates its resources on the target cluster. Optionally, you can migrate the workload without stopping the application.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed the CAM tool.

2. Obtain the CAM web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the CAM web console.

   Note

   If you try to access the CAM web console immediately after installing the Cluster Application Migration Operator, the console may not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster’s API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket <bucket_name> \ 1
       --region <bucket_region> 2

   1

   Specify your S3 bucket name.

   2

   Specify your S3 bucket region, for example, us-east-1.

2. Create the IAM user velero:

   $ aws iam create-user --user-name velero

3. Create an EC2 EBS snapshot policy:

   $ cat > velero-ec2-snapshot-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           }
       ]
   }
   EOF

4. Create an AWS S3 access policy for one or for all S3 buckets:

   $ cat > velero-s3-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>/*" 1
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>" 2
               ]
           }
       ]
   }
   EOF

   1 2

   To grant access to a single S3 bucket, specify the bucket name. To grant access to all AWS S3 buckets, specify * instead of a bucket name:

   "Resource": [
       "arn:aws:s3:::*"

5. Attach the EC2 EBS policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-ebs \
     --policy-document file://velero-ec2-snapshot-policy.json

6. Attach the AWS S3 policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-s3 \
     --policy-document file://velero-s3-policy.json

7. Create an access key for velero:

   $ aws iam create-access-key --user-name velero
   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1
           "AccessKeyId": <AWS_ACCESS_KEY_ID> 2
       }
   }

   1 2

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID for adding the AWS repository to the CAM web console.

Procedure

1. Run gsutil init to log in:

   $ gsutil init
   Welcome! This command will take you through the configuration of gcloud.
   
   Your current configuration has been set to: [default]
   
   To continue, you must login. Would you like to login (Y/n)?

2. Set the BUCKET variable:

   $ BUCKET=<bucket_name> 1

   1

   Specify your bucket name.

3. Create a storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a velero service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero Storage"

6. Set the SERVICE_ACCOUNT_EMAIL variable to the service account’s email address:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
     --filter="displayName:Velero Storage" \
     --format 'value(email)')

7. Grant permissions to the service account:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )
   
   gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
   
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
       --role projects/$PROJECT_ID/roles/velero.server
   
   gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

8. Save the service account’s keys to the credentials-velero file in the current directory:

   $ gcloud iam service-accounts keys create credentials-velero \
     --iam-account $SERVICE_ACCOUNT_EMAIL

Procedure

1. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

2. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1

   1

   Specify your location.

3. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID=velerobackups

4. Create an Azure storage account:

   $ az storage account create \
     --name $AZURE_STORAGE_ACCOUNT_ID \
     --resource-group $AZURE_RESOURCE_GROUP \
     --sku Standard_GRS \
     --encryption-services blob \
     --https-only true \
     --kind BlobStorage \
     --access-tier Hot

5. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

6. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

7. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
   $ AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
   $ AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv`
   $ AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

8. Save the service principal’s credentials in the credentials-velero file:

   $ cat << EOF  > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

Procedure

1. Log in to the cluster.

2. Obtain the service account token:

   $ oc sa get-token mig -n openshift-migration
   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. Log in to the CAM web console.
4. In the Clusters section, click Add cluster.

5. Fill in the following fields:

   • Cluster name: May contain lower-case letters (a-z) and numbers (0-9). Must not contain spaces or international characters.
   • Url: URL of the cluster’s API server, for example, https://<master1.example.com>:8443.
   • Service account token: String that you obtained from the source cluster.
   • Azure cluster: Optional. Select it if you are using Azure snapshots to copy your data.
   • Azure resource group: This field appears if Azure cluster is checked.
   • If you use a custom CA bundle, click Browse and browse to the CA bundle file.

6. Click Add cluster.

   The cluster appears in the Clusters section.

Procedure

1. Log in to the CAM web console.
2. In the Replication repositories section, click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for AWS S3, MCG, and generic S3 providers:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • S3 bucket name: Specify the name of the S3 bucket you created.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for other S3 providers.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG.
     • Require SSL verification: Clear this check box if you are using a generic S3 provider.
     • If you use a custom CA bundle, click Browse and browse to the Base64-encoded CA bundle file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories section.

Procedure

1. Edit the Migration controller CR:

   $ oc get migrationcontroller -n openshift-migration
   NAME AGE
   migration-controller 5d19h
   
   $ oc edit migrationcontroller -n openshift-migration

2. Update the following parameters:

   ...
   migration_controller: true
   
   # This configuration is loaded into mig-controller, and should be set on the
   # cluster where `migration_controller: true`
   mig_pv_limit: 100
   mig_pod_limit: 100
   mig_namespace_limit: 10
   ...

Procedure

1. Log in to the CAM web console.
2. In the Plans section, click Add plan.

3. Enter the Plan name and click Next.

   The Plan name can contain up to 253 lower-case alphanumeric characters (a-z, 0-9). It must not contain spaces or underscores (_).

4. Select a Source cluster.
5. Select a Target cluster.
6. Select a Replication repository.
7. Select the projects to be migrated and click Next.

8. Select Copy or Move for the PVs:

   • Copy copies the data in a source cluster’s PV to the replication repository and then restores it on a newly created PV, with similar characteristics, in the target cluster.

     Optional: You can verify data copied with the filesystem method by selecting Verify copy. This option, which generates a checksum for each source file and checks it after restoration, significantly reduces performance.

   • Move unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.
9. Click Next.

10. Select a Copy method for the PVs:

    • Snapshot backs up and restores the disk using the cloud provider’s snapshot functionality. It is significantly faster than Filesystem.

      Note

      The storage and clusters must be in the same region and the storage class must be compatible.

    • Filesystem copies the data files from the source disk to a newly created target disk.

11. Select a Storage class for the PVs.

    If you selected the Filesystem copy method, you can change the storage class during migration, for example, from Red Hat Gluster Storage or NFS storage to Red Hat Ceph Storage.

12. Click Next.

13. If you want to add a migration hook, click Add Hook and perform the following steps:

    1. Specify the name of the hook.
    2. Select Ansible playbook to use your own playbook or Custom container image for a hook written in another language.
    3. Click Browse to upload the playbook.
    4. Optional: If you are not using the default Ansible runtime image, specify your custom Ansible image.
    5. Specify the cluster on which you want the hook to run.
    6. Specify the service account name.
    7. Specify the namespace.

    8. Select the migration step at which you want the hook to run:

       • PreBackup: Before backup tasks are started on the source cluster
       • PostBackup: After backup tasks are complete on the source cluster
       • PreRestore: Before restore tasks are started on the target cluster
       • PostRestore: After restore tasks are complete on the target cluster

14. Click Add.

    You can add up to four hooks to a migration plan, assigning each hook to a different migration step.

15. Click Finish.

16. Click Close.

    The migration plan appears in the Plans section.

Procedure

1. Log in to the CAM web console on the target cluster.
2. Select a migration plan.

3. Click Stage to copy data from the source cluster to the target cluster without stopping the application.

   You can run Stage multiple times to reduce the actual migration time.

4. When you are ready to migrate the application workload, click Migrate.

   Migrate stops the application workload on the source cluster and recreates its resources on the target cluster.

5. Optional: In the Migrate window, you can select Do not stop applications on the source cluster during migration.
6. Click Migrate.
7. Optional: To stop a migration in progress, click the Options menu and select Cancel.

8. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the Pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. In the Red Hat Customer Portal, navigate to Downloads → Red Hat OpenShift Container Platform.
2. On the Download Red Hat OpenShift Container Platform page, select Red Hat OpenShift Container Platform from the Product Variant list.
3. Select CPMA 1.0 for RHEL 7 from the Version list. This binary works on RHEL 7 and RHEL 8.
4. Click Download Now to download cpma for Linux or MacOSX or cpma.exe for Windows.
5. Save the file in a directory defined as $PATH for Linux or MacOSX or %PATH% for Windows.

6. For Linux, make the file executable:

   $ sudo chmod +x cpma

Procedure

1. Log in to the OpenShift Container Platform 3 cluster:

   $ oc login https://<master1.example.com> 1

   1

   OpenShift Container Platform 3 master node. You must be logged in to the cluster to receive a token for the Kubernetes and OpenShift Container Platform APIs.

2. Run the CPMA. Each prompt requires you to provide input, as in the following example:

   $ cpma --manifests=false 1
   ? Do you wish to save configuration for future use? true
   ? What will be the source for OCP3 config files? Remote host 2
   ? Path to crio config file /etc/crio/crio.conf
   ? Path to etcd config file /etc/etcd/etcd.conf
   ? Path to master config file /etc/origin/master/master-config.yaml
   ? Path to node config file /etc/origin/node/node-config.yaml
   ? Path to registries config file /etc/containers/registries.conf
   ? Do wish to find source cluster using KUBECONFIG or prompt it? KUBECONFIG
   ? Select cluster obtained from KUBECONFIG contexts master1-example-com:443
   ? Select master node master1.example.com
   ? SSH login root 3
   ? SSH Port 22
   ? Path to private SSH key /home/user/.ssh/openshift_key
   ? Path to application data, skip to use current directory .
   INFO[29 Aug 19 00:07 UTC] Starting manifest and report generation
   INFO[29 Aug 19 00:07 UTC] Transform:Starting for - API
   INFO[29 Aug 19 00:07 UTC] APITransform::Extract
   INFO[29 Aug 19 00:07 UTC] APITransform::Transform:Reports
   INFO[29 Aug 19 00:07 UTC] Transform:Starting for - Cluster
   INFO[29 Aug 19 00:08 UTC] ClusterTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportQuotas
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportPVs
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportNamespaces
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportNodes
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportRBAC
   INFO[29 Aug 19 00:08 UTC] ClusterReport::ReportStorageClasses
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - Crio
   INFO[29 Aug 19 00:08 UTC] CrioTransform::Extract
   WARN[29 Aug 19 00:08 UTC] Skipping Crio: No configuration file available
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - Docker
   INFO[29 Aug 19 00:08 UTC] DockerTransform::Extract
   INFO[29 Aug 19 00:08 UTC] DockerTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - ETCD
   INFO[29 Aug 19 00:08 UTC] ETCDTransform::Extract
   INFO[29 Aug 19 00:08 UTC] ETCDTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - OAuth
   INFO[29 Aug 19 00:08 UTC] OAuthTransform::Extract
   INFO[29 Aug 19 00:08 UTC] OAuthTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - SDN
   INFO[29 Aug 19 00:08 UTC] SDNTransform::Extract
   INFO[29 Aug 19 00:08 UTC] SDNTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - Image
   INFO[29 Aug 19 00:08 UTC] ImageTransform::Extract
   INFO[29 Aug 19 00:08 UTC] ImageTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Transform:Starting for - Project
   INFO[29 Aug 19 00:08 UTC] ProjectTransform::Extract
   INFO[29 Aug 19 00:08 UTC] ProjectTransform::Transform:Reports
   INFO[29 Aug 19 00:08 UTC] Flushing reports to disk
   INFO[29 Aug 19 00:08 UTC] Report:Added: report.json
   INFO[29 Aug 19 00:08 UTC] Report:Added: report.html
   INFO[29 Aug 19 00:08 UTC] Successfully finished transformations

   1

   --manifests=false: Without generating CR manifests

   2

   Remote host: Remote mode

   3

   SSH login: The SSH user must have sudo permissions on the OpenShift Container Platform 3 cluster in order to access the configuration files.

   The CPMA creates the following files and directory in the current directory if you did not specify an output directory:

   • cpma.yaml file: Configuration options that you provided when you ran the CPMA
   • master1.example.com/: Configuration files from the master node
   • report.json: JSON-encoded report
   • report.html: HTML-encoded report
3. Open the report.html file in a browser to view the CPMA report.

4. If you generate CR manifests, apply the CR manifests to the OpenShift Container Platform 4.2 cluster, as in the following example:

   $ oc apply -f 100_CPMA-cluster-config-secret-htpasswd-secret.yaml

Procedure

1. Get the CR name:

   $ oc get <migration_cr> -n openshift-migration 1

   1

   Specify the migration CR, for example, migmigration.

   The output is similar to the following:

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. View the CR:

   $ oc describe <migration_cr> <88435fe0-c9f8-11e9-85e6-5d593ce65e10> -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Log in to the CAM console.
2. Click Plans to view the list of migration plans.
3. Click the Options menu of a specific migration plan and select Logs.
4. Click Download Logs to download the logs of the Migration controller, Velero, and Restic for all clusters.

5. To download a specific log:

   1. Specify the log options:

      • Cluster: Select the source, target, or CAM host cluster.
      • Log source: Select Velero, Restic, or Controller.

      • Pod source: Select the Pod name, for example, controller-manager-78c469849c-v6wcf

        The selected log is displayed.

        You can clear the log selection settings by changing your selection.

   2. Click Download Selected to download the selected log.

Procedure

1. Run the migration plan.

2. View the MigPlan CR:

   $ oc describe migplan <migplan_name> -n openshift-migration 1

   1

   Specify the name of the migration plan.

   The output is similar to the following:

   metadata:
     ...
     uid: 79509e05-61d6-11e9-bc55-02ce4781844a 1
   status:
     ...
     conditions:
     - category: Warn
       lastTransitionTime: 2020-04-30T17:16:23Z
       message: 'Some namespaces contain GVKs incompatible with destination cluster.
         See: `incompatibleNamespaces` for details'
       status: "True"
       type: GVKsIncompatible
     incompatibleNamespaces:
     - gvks:
       - group: batch
         kind: cronjobs 2
         version: v2alpha1
       - group: batch
         kind: scheduledjobs 3
         version: v2alpha1

   1

   Record the MigPlan UID.

   2 3

   Record the deprecated GVKs.

3. Get the MigMigration name associated with the MigPlan UID:

   $ oc get migmigration -o json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=="<migplan_uid>") | .metadata.name' 1

   1

   Specify the MigPlan UID.

4. Get the MigMigration UID associated with the MigMigration name:

   $ oc get migmigration <migmigration_name> -o jsonpath='{.metadata.uid}' 1

   1

   Specify the MigMigration name.

5. Get the Velero Backup name associated with the MigMigration UID:

   $ oc get backup.velero.io --selector migration-initial-backup="<migmigration_uid>" -o jsonpath={.items[*].metadata.name} 1

   1

   Specify the MigMigration UID.

6. Download the contents of the Velero Backup to your local machine:

   • For AWS S3:

     $ aws s3 cp s3://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • For GCP:

     $ gsutil cp gs://<bucket_name>/velero/backups/<backup_name> <backup_local_dir> --recursive 1

     1

     Specify the bucket, backup name, and your local backup directory name.

   • For Azure:

     $ azcopy copy 'https://velerobackups.blob.core.windows.net/velero/backups/<backup_name>' '<backup_local_dir>' --recursive 1

     1

     Specify the backup name and your local backup directory name.

7. Extract the Velero Backup archive file:

   $ tar -xfv <backup_local_dir>/<backup_name>.tar.gz -C <backup_local_dir>

8. Run oc convert in offline mode on each deprecated GVK:

   $ oc convert <backup_local_dir>/resources/<gvk>.yaml 1

   1

   Specify the deprecated GVK.

9. Create the converted GVK on the target cluster:

   $ oc create -f <gvk>.yaml 1

   1

   Specify the converted GVK.

Procedure

1. On the target cluster, switch to the migrated project:

   $ oc project <project>

2. Get the deployed resources:

   $ oc get all

3. Delete the deployed resources to ensure that the application is not running on the target cluster and accessing data on the PVC:

   $ oc delete <resource_type>

4. To stop a DaemonSet without deleting it, update the nodeSelector in the YAML file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset
     template:
       metadata:
         labels:
           name: hello-daemonset
       spec:
         nodeSelector:
           role: worker 1

   1

   Specify a nodeSelector value that does not exist on any node.

5. Update each PV’s reclaim policy so that unnecessary data is removed. During migration, the reclaim policy for bound PVs is Retain, to ensure that data is not lost when an application is removed from the source cluster. You can remove these PVs during rollback.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain 1
     ...
   status:
     ...

   1

   Specify Recycle or Delete.

6. On the source cluster, switch to your migrated project:

   $ oc project <project_name>

7. Obtain the project’s deployed resources:

   $ oc get all

8. Start one or more replicas of each deployed resource:

   $ oc scale --replicas=1 <resource_type>/<resource_name>

9. Update the nodeSelector of a DaemonSet to its original value, if you changed it during the procedure.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8

   The must-gather tool collects the cluster information and stores it in a must-gather.local.<uid> directory.

3. Remove authentication keys and other sensitive information from the must-gather data.

4. Create an archive file containing the contents of the must-gather.local.<uid> directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

   You can attach the compressed file to your customer support case on the Red Hat Customer Portal.

1. Install the Cluster Application Migration Operator on all clusters.

   You can install the Cluster Application Migration Operator in a restricted environment with limited or no internet access. The source and target clusters must have network access to each other and to a mirror registry.

2. Configure the replication repository, an intermediate object storage that the CAM tool uses to migrate data.

   The source and target clusters must have network access to the replication repository during migration. In a restricted environment, you can use an internally hosted S3 storage repository. If you use a proxy server, you must ensure that replication repository is whitelisted.

3. Add the source cluster to the CAM web console.
4. Add the replication repository to the CAM web console.

5. Create a migration plan, with one of the following data migration options:

   • Copy: The CAM tool copies the data from the source cluster to the replication repository, and from the replication repository to the target cluster.

     

   • Move: The CAM tool unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.

     Note

     Although the replication repository does not appear in this diagram, it is required for the actual migration.

     

6. Run the migration plan, with one of the following options:

   • Stage (optional) copies data to the target cluster without stopping the application.

     Staging can be run multiple times so that most of the data is copied to the target before migration. This minimizes the actual migration time and application downtime.

   • Migrate stops the application on the source cluster and recreates its resources on the target cluster. Optionally, you can migrate the workload without stopping the application.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed the CAM tool.

2. Obtain the CAM web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the CAM web console.

   Note

   If you try to access the CAM web console immediately after installing the Cluster Application Migration Operator, the console may not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster’s API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket <bucket_name> \ 1
       --region <bucket_region> 2

   1

   Specify your S3 bucket name.

   2

   Specify your S3 bucket region, for example, us-east-1.

2. Create the IAM user velero:

   $ aws iam create-user --user-name velero

3. Create an EC2 EBS snapshot policy:

   $ cat > velero-ec2-snapshot-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           }
       ]
   }
   EOF

4. Create an AWS S3 access policy for one or for all S3 buckets:

   $ cat > velero-s3-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>/*" 1
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>" 2
               ]
           }
       ]
   }
   EOF

   1 2

   To grant access to a single S3 bucket, specify the bucket name. To grant access to all AWS S3 buckets, specify * instead of a bucket name:

   "Resource": [
       "arn:aws:s3:::*"

5. Attach the EC2 EBS policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-ebs \
     --policy-document file://velero-ec2-snapshot-policy.json

6. Attach the AWS S3 policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-s3 \
     --policy-document file://velero-s3-policy.json

7. Create an access key for velero:

   $ aws iam create-access-key --user-name velero
   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1
           "AccessKeyId": <AWS_ACCESS_KEY_ID> 2
       }
   }

   1 2

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID for adding the AWS repository to the CAM web console.

Procedure

1. Run gsutil init to log in:

   $ gsutil init
   Welcome! This command will take you through the configuration of gcloud.
   
   Your current configuration has been set to: [default]
   
   To continue, you must login. Would you like to login (Y/n)?

2. Set the BUCKET variable:

   $ BUCKET=<bucket_name> 1

   1

   Specify your bucket name.

3. Create a storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a velero service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero Storage"

6. Set the SERVICE_ACCOUNT_EMAIL variable to the service account’s email address:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
     --filter="displayName:Velero Storage" \
     --format 'value(email)')

7. Grant permissions to the service account:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )
   
   gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
   
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
       --role projects/$PROJECT_ID/roles/velero.server
   
   gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

8. Save the service account’s keys to the credentials-velero file in the current directory:

   $ gcloud iam service-accounts keys create credentials-velero \
     --iam-account $SERVICE_ACCOUNT_EMAIL

Procedure

1. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

2. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1

   1

   Specify your location.

3. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID=velerobackups

4. Create an Azure storage account:

   $ az storage account create \
     --name $AZURE_STORAGE_ACCOUNT_ID \
     --resource-group $AZURE_RESOURCE_GROUP \
     --sku Standard_GRS \
     --encryption-services blob \
     --https-only true \
     --kind BlobStorage \
     --access-tier Hot

5. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

6. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

7. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
   $ AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
   $ AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv`
   $ AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

8. Save the service principal’s credentials in the credentials-velero file:

   $ cat << EOF  > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

Procedure

1. Log in to the cluster.

2. Obtain the service account token:

   $ oc sa get-token mig -n openshift-migration
   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. Log in to the CAM web console.
4. In the Clusters section, click Add cluster.

5. Fill in the following fields:

   • Cluster name: May contain lower-case letters (a-z) and numbers (0-9). Must not contain spaces or international characters.
   • Url: URL of the cluster’s API server, for example, https://<master1.example.com>:8443.
   • Service account token: String that you obtained from the source cluster.
   • Azure cluster: Optional. Select it if you are using Azure snapshots to copy your data.
   • Azure resource group: This field appears if Azure cluster is checked.
   • If you use a custom CA bundle, click Browse and browse to the CA bundle file.

6. Click Add cluster.

   The cluster appears in the Clusters section.

Procedure

1. Log in to the CAM web console.
2. In the Replication repositories section, click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for AWS S3, MCG, and generic S3 providers:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • S3 bucket name: Specify the name of the S3 bucket you created.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for other S3 providers.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG.
     • Require SSL verification: Clear this check box if you are using a generic S3 provider.
     • If you use a custom CA bundle, click Browse and browse to the Base64-encoded CA bundle file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories section.

Procedure

1. Edit the Migration controller CR:

   $ oc get migrationcontroller -n openshift-migration
   NAME AGE
   migration-controller 5d19h
   
   $ oc edit migrationcontroller -n openshift-migration

2. Update the following parameters:

   ...
   migration_controller: true
   
   # This configuration is loaded into mig-controller, and should be set on the
   # cluster where `migration_controller: true`
   mig_pv_limit: 100
   mig_pod_limit: 100
   mig_namespace_limit: 10
   ...

Procedure

1. Log in to the CAM web console.
2. In the Plans section, click Add plan.

3. Enter the Plan name and click Next.

   The Plan name can contain up to 253 lower-case alphanumeric characters (a-z, 0-9). It must not contain spaces or underscores (_).

4. Select a Source cluster.
5. Select a Target cluster.
6. Select a Replication repository.
7. Select the projects to be migrated and click Next.

8. Select Copy or Move for the PVs:

   • Copy copies the data in a source cluster’s PV to the replication repository and then restores it on a newly created PV, with similar characteristics, in the target cluster.

     Optional: You can verify data copied with the filesystem method by selecting Verify copy. This option, which generates a checksum for each source file and checks it after restoration, significantly reduces performance.

   • Move unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.
9. Click Next.

10. Select a Copy method for the PVs:

    • Snapshot backs up and restores the disk using the cloud provider’s snapshot functionality. It is significantly faster than Filesystem.

      Note

      The storage and clusters must be in the same region and the storage class must be compatible.

    • Filesystem copies the data files from the source disk to a newly created target disk.

11. Select a Storage class for the PVs.

    If you selected the Filesystem copy method, you can change the storage class during migration, for example, from Red Hat Gluster Storage or NFS storage to Red Hat Ceph Storage.

12. Click Next.

13. If you want to add a migration hook, click Add Hook and perform the following steps:

    1. Specify the name of the hook.
    2. Select Ansible playbook to use your own playbook or Custom container image for a hook written in another language.
    3. Click Browse to upload the playbook.
    4. Optional: If you are not using the default Ansible runtime image, specify your custom Ansible image.
    5. Specify the cluster on which you want the hook to run.
    6. Specify the service account name.
    7. Specify the namespace.

    8. Select the migration step at which you want the hook to run:

       • PreBackup: Before backup tasks are started on the source cluster
       • PostBackup: After backup tasks are complete on the source cluster
       • PreRestore: Before restore tasks are started on the target cluster
       • PostRestore: After restore tasks are complete on the target cluster

14. Click Add.

    You can add up to four hooks to a migration plan, assigning each hook to a different migration step.

15. Click Finish.

16. Click Close.

    The migration plan appears in the Plans section.

Procedure

1. Log in to the CAM web console on the target cluster.
2. Select a migration plan.

3. Click Stage to copy data from the source cluster to the target cluster without stopping the application.

   You can run Stage multiple times to reduce the actual migration time.

4. When you are ready to migrate the application workload, click Migrate.

   Migrate stops the application workload on the source cluster and recreates its resources on the target cluster.

5. Optional: In the Migrate window, you can select Do not stop applications on the source cluster during migration.
6. Click Migrate.
7. Optional: To stop a migration in progress, click the Options menu and select Cancel.

8. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the Pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. Get the CR name:

   $ oc get <migration_cr> -n openshift-migration 1

   1

   Specify the migration CR, for example, migmigration.

   The output is similar to the following:

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. View the CR:

   $ oc describe <migration_cr> <88435fe0-c9f8-11e9-85e6-5d593ce65e10> -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Log in to the CAM console.
2. Click Plans to view the list of migration plans.
3. Click the Options menu of a specific migration plan and select Logs.
4. Click Download Logs to download the logs of the Migration controller, Velero, and Restic for all clusters.

5. To download a specific log:

   1. Specify the log options:

      • Cluster: Select the source, target, or CAM host cluster.
      • Log source: Select Velero, Restic, or Controller.

      • Pod source: Select the Pod name, for example, controller-manager-78c469849c-v6wcf

        The selected log is displayed.

        You can clear the log selection settings by changing your selection.

   2. Click Download Selected to download the selected log.

Procedure

1. On the target cluster, switch to the migrated project:

   $ oc project <project>

2. Get the deployed resources:

   $ oc get all

3. Delete the deployed resources to ensure that the application is not running on the target cluster and accessing data on the PVC:

   $ oc delete <resource_type>

4. To stop a DaemonSet without deleting it, update the nodeSelector in the YAML file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset
     template:
       metadata:
         labels:
           name: hello-daemonset
       spec:
         nodeSelector:
           role: worker 1

   1

   Specify a nodeSelector value that does not exist on any node.

5. Update each PV’s reclaim policy so that unnecessary data is removed. During migration, the reclaim policy for bound PVs is Retain, to ensure that data is not lost when an application is removed from the source cluster. You can remove these PVs during rollback.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain 1
     ...
   status:
     ...

   1

   Specify Recycle or Delete.

6. On the source cluster, switch to your migrated project:

   $ oc project <project_name>

7. Obtain the project’s deployed resources:

   $ oc get all

8. Start one or more replicas of each deployed resource:

   $ oc scale --replicas=1 <resource_type>/<resource_name>

9. Update the nodeSelector of a DaemonSet to its original value, if you changed it during the procedure.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8

   The must-gather tool collects the cluster information and stores it in a must-gather.local.<uid> directory.

3. Remove authentication keys and other sensitive information from the must-gather data.

4. Create an archive file containing the contents of the must-gather.local.<uid> directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

   You can attach the compressed file to your customer support case on the Red Hat Customer Portal.

1. Install the Cluster Application Migration Operator on all clusters.

   You can install the Cluster Application Migration Operator in a restricted environment with limited or no internet access. The source and target clusters must have network access to each other and to a mirror registry.

2. Configure the replication repository, an intermediate object storage that the CAM tool uses to migrate data.

   The source and target clusters must have network access to the replication repository during migration. In a restricted environment, you can use an internally hosted S3 storage repository. If you use a proxy server, you must ensure that replication repository is whitelisted.

3. Add the source cluster to the CAM web console.
4. Add the replication repository to the CAM web console.

5. Create a migration plan, with one of the following data migration options:

   • Copy: The CAM tool copies the data from the source cluster to the replication repository, and from the replication repository to the target cluster.

     

   • Move: The CAM tool unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.

     Note

     Although the replication repository does not appear in this diagram, it is required for the actual migration.

     

6. Run the migration plan, with one of the following options:

   • Stage (optional) copies data to the target cluster without stopping the application.

     Staging can be run multiple times so that most of the data is copied to the target before migration. This minimizes the actual migration time and application downtime.

   • Migrate stops the application on the source cluster and recreates its resources on the target cluster. Optionally, you can migrate the workload without stopping the application.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed the CAM tool.

2. Obtain the CAM web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the CAM web console.

   Note

   If you try to access the CAM web console immediately after installing the Cluster Application Migration Operator, the console may not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster’s API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket <bucket_name> \ 1
       --region <bucket_region> 2

   1

   Specify your S3 bucket name.

   2

   Specify your S3 bucket region, for example, us-east-1.

2. Create the IAM user velero:

   $ aws iam create-user --user-name velero

3. Create an EC2 EBS snapshot policy:

   $ cat > velero-ec2-snapshot-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           }
       ]
   }
   EOF

4. Create an AWS S3 access policy for one or for all S3 buckets:

   $ cat > velero-s3-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>/*" 1
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>" 2
               ]
           }
       ]
   }
   EOF

   1 2

   To grant access to a single S3 bucket, specify the bucket name. To grant access to all AWS S3 buckets, specify * instead of a bucket name:

   "Resource": [
       "arn:aws:s3:::*"

5. Attach the EC2 EBS policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-ebs \
     --policy-document file://velero-ec2-snapshot-policy.json

6. Attach the AWS S3 policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-s3 \
     --policy-document file://velero-s3-policy.json

7. Create an access key for velero:

   $ aws iam create-access-key --user-name velero
   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1
           "AccessKeyId": <AWS_ACCESS_KEY_ID> 2
       }
   }

   1 2

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID for adding the AWS repository to the CAM web console.

Procedure

1. Run gsutil init to log in:

   $ gsutil init
   Welcome! This command will take you through the configuration of gcloud.
   
   Your current configuration has been set to: [default]
   
   To continue, you must login. Would you like to login (Y/n)?

2. Set the BUCKET variable:

   $ BUCKET=<bucket_name> 1

   1

   Specify your bucket name.

3. Create a storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a velero service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero Storage"

6. Set the SERVICE_ACCOUNT_EMAIL variable to the service account’s email address:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
     --filter="displayName:Velero Storage" \
     --format 'value(email)')

7. Grant permissions to the service account:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )
   
   gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
   
   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
       --role projects/$PROJECT_ID/roles/velero.server
   
   gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

8. Save the service account’s keys to the credentials-velero file in the current directory:

   $ gcloud iam service-accounts keys create credentials-velero \
     --iam-account $SERVICE_ACCOUNT_EMAIL

Procedure

1. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

2. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1

   1

   Specify your location.

3. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID=velerobackups

4. Create an Azure storage account:

   $ az storage account create \
     --name $AZURE_STORAGE_ACCOUNT_ID \
     --resource-group $AZURE_RESOURCE_GROUP \
     --sku Standard_GRS \
     --encryption-services blob \
     --https-only true \
     --kind BlobStorage \
     --access-tier Hot

5. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

6. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

7. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
   $ AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
   $ AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv`
   $ AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

8. Save the service principal’s credentials in the credentials-velero file:

   $ cat << EOF  > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

Procedure

1. Log in to the cluster.

2. Obtain the service account token:

   $ oc sa get-token mig -n openshift-migration
   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. Log in to the CAM web console.
4. In the Clusters section, click Add cluster.

5. Fill in the following fields:

   • Cluster name: May contain lower-case letters (a-z) and numbers (0-9). Must not contain spaces or international characters.
   • Url: URL of the cluster’s API server, for example, https://<master1.example.com>:8443.
   • Service account token: String that you obtained from the source cluster.
   • Azure cluster: Optional. Select it if you are using Azure snapshots to copy your data.
   • Azure resource group: This field appears if Azure cluster is checked.
   • If you use a custom CA bundle, click Browse and browse to the CA bundle file.

6. Click Add cluster.

   The cluster appears in the Clusters section.

Procedure

1. Log in to the CAM web console.
2. In the Replication repositories section, click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for AWS S3, MCG, and generic S3 providers:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • S3 bucket name: Specify the name of the S3 bucket you created.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for other S3 providers.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG.
     • Require SSL verification: Clear this check box if you are using a generic S3 provider.
     • If you use a custom CA bundle, click Browse and browse to the Base64-encoded CA bundle file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the CAM web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories section.

Procedure

1. Edit the Migration controller CR:

   $ oc get migrationcontroller -n openshift-migration
   NAME AGE
   migration-controller 5d19h
   
   $ oc edit migrationcontroller -n openshift-migration

2. Update the following parameters:

   ...
   migration_controller: true
   
   # This configuration is loaded into mig-controller, and should be set on the
   # cluster where `migration_controller: true`
   mig_pv_limit: 100
   mig_pod_limit: 100
   mig_namespace_limit: 10
   ...

Procedure

1. Log in to the CAM web console.
2. In the Plans section, click Add plan.

3. Enter the Plan name and click Next.

   The Plan name can contain up to 253 lower-case alphanumeric characters (a-z, 0-9). It must not contain spaces or underscores (_).

4. Select a Source cluster.
5. Select a Target cluster.
6. Select a Replication repository.
7. Select the projects to be migrated and click Next.

8. Select Copy or Move for the PVs:

   • Copy copies the data in a source cluster’s PV to the replication repository and then restores it on a newly created PV, with similar characteristics, in the target cluster.

     Optional: You can verify data copied with the filesystem method by selecting Verify copy. This option, which generates a checksum for each source file and checks it after restoration, significantly reduces performance.

   • Move unmounts a remote volume (for example, NFS) from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.
9. Click Next.

10. Select a Copy method for the PVs:

    • Snapshot backs up and restores the disk using the cloud provider’s snapshot functionality. It is significantly faster than Filesystem.

      Note

      The storage and clusters must be in the same region and the storage class must be compatible.

    • Filesystem copies the data files from the source disk to a newly created target disk.

11. Select a Storage class for the PVs.

    If you selected the Filesystem copy method, you can change the storage class during migration, for example, from Red Hat Gluster Storage or NFS storage to Red Hat Ceph Storage.

12. Click Next.

13. If you want to add a migration hook, click Add Hook and perform the following steps:

    1. Specify the name of the hook.
    2. Select Ansible playbook to use your own playbook or Custom container image for a hook written in another language.
    3. Click Browse to upload the playbook.
    4. Optional: If you are not using the default Ansible runtime image, specify your custom Ansible image.
    5. Specify the cluster on which you want the hook to run.
    6. Specify the service account name.
    7. Specify the namespace.

    8. Select the migration step at which you want the hook to run:

       • PreBackup: Before backup tasks are started on the source cluster
       • PostBackup: After backup tasks are complete on the source cluster
       • PreRestore: Before restore tasks are started on the target cluster
       • PostRestore: After restore tasks are complete on the target cluster

14. Click Add.

    You can add up to four hooks to a migration plan, assigning each hook to a different migration step.

15. Click Finish.

16. Click Close.

    The migration plan appears in the Plans section.

Procedure

1. Log in to the CAM web console on the target cluster.
2. Select a migration plan.

3. Click Stage to copy data from the source cluster to the target cluster without stopping the application.

   You can run Stage multiple times to reduce the actual migration time.

4. When you are ready to migrate the application workload, click Migrate.

   Migrate stops the application workload on the source cluster and recreates its resources on the target cluster.

5. Optional: In the Migrate window, you can select Do not stop applications on the source cluster during migration.
6. Click Migrate.
7. Optional: To stop a migration in progress, click the Options menu and select Cancel.

8. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the Pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. Get the CR name:

   $ oc get <migration_cr> -n openshift-migration 1

   1

   Specify the migration CR, for example, migmigration.

   The output is similar to the following:

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. View the CR:

   $ oc describe <migration_cr> <88435fe0-c9f8-11e9-85e6-5d593ce65e10> -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Log in to the CAM console.
2. Click Plans to view the list of migration plans.
3. Click the Options menu of a specific migration plan and select Logs.
4. Click Download Logs to download the logs of the Migration controller, Velero, and Restic for all clusters.

5. To download a specific log:

   1. Specify the log options:

      • Cluster: Select the source, target, or CAM host cluster.
      • Log source: Select Velero, Restic, or Controller.

      • Pod source: Select the Pod name, for example, controller-manager-78c469849c-v6wcf

        The selected log is displayed.

        You can clear the log selection settings by changing your selection.

   2. Click Download Selected to download the selected log.

Procedure

1. On the target cluster, switch to the migrated project:

   $ oc project <project>

2. Get the deployed resources:

   $ oc get all

3. Delete the deployed resources to ensure that the application is not running on the target cluster and accessing data on the PVC:

   $ oc delete <resource_type>

4. To stop a DaemonSet without deleting it, update the nodeSelector in the YAML file:

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: hello-daemonset
   spec:
     selector:
         matchLabels:
           name: hello-daemonset
     template:
       metadata:
         labels:
           name: hello-daemonset
       spec:
         nodeSelector:
           role: worker 1

   1

   Specify a nodeSelector value that does not exist on any node.

5. Update each PV’s reclaim policy so that unnecessary data is removed. During migration, the reclaim policy for bound PVs is Retain, to ensure that data is not lost when an application is removed from the source cluster. You can remove these PVs during rollback.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain 1
     ...
   status:
     ...

   1

   Specify Recycle or Delete.

6. On the source cluster, switch to your migrated project:

   $ oc project <project_name>

7. Obtain the project’s deployed resources:

   $ oc get all

8. Start one or more replicas of each deployed resource:

   $ oc scale --replicas=1 <resource_type>/<resource_name>

9. Update the nodeSelector of a DaemonSet to its original value, if you changed it during the procedure.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8

   The must-gather tool collects the cluster information and stores it in a must-gather.local.<uid> directory.

3. Remove authentication keys and other sensitive information from the must-gather data.

4. Create an archive file containing the contents of the must-gather.local.<uid> directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

   You can attach the compressed file to your customer support case on the Red Hat Customer Portal.