Chapter 4. OADP Application backup and restore

Procedure

1. Set the BUCKET variable:

   $ BUCKET=<your_bucket>

2. Set the REGION variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION 1

   1

   us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero 1

   1

   If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a velero-policy.json file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the velero user the minimum necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the velero user:

   $ aws iam create-access-key --user-name velero

   Example output

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

8. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - openshift 1
           - aws
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - name: default
         velero:
           provider: aws
           default: true
           objectStorage:
             bucket: <bucket_name> 5
             prefix: <prefix> 6
           config:
             region: <region>
             profile: "default"
           credential:
             key: cloud
             name: cloud-credentials 7
     snapshotLocations: 8
       - velero:
           provider: aws
           config:
             region: <region> 9
             profile: "default"

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set this value to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding spec.defaultVolumesToFsBackup: true to the Backup CR. In OADP version 1.1, add spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   6

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   7

   Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - azure
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             resourceGroup: <azure_resource_group> 5
             storageAccount: <azure_storage_account_id> 6
             subscriptionId: <azure_subscription_id> 7
             storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
           credential:
             key: cloud
             name: cloud-credentials-azure  8
           provider: azure
           default: true
           objectStorage:
             bucket: <bucket_name> 9
             prefix: <prefix> 10
     snapshotLocations: 11
       - velero:
           config:
             resourceGroup: <azure_resource_group>
             subscriptionId: <azure_subscription_id>
             incremental: "true"
           name: default
           provider: azure

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set this value to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding spec.defaultVolumesToFsBackup: true to the Backup CR. In OADP version 1.1, add spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the Azure resource group.

   6

   Specify the Azure storage account ID.

   7

   Specify the Azure subscription ID.

   8

   If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.

   9

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   10

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   11

   You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Log in to GCP:

   $ gcloud auth login

2. Set the BUCKET variable:

   $ BUCKET=<bucket> 1

   1

   Specify your bucket name.

3. Create the storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero service account"

6. List your service accounts:

   $ gcloud iam service-accounts list

7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:Velero service account" \
       --format 'value(email)')

8. Attach the policies to give the velero user the minimum necessary permissions:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
       storage.objects.create
       storage.objects.delete
       storage.objects.get
       storage.objects.list
       iam.serviceAccounts.signBlob
   )

9. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server

11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - gcp
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           provider: gcp
           default: true
           credential:
             key: cloud
             name: cloud-credentials-gcp 5
           objectStorage:
             bucket: <bucket_name> 6
             prefix: <prefix> 7
     snapshotLocations: 8
       - velero:
           provider: gcp
           default: true
           config:
             project: <project>
             snapshotLocation: us-west1 9

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set this value to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding spec.defaultVolumesToFsBackup: true to the Backup CR. In OADP version 1.1, add spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.

   6

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   7

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

2. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - aws
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: minio
             s3Url: <url> 5
             insecureSkipTLSVerify: "true"
             s3ForcePathStyle: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name: cloud-credentials 6
           objectStorage:
             bucket: <bucket_name> 7
             prefix: <prefix> 8

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set this value to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding spec.defaultVolumesToFsBackup: true to the Backup CR. In OADP version 1.1, add spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the URL of the S3 endpoint.

   6

   If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   7

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   8

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - aws
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: minio
             s3Url: <url> 5
             insecureSkipTLSVerify: "true"
             s3ForcePathStyle: "true"
           provider: gcp
           default: true
           credential:
             key: cloud
             name: cloud-credentials 6
           objectStorage:
             bucket: <bucket_name> 7
             prefix: <prefix> 8

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set this value to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding spec.defaultVolumesToFsBackup: true to the Backup CR. In OADP version 1.1, add spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the URL of the S3 endpoint.

   6

   If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   7

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   8

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Create a Restore CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: <restore>
     namespace: openshift-adp
   spec:
     backupName: <backup> 1
     includedResources: [] 2
     excludedResources:
     - nodes
     - events
     - events.events.k8s.io
     - backups.velero.io
     - restores.velero.io
     - resticrepositories.velero.io
     restorePVs: true 3

   1

   Name of the Backup CR.

   2

   Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, po for pods) or fully-qualified. If unspecified, all resources are included.

   3

   Optional: The restorePVs parameter can be set to false to turn off restore of PersistentVolumes from VolumeSnapshot of Container Storage Interface (CSI) snapshots or from native snapshots when VolumeSnapshotLocation is configured.

2. Verify that the status of the Restore CR is Completed by entering the following command:

   $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

3. Verify that the backup resources have been restored by entering the following command:

   $ oc get all -n <namespace> 1

   1

   Namespace that you backed up.

4. If you use Restic to restore DeploymentConfig objects or if you use post-restore hooks, run the dc-restic-post-restore.sh cleanup script by entering the following command:

   $ bash dc-restic-post-restore.sh <restore-name>

   Note

   During the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow Restic and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales any DeploymentConfig objects back up to the appropriate number of replicas.

   Example 4.1. dc-restic-post-restore.sh cleanup script

   #!/bin/bash
   set -e
   
   # if sha256sum exists, use it to check the integrity of the file
   if command -v sha256sum >/dev/null 2>&1; then
     CHECKSUM_CMD="sha256sum"
   else
     CHECKSUM_CMD="shasum -a 256"
   fi
   
   label_name () {
       if [ "${#1}" -le "63" ]; then
   	echo $1
   	return
       fi
       sha=$(echo -n $1|$CHECKSUM_CMD)
       echo "${1:0:57}${sha:0:6}"
   }
   
   OADP_NAMESPACE=${OADP_NAMESPACE:=openshift-adp}
   
   if [[ $# -ne 1 ]]; then
       echo "usage: ${BASH_SOURCE} restore-name"
       exit 1
   fi
   
   echo using OADP Namespace $OADP_NAMESPACE
   echo restore: $1
   
   label=$(label_name $1)
   echo label: $label
   
   echo Deleting disconnected restore pods
   oc delete pods -l oadp.openshift.io/disconnected-from-dc=$label
   
   for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
   do
       IFS=',' read -ra dc_arr <<< "$dc"
       if [ ${#dc_arr[0]} -gt 0 ]; then
   	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
   	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
   spec:
     replicas: ${dc_arr[2]}
     paused: ${dc_arr[3]}
   EOF
       fi
   done

Procedure

1. Create the following environment variables by running the following commands:

   Important

   Change the cluster name to match your ROSA cluster, and ensure you are logged into the cluster as an administrator. Ensure that all fields are outputted correctly before continuing.

   $ export CLUSTER_NAME=my-cluster 1
     export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id)
     export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)
     export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')
     export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
     export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.')
     export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"
     export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"
     mkdir -p ${SCRATCH}
     echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint:
     ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

   1

   Replace my-cluster with your ROSA cluster name.

2. On the AWS account, create an IAM policy to allow access to AWS S3:

   1. Check to see if the policy exists by running the following command:

      $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text) 1

      1

      Replace RosaOadp with your policy name.

   2. Enter the following command to create the policy JSON file and then create the policy in ROSA:

      Note

      If the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the if statement intentionally skips the policy creation.

      $ if [[ -z "${POLICY_ARN}" ]]; then
        cat << EOF > ${SCRATCH}/policy.json 1
        {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "s3:CreateBucket",
              "s3:DeleteBucket",
              "s3:PutBucketTagging",
              "s3:GetBucketTagging",
              "s3:PutEncryptionConfiguration",
              "s3:GetEncryptionConfiguration",
              "s3:PutLifecycleConfiguration",
              "s3:GetLifecycleConfiguration",
              "s3:GetBucketLocation",
              "s3:ListBucket",
              "s3:GetObject",
              "s3:PutObject",
              "s3:DeleteObject",
              "s3:ListBucketMultipartUploads",
              "s3:AbortMultipartUploads",
              "s3:ListMultipartUploadParts",
              "s3:DescribeSnapshots",
              "ec2:DescribeVolumes",
              "ec2:DescribeVolumeAttribute",
              "ec2:DescribeVolumesModifications",
              "ec2:DescribeVolumeStatus",
              "ec2:CreateTags",
              "ec2:CreateVolume",
              "ec2:CreateSnapshot",
              "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
          }
         ]}
      EOF
      
        POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \
        --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \
        --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \
        --output text)
        fi

      1

      SCRATCH is a name for a temporary directory created for the environment variables.

   3. View the policy ARN by running the following command:

      $ echo ${POLICY_ARN}

3. Create an IAM role trust policy for the cluster:

   1. Create the trust policy file by running the following command:

      $ cat <<EOF > ${SCRATCH}/trust-policy.json
        {
            "Version":2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "${OIDC_ENDPOINT}:sub": [
                    "system:serviceaccount:openshift-adp:openshift-adp-controller-manager",
                    "system:serviceaccount:openshift-adp:velero"]
                }
              }
            }]
        }
      EOF

   2. Create the role by running the following command:

      $ ROLE_ARN=$(aws iam create-role --role-name \
        "${ROLE_NAME}" \
        --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
      --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID} Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=openshift-oadp \
         --query Role.Arn --output text)

   3. View the role ARN by running the following command:

      $ echo ${ROLE_ARN}

4. Attach the IAM policy to the IAM role by running the following command:

   $ aws iam attach-role-policy --role-name "${ROLE_NAME}" \
     --policy-arn ${POLICY_ARN}

Procedure

1. Create an OpenShift Container Platform secret from your AWS token file by entering the following commands:

   1. Create the credentials file:

      $ cat <<EOF > ${SCRATCH}/credentials
        [default]
        role_arn = ${ROLE_ARN}
        web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
      EOF

   2. Create a namespace for OADP:

      $ oc create namespace openshift-adp

   3. Create the OpenShift Container Platform secret:

      $ oc -n openshift-adp create secret generic cloud-credentials \
        --from-file=${SCRATCH}/credentials

      Note

      In OpenShift Container Platform versions 4.14 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the OpenShift Container Platform web console, for more information see Installing from OperatorHub using the web console.

      The preceding secret is created automatically by CCO.

2. Install the OADP Operator:

   1. In the OpenShift Container Platform web console, browse to Operators OperatorHub.
   2. Search for the OADP Operator.
   3. In the role_ARN field, paste the role_arn that you created previously and click Install.

3. Create AWS cloud storage using your AWS credentials by entering the following command:

   $ cat << EOF | oc create -f -
     apiVersion: oadp.openshift.io/v1alpha1
     kind: CloudStorage
     metadata:
       name: ${CLUSTER_NAME}-oadp
       namespace: openshift-adp
     spec:
       creationSecret:
         key: credentials
         name: cloud-credentials
       enableSharedConfig: true
       name: ${CLUSTER_NAME}-oadp
       provider: aws
       region: $REGION
   EOF

4. Check your application’s storage default storage class by entering the following command:

   $ oc get pvc -n <namespace>

   Example output

   NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   applog   Bound    pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8   1Gi        RWO            gp3-csi        4d19h
   mysql    Bound    pvc-16b8e009-a20a-4379-accc-bc81fedd0621   1Gi        RWO            gp3-csi        4d19h

5. Get the storage class by running the following command:

   $ oc get storageclass

   Example output

   NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   4d21h
   gp2-csi             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3                 ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3-csi (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h

   Note

   The following storage classes will work:

   • gp3-csi
   • gp2-csi
   • gp3
   • gp2

   If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.

6. Create the DataProtectionApplication resource to configure the connection to the storage where the backups and volume snapshots are stored:

   1. If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:

      $ cat << EOF | oc create -f -
        apiVersion: oadp.openshift.io/v1alpha1
        kind: DataProtectionApplication
        metadata:
          name: ${CLUSTER_NAME}-dpa
          namespace: openshift-adp
        spec:
          backupImages: true 1
          features:
            dataMover:
              enable: false
          backupLocations:
          - bucket:
              cloudStorageRef:
                name: ${CLUSTER_NAME}-oadp
              credential:
                key: credentials
                name: cloud-credentials
              prefix: velero
              default: true
              config:
                region: ${REGION}
          configuration:
            velero:
              defaultPlugins:
              - openshift
              - aws
              - csi
            restic:
              enable: false
      EOF

      1

      ROSA supports internal image backup. Set this field to false if you do not want to use image backup.

1. If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:

   $ cat << EOF | oc create -f -
     apiVersion: oadp.openshift.io/v1alpha1
     kind: DataProtectionApplication
     metadata:
       name: ${CLUSTER_NAME}-dpa
       namespace: openshift-adp
     spec:
       backupImages: true 1
       features:
         dataMover:
            enable: false
       backupLocations:
       - bucket:
           cloudStorageRef:
             name: ${CLUSTER_NAME}-oadp
           credential:
             key: credentials
             name: cloud-credentials
           prefix: velero
           default: true
           config:
             region: ${REGION}
       configuration:
         velero:
           defaultPlugins:
           - openshift
           - aws
         nodeAgent: 2
           enable: false
           uploaderType: restic
       snapshotLocations:
         - velero:
             config:
               credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials 3
               enableSharedConfig: "true" 4
               profile: default 5
               region: ${REGION} 6
             provider: aws
   EOF

   1

   ROSA supports internal image backup. Set this field to false if you do not want to use image backup.

   2

   See the following note.

   3

   The credentialsFile field is the mounted location of the bucket credential on the pod.

   4

   The enableSharedConfig field allows the snapshotLocations to share or reuse the credential defined for the bucket.

   5

   Use the profile name set in the AWS credentials file.

   6

   Specify region as your AWS region. This must be the same as the cluster region.

   You are now ready to back up and restore OpenShift Container Platform applications, as described in Backing up applications.

Procedure

1. Create a YAML file for the Backup object, as in the following example:

   Example Backup CR

   kind: Backup
   apiVersion: velero.io/v1
   metadata:
     name: backup
     namespace: openshift-adp
   spec:
     csiSnapshotTimeout: 10m0s
     defaultVolumesToFsBackup: false
     includedNamespaces:
     - mysql-persistent
     itemOperationTimeout: 4h0m0s
     snapshotMoveData: true 1
     storageLocation: default
     ttl: 720h0m0s
     volumeSnapshotLocations:
     - dpa-sample-1
   # ...

   1

   Set to true to enable movement of CSI snapshots to remote object storage.

   Note

   If you format the volume by using XFS filesystem and the volume is at 100% capacity, the backup fails with a no space left on device error. For example:

   Error: relabel failed /var/lib/kubelet/pods/3ac..34/volumes/ \
   kubernetes.io~csi/pvc-684..12c/mount: lsetxattr /var/lib/kubelet/ \
   pods/3ac..34/volumes/kubernetes.io~csi/pvc-68..2c/mount/data-xfs-103: \
   no space left on device

   In this scenario, consider resizing the volume or using a different filesystem type, for example, ext4, so that the backup completes successfully.

2. Apply the manifest:

   $ oc create -f backup.yaml

   A DataUpload CR is created after the snapshot creation is complete.

Procedure

1. Create a YAML file for the Restore CR, as in the following example:

   Example Restore CR

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: restore
     namespace: openshift-adp
   spec:
     backupName: <backup>
   # ...

2. Apply the manifest:

   $ oc create -f restore.yaml

   A DataDownload CR is created when the restore starts.