Chapter 4. OADP Application backup and restore

Procedure

1. Create an OBC manifest file to request a NooBaa MCG bucket as shown in the following example:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: test-obc
     namespace: openshift-adp
   spec:
     storageClassName: openshift-storage.noobaa.io
     generateBucketName: test-backup-bucket

   where:

   test-obc

   Specifies the name of the object bucket claim.

   test-backup-bucket

   Specifies the name of the bucket.

2. Create the OBC by running the following command:

   $ oc create -f <obc_file_name>

   where:

   <obc_file_name>

   Specifies the file name of the object bucket claim manifest.

3. When you create an OBC, ODF creates a

   secret

   and a

   config map

   with the same name as the object bucket claim. The

   secret

   has the bucket credentials, and the

   config map

   has information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:

   $ oc extract --to=- cm/test-obc

   test-obc

   is the name of the OBC.

   Example output

   # BUCKET_NAME
   backup-c20...41fd
   # BUCKET_PORT
   443
   # BUCKET_REGION
   
   # BUCKET_SUBREGION
   
   # BUCKET_HOST
   s3.openshift-storage.svc

4. To get the bucket credentials from the generated

   secret

   , run the following command:

   $ oc extract --to=- secret/test-obc

   Example output

   # AWS_ACCESS_KEY_ID
   ebYR....xLNMc
   # AWS_SECRET_ACCESS_KEY
   YXf...+NaCkdyC3QPym

5. Get the public URL for the S3 endpoint from the s3 route in the

   openshift-storage

   namespace by running the following command:

   $ oc get route s3 -n openshift-storage

6. Create a

   cloud-credentials

   file with the object bucket credentials as shown in the following command:

   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

7. Create the

   cloud-credentials

   secret with the

   cloud-credentials

   file content as shown in the following command:

   $ oc create secret generic \
     cloud-credentials \
     -n openshift-adp \
     --from-file cloud=cloud-credentials

8. Configure the Data Protection Application (DPA) as shown in the following example:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: oadp-backup
     namespace: openshift-adp
   spec:
     configuration:
       nodeAgent:
         enable: true
         uploaderType: kopia
       velero:
         defaultPlugins:
           - aws
           - openshift
           - csi
         defaultSnapshotMoveData: true
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: noobaa
             s3Url: https://s3.openshift-storage.svc
             s3ForcePathStyle: "true"
             insecureSkipTLSVerify: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name:  cloud-credentials
           objectStorage:
             bucket: <bucket_name>
             prefix: oadp

   where:

   defaultSnapshotMoveData

   Set to true to use the OADP Data Mover to enable movement of Container Storage Interface (CSI) snapshots to a remote object storage.

   s3Url

   Specifies the S3 URL of ODF storage.

   <bucket_name>

   Specifies the bucket name.

9. Create the DPA by running the following command:

   $ oc apply -f <dpa_filename>

10. Verify that the DPA is created successfully by running the following command. In the example output, you can see the

    status

    object has

    type

    field set to

    Reconciled

    . This means, the DPA is successfully created.

    $ oc get dpa -o yaml

    Example output

    apiVersion: v1
    items:
    - apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        namespace: openshift-adp
        #...#
      spec:
        backupLocations:
        - velero:
            config:
              #...#
      status:
        conditions:
        - lastTransitionTime: "20....9:54:02Z"
          message: Reconcile complete
          reason: Complete
          status: "True"
          type: Reconciled
    kind: List
    metadata:
      resourceVersion: ""

11. Verify that the backup storage location (BSL) is available by running the following command:

    $ oc get backupstoragelocations.velero.io -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE   DEFAULT
    dpa-sample-1   Available   3s               15s   true

12. Configure a backup CR as shown in the following example:

    apiVersion: velero.io/v1
    kind: Backup
    metadata:
      name: test-backup
      namespace: openshift-adp
    spec:
      includedNamespaces:
      - <application_namespace>

    where:

    <application_namespace>

    Specifies the namespace for the application to back up.

13. Create the backup CR by running the following command:

    $ oc apply -f <backup_cr_filename>

Procedure

1. Create a restore CR as shown in the following example:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: test-restore
     namespace: openshift-adp
   spec:
     backupName: <backup_name>
     restorePVs: true
     namespaceMapping:
       <application_namespace>: test-restore-application

   where:

   test-restore

   Specifies the name of the restore CR.

   <backup_name>

   Specifies the name of the backup.

   <application_namespace>

   Specifies the target namespace to restore to. namespaceMapping maps the source application namespace to the target application namespace. test-restore-application is the name of target namespace where you want to restore the backup.

2. Apply the restore CR by running the following command:

   $ oc apply -f <restore_cr_filename>

Verification

1. Verify that the restore is in the

   Completed

   phase by running the following command:

   $ oc describe restores.velero.io <restore_name> -n openshift-adp

2. Change to the restored namespace

   test-restore-application

   by running the following command:

   $ oc project test-restore-application

3. Verify the restored resources such as persistent volume claim (pvc), service (svc), deployment, secret, and config map by running the following command:

   $ oc get pvc,svc,deployment,secret,configmap

   Example output

   NAME                          STATUS   VOLUME
   persistentvolumeclaim/mysql   Bound    pvc-9b3583db-...-14b86
   
   NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
   service/mysql      ClusterIP   172....157     <none>        3306/TCP   2m56s
   service/todolist   ClusterIP   172.....15     <none>        8000/TCP   2m56s
   
   NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/mysql   0/1     1            0           2m55s
   
   NAME                                         TYPE                      DATA   AGE
   secret/builder-dockercfg-6bfmd               kubernetes.io/dockercfg   1      2m57s
   secret/default-dockercfg-hz9kz               kubernetes.io/dockercfg   1      2m57s
   secret/deployer-dockercfg-86cvd              kubernetes.io/dockercfg   1      2m57s
   secret/mysql-persistent-sa-dockercfg-rgp9b   kubernetes.io/dockercfg   1      2m57s
   
   NAME                                 DATA   AGE
   configmap/kube-root-ca.crt           1      2m57s
   configmap/openshift-service-ca.crt   1      2m57s

Procedure

1. Create an OBC manifest to request a NooBaa bucket as shown in the following example:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: test-obc
     namespace: openshift-adp
   spec:
     storageClassName: openshift-storage.noobaa.io
     generateBucketName: test-backup-bucket

   where:

   test-obc

   Specifies the name of the object bucket claim.

   test-backup-bucket

   Specifies the name of the bucket.

2. Create the OBC by running the following command:

   $ oc create -f <obc_file_name>

3. When you create an OBC, ODF creates a

   secret

   and a

   ConfigMap

   with the same name as the object bucket claim. The

   secret

   object contains the bucket credentials, and the

   ConfigMap

   object contains information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:

   $ oc extract --to=- cm/test-obc

   test-obc

   is the name of the OBC.

   Example output

   # BUCKET_NAME
   backup-c20...41fd
   # BUCKET_PORT
   443
   # BUCKET_REGION
   
   # BUCKET_SUBREGION
   
   # BUCKET_HOST
   s3.openshift-storage.svc

4. To get the bucket credentials from the

   secret

   object, run the following command:

   $ oc extract --to=- secret/test-obc

   Example output

   # AWS_ACCESS_KEY_ID
   ebYR....xLNMc
   # AWS_SECRET_ACCESS_KEY
   YXf...+NaCkdyC3QPym

5. Create a

   cloud-credentials

   file with the object bucket credentials by using the following example configuration:

   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

6. Create the

   cloud-credentials

   secret with the

   cloud-credentials

   file content by running the following command:

   $ oc create secret generic \
     cloud-credentials \
     -n openshift-adp \
     --from-file cloud=cloud-credentials

7. Extract the service CA certificate from the

   openshift-service-ca.crt

   config map by running the following command. Ensure that you encode the certificate in

   Base64

   format and note the value to use in the next step.

   $ oc get cm/openshift-service-ca.crt \
     -o jsonpath='{.data.service-ca\.crt}' | base64 -w0; echo

   Example output

   LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0...
   ....gpwOHMwaG9CRmk5a3....FLS0tLS0K

8. Configure the

   DataProtectionApplication

   CR manifest file with the bucket name and CA certificate as shown in the following example:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: oadp-backup
     namespace: openshift-adp
   spec:
     configuration:
       nodeAgent:
         enable: true
         uploaderType: kopia
       velero:
         defaultPlugins:
           - aws
           - openshift
           - csi
         defaultSnapshotMoveData: true
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: noobaa
             s3Url: https://s3.openshift-storage.svc
             s3ForcePathStyle: "true"
             insecureSkipTLSVerify: "false"
           provider: aws
           default: true
           credential:
             key: cloud
             name:  cloud-credentials
           objectStorage:
             bucket: <bucket_name>
             prefix: oadp
             caCert: <ca_cert>

   where:

   insecureSkipTLSVerify

   Specifies whether SSL/TLS security is enabled. If set to true, SSL/TLS security is disabled. If set to false, SSL/TLS security is enabled.

   <bucket_name>

   Specifies the name of the bucket extracted in an earlier step.

   <ca_cert>

   Specifies the Base64 encoded certificate from the previous step.

9. Create the

   DataProtectionApplication

   CR by running the following command:

   $ oc apply -f <dpa_filename>

10. Verify that the

    DataProtectionApplication

    CR is created successfully by running the following command:

    $ oc get dpa -o yaml

    Example output

    apiVersion: v1
    items:
    - apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        namespace: openshift-adp
        #...#
      spec:
        backupLocations:
        - velero:
            config:
              #...#
      status:
        conditions:
        - lastTransitionTime: "20....9:54:02Z"
          message: Reconcile complete
          reason: Complete
          status: "True"
          type: Reconciled
    kind: List
    metadata:
      resourceVersion: ""

11. Verify that the backup storage location (BSL) is available by running the following command:

    $ oc get backupstoragelocations.velero.io -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE   DEFAULT
    dpa-sample-1   Available   3s               15s   true

12. Configure the

    Backup

    CR by using the following example:

    apiVersion: velero.io/v1
    kind: Backup
    metadata:
      name: test-backup
      namespace: openshift-adp
    spec:
      includedNamespaces:
      - <application_namespace>

    where:

    <application_namespace>

    Specifies the namespace for the application to back up.

13. Create the

    Backup

    CR by running the following command:

    $ oc apply -f <backup_cr_filename>

Procedure

1. Configure the

   DataProtectionApplication

   CR to use the

   legacy-aws

   Velero plugin as shown in the following example:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: oadp-backup
     namespace: openshift-adp
   spec:
     configuration:
       nodeAgent:
         enable: true
         uploaderType: kopia
       velero:
         defaultPlugins:
           - legacy-aws
           - openshift
           - csi
         defaultSnapshotMoveData: true
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: noobaa
             s3Url: https://s3.openshift-storage.svc
             s3ForcePathStyle: "true"
             insecureSkipTLSVerify: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name:  cloud-credentials
           objectStorage:
             bucket: <bucket_name>
             prefix: oadp

   where:

   legacy-aws

   Specifies to use the legacy-aws plugin.

   <bucket_name>

   Specifies the bucket name.

2. Create the

   DataProtectionApplication

   CR by running the following command:

   $ oc apply -f <dpa_filename>

3. Verify that the

   DataProtectionApplication

   CR is created successfully by running the following command. In the example output, you can see the

   status

   object has the

   type

   field set to

   Reconciled

   and the

   status

   field set to

   "True"

   . That status indicates that the

   DataProtectionApplication

   CR is successfully created.

   $ oc get dpa -o yaml

   Example output

   apiVersion: v1
   items:
   - apiVersion: oadp.openshift.io/v1alpha1
     kind: DataProtectionApplication
     metadata:
       namespace: openshift-adp
       #...#
     spec:
       backupLocations:
       - velero:
           config:
             #...#
     status:
       conditions:
       - lastTransitionTime: "20....9:54:02Z"
         message: Reconcile complete
         reason: Complete
         status: "True"
         type: Reconciled
   kind: List
   metadata:
     resourceVersion: ""

4. Verify that the backup storage location (BSL) is available by running the following command:

   $ oc get backupstoragelocations.velero.io -n openshift-adp

   You should see an output similar to the following example:

   NAME           PHASE       LAST VALIDATED   AGE   DEFAULT
   dpa-sample-1   Available   3s               15s   true

5. Configure a

   Backup

   CR as shown in the following example:

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: test-backup
     namespace: openshift-adp
   spec:
     includedNamespaces:
     - <application_namespace>

   where:

   <application_namespace>

   Specifies the namespace for the application to back up.

6. Create the

   Backup

   CR by running the following command:

   $ oc apply -f <backup_cr_filename>

Procedure

1. Set the

   BUCKET

   variable:

   $ BUCKET=<your_bucket>

2. Set the

   REGION

   variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION

   where:

   LocationConstraint

   Specifies the bucket configuration location constraint. us-east-1 does not support LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero

   where:

   velero

   Specifies the user name. If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a

   velero-policy.json

   file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the

   velero

   user the minimum necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the

   velero

   user:

   $ aws iam create-access-key --user-name velero

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

8. Create a

   credentials-velero

   file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the

   credentials-velero

   file to create a

   Secret

   object for AWS before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the

   DataProtectionApplication

   manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - openshift
           - aws
         resourceTimeout: 10m
       nodeAgent:
         enable: true
         uploaderType: kopia
         podConfig:
           nodeSelector: <node_selector>
     backupLocations:
       - name: default
         velero:
           provider: aws
           default: true
           objectStorage:
             bucket: <bucket_name>
             prefix: <prefix>
           config:
             region: <region>
             profile: "default"
             s3ForcePathStyle: "true"
             s3Url: <s3_url>
           credential:
             key: cloud
             name: cloud-credentials
     snapshotLocations:
       - name: default
         velero:
           provider: aws
           config:
             region: <region>
             profile: "default"
           credential:
             key: cloud
             name: cloud-credentials

   where:

   namespace

   Specifies the default namespace for OADP which is openshift-adp. The namespace is a variable and is configurable.

   openshift

   Specifies that the openshift plugin is mandatory.

   resourceTimeout

   Specifies how many minutes to wait for several Velero resources such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability, before timeout occurs. The default is 10m.

   nodeAgent

   Specifies the administrative agent that routes the administrative requests to servers.

   enable

   Set this value to true if you want to enable nodeAgent and perform File System Backup.

   uploaderType

   Specifies the uploader type. Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.

   nodeSelector

   Specifies the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.

   bucket

   Specifies a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   prefix

   Specifies a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   s3ForcePathStyle

   Specifies whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.

   s3Url

   Specifies the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.

   name

   Specifies the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   snapshotLocations

   Specifies a snapshot location, unless you use CSI snapshots or a File System Backup (FSB) to back up PVs.

   region

   Specifies that the snapshot location must be in the same region as the PVs.

   name

   Specifies the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in the credentials-velero file.

4. Click Create.

Verification

1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

   $ oc get all -n openshift-adp

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/node-agent-9cq4q                                     1/1     Running   0          94s
   pod/node-agent-m4lts                                     1/1     Running   0          94s
   pod/node-agent-pv4kr                                     1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
   
   NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

2. Verify that the

   DataProtectionApplication

   (DPA) is reconciled by running the following command:

   $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

   {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

3. Verify the

   type

   is set to

   Reconciled

   .

4. Verify the backup storage location and confirm that the

   PHASE

   is

   Available

   by running the following command:

   $ oc get backupstoragelocations.velero.io -n openshift-adp

   NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
   dpa-sample-1   Available   1s               3d16h   true

5. Verify that the

   PHASE

   is in

   Available

   .