Chapter 4. Application backup and restore

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators Installed Operators to verify the installation.

Procedure

1. Set the BUCKET variable:

   $ BUCKET=<your_bucket>

2. Set the REGION variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION 1

   1

   us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero 1

   1

   If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a velero-policy.json file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the velero user the minimum necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the velero user:

   $ aws iam create-access-key --user-name velero

   Example output

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

8. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - openshift 1
           - aws
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - name: default
         velero:
           provider: aws
           default: true
           objectStorage:
             bucket: <bucket_name> 5
             prefix: <prefix> 6
           config:
             region: <region>
             profile: "default"
           credential:
             key: cloud
             name: cloud-credentials 7
     snapshotLocations: 8
       - name: default
         velero:
           provider: aws
           config:
             region: <region> 9
             profile: "default"

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   6

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   7

   Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators Installed Operators to verify the installation.

Procedure

1. Log in to Azure:

   $ az login

2. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

3. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location CentralUS 1

   1

   Specify your location.

4. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

5. Create an Azure storage account:

   $ az storage account create \
       --name $AZURE_STORAGE_ACCOUNT_ID \
       --resource-group $AZURE_RESOURCE_GROUP \
       --sku Standard_GRS \
       --encryption-services blob \
       --https-only true \
       --kind BlobStorage \
       --access-tier Hot

6. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

7. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

8. Obtain the storage account access key:

   $ AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list \
     --account-name $AZURE_STORAGE_ACCOUNT_ID \
     --query "[?keyName == 'key1'].value" -o tsv`

9. Create a custom role that has the minimum required permissions:

   AZURE_ROLE=Velero
   az role definition create --role-definition '{
      "Name": "'$AZURE_ROLE'",
      "Description": "Velero related permissions to perform backups, restores and deletions",
      "Actions": [
          "Microsoft.Compute/disks/read",
          "Microsoft.Compute/disks/write",
          "Microsoft.Compute/disks/endGetAccess/action",
          "Microsoft.Compute/disks/beginGetAccess/action",
          "Microsoft.Compute/snapshots/read",
          "Microsoft.Compute/snapshots/write",
          "Microsoft.Compute/snapshots/delete",
          "Microsoft.Storage/storageAccounts/listkeys/action",
          "Microsoft.Storage/storageAccounts/regeneratekey/action"
      ],
      "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"]
      }'

10. Create a credentials-velero file:

    $ cat << EOF > ./credentials-velero
    AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
    AZURE_TENANT_ID=${AZURE_TENANT_ID}
    AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
    AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
    AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
    AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} 1
    AZURE_CLOUD_NAME=AzurePublicCloud
    EOF

    1

    Mandatory. You cannot back up internal images if the credentials-velero file contains only the service principal credentials.

    You use the credentials-velero file to create a Secret object for Azure before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - azure
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             resourceGroup: <azure_resource_group> 5
             storageAccount: <azure_storage_account_id> 6
             subscriptionId: <azure_subscription_id> 7
             storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
           credential:
             key: cloud
             name: cloud-credentials-azure  8
           provider: azure
           default: true
           objectStorage:
             bucket: <bucket_name> 9
             prefix: <prefix> 10
     snapshotLocations: 11
       - velero:
           config:
             resourceGroup: <azure_resource_group>
             subscriptionId: <azure_subscription_id>
             incremental: "true"
           name: default
           provider: azure

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the Azure resource group.

   6

   Specify the Azure storage account ID.

   7

   Specify the Azure subscription ID.

   8

   If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.

   9

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   10

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   11

   You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators Installed Operators to verify the installation.

Procedure

1. Log in to GCP:

   $ gcloud auth login

2. Set the BUCKET variable:

   $ BUCKET=<bucket> 1

   1

   Specify your bucket name.

3. Create the storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero service account"

6. List your service accounts:

   $ gcloud iam service-accounts list

7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:Velero service account" \
       --format 'value(email)')

8. Attach the policies to give the velero user the minimum necessary permissions:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
       storage.objects.create
       storage.objects.delete
       storage.objects.get
       storage.objects.list
       iam.serviceAccounts.signBlob
   )

9. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server

11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - gcp
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           provider: gcp
           default: true
           credential:
             key: cloud
             name: cloud-credentials-gcp 5
           objectStorage:
             bucket: <bucket_name> 6
             prefix: <prefix> 7
     snapshotLocations: 8
       - velero:
           provider: gcp
           default: true
           config:
             project: <project>
             snapshotLocation: us-west1 9

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.

   6

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   7

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators Installed Operators to verify the installation.

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

2. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - aws
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: minio
             s3Url: <url> 5
             insecureSkipTLSVerify: "true"
             s3ForcePathStyle: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name: cloud-credentials 6
           objectStorage:
             bucket: <bucket_name> 7
             prefix: <prefix> 8

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the URL of the S3 endpoint.

   6

   If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   7

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   8

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators Installed Operators to verify the installation.

Procedure

1. Click Operators Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - kubevirt 1
           - gcp 2
           - csi 3
           - openshift 4
         resourceTimeout: 10m 5
       restic:
         enable: true 6
         podConfig:
           nodeSelector: <node_selector> 7
     backupLocations:
       - velero:
           provider: gcp 8
           default: true
           credential:
             key: cloud
             name: <default_secret> 9
           objectStorage:
             bucket: <bucket_name> 10
             prefix: <prefix> 11

   1

   Optional: The kubevirt plugin is used with OpenShift Virtualization.

   2

   Specify the default plugin for the backup provider, for example, gcp, if appropriate.

   3

   Specify the csi default plugin if you use CSI snapshots to back up PVs. The csi plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location.

   4

   The openshift plugin is mandatory.

   5

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   6

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   7

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   8

   Specify the backup provider.

   9

   Specify the correct default name for the Secret, for example, cloud-credentials-gcp, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify a Secret name, the default name is used.

   10

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   11

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Retrieve the backupStorageLocations CRs by entering the following command:

   $ oc get backupStorageLocations

   Example output

   NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   velero-sample-1   Available   11s              31m

2. Create a Backup CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: <backup>
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     hooks: {}
     includedNamespaces:
     - <namespace> 1
     includedResources: [] 2
     excludedResources: [] 3
     storageLocation: <velero-sample-1> 4
     ttl: 720h0m0s
     labelSelector: 5
       matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>
     orLabelSelectors: 6
     - matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>

   1

   Specify an array of namespaces to back up.

   2

   Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.

   3

   Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.

   4

   Specify the name of the backupStorageLocations CR.

   5

   Map of {key,value} pairs of backup resources that have all of the specified labels.

   6

   Map of {key,value} pairs of backup resources that have one or more of the specified labels.

3. Verify that the status of the Backup CR is Completed:

   $ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'

Procedure

1. Configure a Restic secret by creating a .yaml file as following:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name>
     namespace: openshift-adp
   type: Opaque
   stringData:
     RESTIC_PASSWORD: <secure_restic_password>

   Note

   By default, the Operator looks for a secret named dm-credential. If you are using a different name, you need to specify the name through a Data Protection Application (DPA) CR using dpa.spec.features.dataMover.credentialName.

2. Create a DPA CR similar to the following example. The default plugins include CSI.

   Example Data Protection Application (DPA) CR

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: velero-sample
     namespace: openshift-adp
   spec:
     features:
       dataMover:
         enable: true
         credentialName: <secret_name> 1
     backupLocations:
       - velero:
           config:
             profile: default
             region: us-east-1
           credential:
             key: cloud
             name: cloud-credentials
           default: true
           objectStorage:
             bucket: <bucket_name>
             prefix: <bucket_prefix>
           provider: aws
     configuration:
       restic:
         enable: <true_or_false>
       velero:
         defaultPlugins:
           - openshift
           - aws
           - csi

   1

   Add the Restic secret name from the previous step. If this is not done, the default secret name dm-credential is used.

   The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

   Example VolumeSnapshotBackup CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotBackup
   metadata:
     name: <vsb_name>
     namespace: <namespace_name> 1
   spec:
     volumeSnapshotContent:
       name: <snapcontent_name>
     protectedNamespace: <adp_namespace>
     resticSecretRef:
       name: <restic_secret_name>

   1

   Specify the namespace where the volume snapshot exists.

   Example VolumeSnapshotRestore CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotRestore
   metadata:
     name: <vsr_name>
     namespace: <namespace_name> 1
   spec:
     protectedNamespace: <protected_ns> 2
     resticSecretRef:
       name: <restic_secret_name>
     volumeSnapshotMoverBackupRef:
       sourcePVCData:
         name: <source_pvc_name>
         size: <source_pvc_size>
       resticrepository: <your_restic_repo>
       volumeSnapshotClassName: <vsclass_name>

   1

   Specify the namespace where the volume snapshot exists.

   2

   Specify the namespace where the Operator is installed. The default is openshift-adp.

3. You can back up a volume snapshot by performing the following steps:

   1. Create a backup CR:

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: <backup_name>
        namespace: <protected_ns> 1
      spec:
        includedNamespaces:
        - <app_ns>
        storageLocation: velero-sample-1

      1

      Specify the namespace where the Operator is installed. The default namespace is openshift-adp.

   2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands:

      $ oc get vsb -n <app_ns>

      $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"

      A snapshot is created in the object store was configured in the DPA.

      Note

      If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.

4. You can restore a volume snapshot by performing the following steps:

   1. Delete the application namespace and the volumeSnapshotContent that was created by the Velero CSI plugin.

   2. Create a Restore CR and set restorePVs to true.

      Example Restore CR

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: <restore_name>
        namespace: <protected_ns>
      spec:
        backupName: <previous_backup_name>
        restorePVs: true

   3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command:

      $ oc get vsr -n <app_ns>

      $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"

   4. Check whether your application data and resources have been restored.

      Note

      If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

Procedure

1. Retrieve the backupStorageLocations CRs:

   $ oc get backupStorageLocations

   Example output

   NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   velero-sample-1   Available   11s              31m

2. Create a Schedule CR, as in the following example:

   $ cat << EOF | oc apply -f -
   apiVersion: velero.io/v1
   kind: Schedule
   metadata:
     name: <schedule>
     namespace: openshift-adp
   spec:
     schedule: 0 7 * * * 1
     template:
       hooks: {}
       includedNamespaces:
       - <namespace> 2
       storageLocation: <velero-sample-1> 3
       defaultVolumesToRestic: true 4
       ttl: 720h0m0s
   EOF

   1

   cron expression to schedule the backup, for example, 0 7 * * * to perform a backup every day at 7:00.

   2

   Array of namespaces to back up.

   3

   Name of the backupStorageLocations CR.

   4

   Optional: Add the defaultVolumesToRestic: true key-value pair if you are backing up volumes with Restic.

3. Verify that the status of the Schedule CR is Completed after the scheduled backup runs:

   $ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'

Procedure

1. Create a Restore CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: <restore>
     namespace: openshift-adp
   spec:
     backupName: <backup> 1
     includedResources: [] 2
     excludedResources:
     - nodes
     - events
     - events.events.k8s.io
     - backups.velero.io
     - restores.velero.io
     - resticrepositories.velero.io
     restorePVs: true 3

   1

   Name of the Backup CR.

   2

   Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, po for pods) or fully-qualified. If unspecified, all resources are included.

   3

   Optional: The restorePVs parameter can be set to false in order to turn off restore of PersistentVolumes from VolumeSnapshot of Container Storage Interface (CSI) snapshots, or from native snapshots when VolumeSnaphshotLocation is configured.

2. Verify that the status of the Restore CR is Completed by entering the following command:

   $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

3. Verify that the backup resources have been restored by entering the following command:

   $ oc get all -n <namespace> 1

   1

   Namespace that you backed up.

4. If you use Restic to restore DeploymentConfig objects or if you use post-restore hooks, run the dc-restic-post-restore.sh cleanup script by entering the following command:

   $ bash dc-restic-post-restore.sh <restore-name>

   Note

   In the course of the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow Restic and post-restore hooks to complete their actions on the restored pods. The cleanup script removes these disconnected pods and scale any DeploymentConfig objects back up to the appropriate number of replicas.

   Example 4.1. dc-restic-post-restore.sh cleanup script

   #!/bin/bash
   set -e
   
   # if sha256sum exists, use it to check the integrity of the file
   if command -v sha256sum >/dev/null 2>&1; then
     CHECKSUM_CMD="sha256sum"
   else
     CHECKSUM_CMD="shasum -a 256"
   fi
   
   label_name () {
       if [ "${#1}" -le "63" ]; then
   	echo $1
   	return
       fi
       sha=$(echo -n $1|$CHECKSUM_CMD)
       echo "${1:0:57}${sha:0:6}"
   }
   
   OADP_NAMESPACE=${OADP_NAMESPACE:=openshift-adp}
   
   if [[ $# -ne 1 ]]; then
       echo "usage: ${BASH_SOURCE} restore-name"
       exit 1
   fi
   
   echo using OADP Namespace $OADP_NAMESPACE
   echo restore: $1
   
   label=$(label_name $1)
   echo label: $label
   
   echo Deleting disconnected restore pods
   oc delete pods -l oadp.openshift.io/disconnected-from-dc=$label
   
   for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
   do
       IFS=',' read -ra dc_arr <<< "$dc"
       if [ ${#dc_arr[0]} -gt 0 ]; then
   	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
   	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
   spec:
     replicas: ${dc_arr[2]}
     paused: ${dc_arr[3]}
   EOF
       fi
   done

Solution

1. Edit the value of the spec.snapshotLocations.velero.config.region key in the DataProtectionApplication manifest so that the snapshot location is in the same region as the PV.
2. Create a new Backup CR.

Solution

1. Retrieve the details of the Backup CR:

   $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
     backup describe <backup>

2. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

   You do not need to clean up the backup location because a Backup CR in progress has not uploaded files to object storage.

3. Create a new Backup CR.

Solution

1. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

2. If required, clean up the stored data on the BackupStorageLocation to free up space.

3. Apply label velero.io/csi-volumesnapshot-class=true to the VolumeSnapshotClass object:

   $ oc label volumesnapshotclass/<snapclass_name> velero.io/csi-volumesnapshot-class=true

4. Create a new Backup CR.

1. Create a supplemental group for Restic on the NFS data volume.
2. Set the setgid bit on the NFS directories so that group ownership is inherited.

3. Add the spec.configuration.restic.supplementalGroups parameter and the group ID to the DataProtectionApplication manifest, as in the following example:

   spec:
     configuration:
       restic:
         enable: true
         supplementalGroups:
         - <group_id> 1

   1

   Specify the supplemental group ID.

4. Wait for the Restic pods to restart so that the changes are applied.

1. Preferred version of the destination cluster
2. Preferred version of the source_ cluster
3. Common non-preferred supported version with the highest Kubernetes version priority