Backup and restore


OpenShift Container Platform 4.17

Backing up and restoring your OpenShift Container Platform cluster

Red Hat OpenShift Documentation Team

Abstract

This document provides instructions for backing up your cluster's data and for recovering from various disaster scenarios.

Chapter 1. Backup and restore

1.1. Control plane backup and restore operations

As a cluster administrator, you might need to stop an OpenShift Container Platform cluster for a period and restart it later. Some reasons for restarting a cluster are that you need to perform maintenance on a cluster or want to reduce resource costs. In OpenShift Container Platform, you can perform a graceful shutdown of a cluster so that you can easily restart the cluster later.

You must back up etcd data before shutting down a cluster; etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. An etcd backup plays a crucial role in disaster recovery. In OpenShift Container Platform, you can also replace an unhealthy etcd member.

When you want to get your cluster running again, restart the cluster gracefully.

Note

A cluster’s certificates expire one year after the installation date. You can shut down a cluster and expect it to restart gracefully while the certificates are still valid. Although the cluster automatically retrieves the expired control plane certificates, you must still approve the certificate signing requests (CSRs).

You might run into several situations where OpenShift Container Platform does not work as expected, such as:

  • You have a cluster that is not functional after the restart because of unexpected conditions, such as node failure or network connectivity issues.
  • You have deleted something critical in the cluster by mistake.
  • You have lost the majority of your control plane hosts, leading to etcd quorum loss.

You can always recover from a disaster situation by restoring your cluster to its previous state using the saved etcd snapshots.

1.2. Application backup and restore operations

As a cluster administrator, you can back up and restore applications running on OpenShift Container Platform by using the OpenShift API for Data Protection (OADP).

OADP backs up and restores Kubernetes resources and internal images, at the granularity of a namespace, by using the version of Velero that is appropriate for the version of OADP you install, according to the table in Downloading the Velero CLI tool. OADP backs up and restores persistent volumes (PVs) by using snapshots or Restic. For details, see OADP features.

1.2.1. OADP requirements

OADP has the following requirements:

  • You must be logged in as a user with a cluster-admin role.
  • You must have object storage for storing backups, such as one of the following storage types:

    • OpenShift Data Foundation
    • Amazon Web Services
    • Microsoft Azure
    • Google Cloud Platform
    • S3-compatible object storage
    • IBM Cloud® Object Storage S3
Note

If you want to use CSI backup on OCP 4.11 and later, install OADP 1.1.x.

OADP 1.0.x does not support CSI backup on OCP 4.11 and later. OADP 1.0.x includes Velero 1.7.x and expects the API group snapshot.storage.k8s.io/v1beta1, which is not present on OCP 4.11 and later.

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

  • To back up PVs with snapshots, you must have cloud storage that has a native snapshot API or supports Container Storage Interface (CSI) snapshots, such as the following providers:

    • Amazon Web Services
    • Microsoft Azure
    • Google Cloud Platform
    • CSI snapshot-enabled cloud storage, such as Ceph RBD or Ceph FS
Note

If you do not want to back up PVs by using snapshots, you can use Restic, which is installed by the OADP Operator by default.

1.2.2. Backing up and restoring applications

You back up applications by creating a Backup custom resource (CR). See Creating a Backup CR. You can configure the following backup options:

Chapter 2. Shutting down the cluster gracefully

This document describes the process to gracefully shut down your cluster. You might need to temporarily shut down your cluster for maintenance reasons, or to save on resource costs.

2.1. Prerequisites

  • Take an etcd backup prior to shutting down the cluster.

    Important

    It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues when restarting the cluster.

    For example, the following conditions can cause the restarted cluster to malfunction:

    • etcd data corruption during shutdown
    • Node failure due to hardware
    • Network connectivity issues

    If your cluster fails to recover, follow the steps to restore to a previous cluster state.

2.2. Shutting down the cluster

You can shut down your cluster in a graceful manner so that it can be restarted at a later date.

Note

You can shut down a cluster until a year from the installation date and expect it to restart gracefully. After a year from the installation date, the cluster certificates expire.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have taken an etcd backup.

Procedure

  1. If you are shutting the cluster down for an extended period, determine the date on which certificates expire and run the following command:

    $ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}'

    Example output

    2022-08-05T14:37:50Zuser@user:~ $ 1

    1
    To ensure that the cluster can restart gracefully, plan to restart it on or before the specified date. As the cluster restarts, the process might require you to manually approve the pending certificate signing requests (CSRs) to recover kubelet certificates.
  2. Mark all the nodes in the cluster as unschedulable. You can do this from your cloud provider’s web console, or by running the following loop:

    $ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm cordon ${node} ; done

    Example output

    ci-ln-mgdnf4b-72292-n547t-master-0
    node/ci-ln-mgdnf4b-72292-n547t-master-0 cordoned
    ci-ln-mgdnf4b-72292-n547t-master-1
    node/ci-ln-mgdnf4b-72292-n547t-master-1 cordoned
    ci-ln-mgdnf4b-72292-n547t-master-2
    node/ci-ln-mgdnf4b-72292-n547t-master-2 cordoned
    ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl
    node/ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl cordoned
    ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k
    node/ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k cordoned
    ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn
    node/ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn cordoned

  3. Evacuate the pods using the following method:

    $ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm drain ${node} --delete-emptydir-data --ignore-daemonsets=true --timeout=15s --force ; done
  4. Shut down all of the nodes in the cluster. You can do this from the web console for your cloud provider web console, or by running the following loop. Shutting down the nodes by using one of these methods allows pods to terminate gracefully, which reduces the chance for data corruption.

    Note

    Ensure that the control plane node with the API VIP assigned is the last node processed in the loop. Otherwise, the shutdown command fails.

    $ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 1; done 1
    1
    -h 1 indicates how long, in minutes, this process lasts before the control plane nodes are shut down. For large-scale clusters with 10 nodes or more, set to -h 10 or longer to make sure all the compute nodes have time to shut down first.

    Example output

    Starting pod/ip-10-0-130-169us-east-2computeinternal-debug ...
    To use host binaries, run `chroot /host`
    Shutdown scheduled for Mon 2021-09-13 09:36:17 UTC, use 'shutdown -c' to cancel.
    Removing debug pod ...
    Starting pod/ip-10-0-150-116us-east-2computeinternal-debug ...
    To use host binaries, run `chroot /host`
    Shutdown scheduled for Mon 2021-09-13 09:36:29 UTC, use 'shutdown -c' to cancel.

    Note

    It is not necessary to drain control plane nodes of the standard pods that ship with OpenShift Container Platform prior to shutdown. Cluster administrators are responsible for ensuring a clean restart of their own workloads after the cluster is restarted. If you drained control plane nodes prior to shutdown because of custom workloads, you must mark the control plane nodes as schedulable before the cluster will be functional again after restart.

  5. Shut off any cluster dependencies that are no longer needed, such as external storage or an LDAP server. Be sure to consult your vendor’s documentation before doing so.

    Important

    If you deployed your cluster on a cloud-provider platform, do not shut down, suspend, or delete the associated cloud resources. If you delete the cloud resources of a suspended virtual machine, OpenShift Container Platform might not restore successfully.

2.3. Additional resources

Chapter 3. Restarting the cluster gracefully

This document describes the process to restart your cluster after a graceful shutdown.

Even though the cluster is expected to be functional after the restart, the cluster might not recover due to unexpected conditions, for example:

  • etcd data corruption during shutdown
  • Node failure due to hardware
  • Network connectivity issues

If your cluster fails to recover, follow the steps to restore to a previous cluster state.

3.1. Prerequisites

3.2. Restarting the cluster

You can restart your cluster after it has been shut down gracefully.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • This procedure assumes that you gracefully shut down the cluster.

Procedure

  1. Power on any cluster dependencies, such as external storage or an LDAP server.
  2. Start all cluster machines.

    Use the appropriate method for your cloud environment to start the machines, for example, from your cloud provider’s web console.

    Wait approximately 10 minutes before continuing to check the status of control plane nodes.

  3. Verify that all control plane nodes are ready.

    $ oc get nodes -l node-role.kubernetes.io/master

    The control plane nodes are ready if the status is Ready, as shown in the following output:

    NAME                           STATUS   ROLES                  AGE   VERSION
    ip-10-0-168-251.ec2.internal   Ready    control-plane,master   75m   v1.30.3
    ip-10-0-170-223.ec2.internal   Ready    control-plane,master   75m   v1.30.3
    ip-10-0-211-16.ec2.internal    Ready    control-plane,master   75m   v1.30.3
  4. If the control plane nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.

    1. Get the list of current CSRs:

      $ oc get csr
    2. Review the details of a CSR to verify that it is valid:

      $ oc describe csr <csr_name> 1
      1
      <csr_name> is the name of a CSR from the list of current CSRs.
    3. Approve each valid CSR:

      $ oc adm certificate approve <csr_name>
  5. After the control plane nodes are ready, verify that all worker nodes are ready.

    $ oc get nodes -l node-role.kubernetes.io/worker

    The worker nodes are ready if the status is Ready, as shown in the following output:

    NAME                           STATUS   ROLES    AGE   VERSION
    ip-10-0-179-95.ec2.internal    Ready    worker   64m   v1.30.3
    ip-10-0-182-134.ec2.internal   Ready    worker   64m   v1.30.3
    ip-10-0-250-100.ec2.internal   Ready    worker   64m   v1.30.3
  6. If the worker nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.

    1. Get the list of current CSRs:

      $ oc get csr
    2. Review the details of a CSR to verify that it is valid:

      $ oc describe csr <csr_name> 1
      1
      <csr_name> is the name of a CSR from the list of current CSRs.
    3. Approve each valid CSR:

      $ oc adm certificate approve <csr_name>
  7. After the control plane and worker nodes are ready, mark all the nodes in the cluster as schedulable. Run the following command:

    for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm uncordon ${node} ; done
  8. Verify that the cluster started properly.

    1. Check that there are no degraded cluster Operators.

      $ oc get clusteroperators

      Check that there are no cluster Operators with the DEGRADED condition set to True.

      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
      authentication                             4.17.0    True        False         False      59m
      cloud-credential                           4.17.0    True        False         False      85m
      cluster-autoscaler                         4.17.0    True        False         False      73m
      config-operator                            4.17.0    True        False         False      73m
      console                                    4.17.0    True        False         False      62m
      csi-snapshot-controller                    4.17.0    True        False         False      66m
      dns                                        4.17.0    True        False         False      76m
      etcd                                       4.17.0    True        False         False      76m
      ...
    2. Check that all nodes are in the Ready state:

      $ oc get nodes

      Check that the status for all nodes is Ready.

      NAME                           STATUS   ROLES                  AGE   VERSION
      ip-10-0-168-251.ec2.internal   Ready    control-plane,master   82m   v1.30.3
      ip-10-0-170-223.ec2.internal   Ready    control-plane.master   82m   v1.30.3
      ip-10-0-179-95.ec2.internal    Ready    worker                 70m   v1.30.3
      ip-10-0-182-134.ec2.internal   Ready    worker                 70m   v1.30.3
      ip-10-0-211-16.ec2.internal    Ready    control-plane,master   82m   v1.30.3
      ip-10-0-250-100.ec2.internal   Ready    worker                 69m   v1.30.3

      If the cluster did not start properly, you might need to restore your cluster using an etcd backup.

Additional resources

Chapter 4. OADP Application backup and restore

4.1. Introduction to OpenShift API for Data Protection

The OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs).

However, OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.

OADP support is provided to customer workload namespaces, and cluster scope resources.

Full cluster backup and restore are not supported.

4.1.1. OpenShift API for Data Protection APIs

OpenShift API for Data Protection (OADP) provides APIs that enable multiple approaches to customizing backups and preventing the inclusion of unnecessary or inappropriate resources.

OADP provides the following APIs:

4.1.1.1. Support for OpenShift API for Data Protection
Table 4.1. Supported versions of OADP

Version

OCP version

General availability

Full support ends

Maintenance ends

Extended Update Support (EUS)

Extended Update Support Term 2 (EUS Term 2)

1.4

  • 4.14
  • 4.15
  • 4.16

10 Jul 2024

Release of 1.5

Release of 1.6

27 Jun 2026

EUS must be on OCP 4.16

27 Jun 2027

EUS Term 2 must be on OCP 4.16

1.3

  • 4.12
  • 4.13
  • 4.14
  • 4.15

29 Nov 2023

10 Jul 2024

Release of 1.5

31 Oct 2025

EUS must be on OCP 4.14

31 Oct 2026

EUS Term 2 must be on OCP 4.14

4.1.1.1.1. Unsupported versions of the OADP Operator
Table 4.2. Previous versions of the OADP Operator which are no longer supported

Version

General availability

Full support ended

Maintenance ended

1.2

14 Jun 2023

29 Nov 2023

10 Jul 2024

1.1

01 Sep 2022

14 Jun 2023

29 Nov 2023

1.0

09 Feb 2022

01 Sep 2022

14 Jun 2023

For more details about EUS, see Extended Update Support.

For more details about EUS Term 2, see Extended Update Support Term 2.

Additional resources

4.2. OADP release notes

4.2.1. OADP 1.4 release notes

The release notes for OpenShift API for Data Protection (OADP) describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.

Note

For additional information about OADP, see OpenShift API for Data Protection (OADP) FAQs

4.2.1.1. OADP 1.4.1 release notes

The OpenShift API for Data Protection (OADP) 1.4.1 release notes lists new features, resolved issues and bugs, and known issues.

4.2.1.1.1. New features

New DPA fields to update client qps and burst

You can now change Velero Server Kubernetes API queries per second and burst values by using the new Data Protection Application (DPA) fields. The new DPA fields are spec.configuration.velero.client-qps and spec.configuration.velero.client-burst, which both default to 100. OADP-4076

Enabling non-default algorithms with Kopia

With this update, you can now configure the hash, encryption, and splitter algorithms in Kopia to select non-default options to optimize performance for different backup workloads.

To configure these algorithms, set the env variable of a velero pod in the podConfig section of the DataProtectionApplication (DPA) configuration. If this variable is not set, or an unsupported algorithm is chosen, Kopia will default to its standard algorithms. OADP-4640

4.2.1.1.2. Resolved issues

Restoring a backup without pods is now successful

Previously, restoring a backup without pods and having StorageClass VolumeBindingMode set as WaitForFirstConsumer, resulted in the PartiallyFailed status with an error: fail to patch dynamic PV, err: context deadline exceeded. With this update, patching dynamic PV is skipped and restoring a backup is successful without any PartiallyFailed status. OADP-4231

PodVolumeBackup CR now displays correct message

Previously, the PodVolumeBackup custom resource (CR) generated an incorrect message, which was: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed". With this update, the message produced is now:

found a podvolumebackup with status "InProgress" during the server starting,
mark it as "Failed".

OADP-4224

Overriding imagePullPolicy is now possible with DPA

Previously, OADP set the imagePullPolicy parameter to Always for all images. With this update, OADP checks if each image contains sha256 or sha512 digest, then it sets imagePullPolicy to IfNotPresent; otherwise imagePullPolicy is set to Always. You can now override this policy by using the new spec.containerImagePullPolicy DPA field. OADP-4172

OADP Velero can now retry updating the restore status if initial update fails

Previously, OADP Velero failed to update the restored CR status. This left the status at InProgress indefinitely. Components which relied on the backup and restore CR status to determine the completion would fail. With this update, the restore CR status for a restore correctly proceeds to the Completed or Failed status. OADP-3227

Restoring BuildConfig Build from a different cluster is successful without any errors

Previously, when performing a restore of the BuildConfig Build resource from a different cluster, the application generated an error on TLS verification to the internal image registry. The resulting error was failed to verify certificate: x509: certificate signed by unknown authority error. With this update, the restore of the BuildConfig build resources to a different cluster can proceed successfully without generating the failed to verify certificate error. OADP-4692

Restoring an empty PVC is successful

Previously, downloading data failed while restoring an empty persistent volume claim (PVC). It failed with the following error:

data path restore failed: Failed to run kopia restore: Unable to load
    snapshot : snapshot not found

With this update, the downloading of data proceeds to correct conclusion when restoring an empty PVC and the error message is not generated. OADP-3106

There is no Velero memory leak in CSI and DataMover plugins

Previously, a Velero memory leak was caused by using the CSI and DataMover plugins. When the backup ended, the Velero plugin instance was not deleted and the memory leak consumed memory until an Out of Memory (OOM) condition was generated in the Velero pod. With this update, there is no resulting Velero memory leak when using the CSI and DataMover plugins. OADP-4448

Post-hook operation does not start before the related PVs are released

Previously, due to the asynchronous nature of the Data Mover operation, a post-hook might be attempted before the Data Mover persistent volume claim (PVC) releases the persistent volumes (PVs) of the related pods. This problem would cause the backup to fail with a PartiallyFailed status. With this update, the post-hook operation is not started until the related PVs are released by the Data Mover PVC, eliminating the PartiallyFailed backup status. OADP-3140

Deploying a DPA works as expected in namespaces with more than 37 characters

When you install the OADP Operator in a namespace with more than 37 characters to create a new DPA, labeling the "cloud-credentials" Secret fails and the DPA reports the following error:

The generated label name is too long.

With this update, creating a DPA does not fail in namespaces with more than 37 characters in the name. OADP-3960

Restore is successfully completed by overriding the timeout error

Previously, in a large scale environment, the restore operation would result in a Partiallyfailed status with the error: fail to patch dynamic PV, err: context deadline exceeded. With this update, the resourceTimeout Velero server argument is used to override this timeout error resulting in a successful restore. OADP-4344

For a complete list of all issues resolved in this release, see the list of OADP 1.4.1 resolved issues in Jira.

4.2.1.1.3. Known issues

Cassandra application pods enter into the CrashLoopBackoff status after restoring OADP

After OADP restores, the Cassandra application pods might enter CrashLoopBackoff status. To work around this problem, delete the StatefulSet pods that are returning the error CrashLoopBackoff state after restoring OADP. The StatefulSet controller then recreates these pods and it runs normally. OADP-4407

Deployment referencing ImageStream is not restored properly leading to corrupted pod and volume contents

During a File System Backup (FSB) restore operation, a Deployment resource referencing an ImageStream is not restored properly. The restored pod that runs the FSB, and the postHook is terminated prematurely.

During the restore operation, the OpenShift Container Platform controller updates the spec.template.spec.containers[0].image field in the Deployment resource with an updated ImageStreamTag hash. The update triggers the rollout of a new pod, terminating the pod on which velero runs the FSB along with the post-hook. For more information about image stream trigger, see Triggering updates on image stream changes.

The workaround for this behavior is a two-step restore process:

  1. Perform a restore excluding the Deployment resources, for example:

    $ velero restore create <RESTORE_NAME> \
      --from-backup <BACKUP_NAME> \
      --exclude-resources=deployment.apps
  2. Once the first restore is successful, perform a second restore by including these resources, for example:

    $ velero restore create <RESTORE_NAME> \
      --from-backup <BACKUP_NAME> \
      --include-resources=deployment.apps

    OADP-3954

4.2.1.2. OADP 1.4.0 release notes

The OpenShift API for Data Protection (OADP) 1.4.0 release notes lists resolved issues and known issues.

4.2.1.2.1. Resolved issues

Restore works correctly in OpenShift Container Platform 4.16

Previously, while restoring the deleted application namespace, the restore operation partially failed with the resource name may not be empty error in OpenShift Container Platform 4.16. With this update, restore works as expected in OpenShift Container Platform 4.16. OADP-4075

Data Mover backups work properly in the OpenShift Container Platform 4.16 cluster

Previously, Velero was using the earlier version of SDK where the Spec.SourceVolumeMode field did not exist. As a consequence, Data Mover backups failed in the OpenShift Container Platform 4.16 cluster on the external snapshotter with version 4.2. With this update, external snapshotter is upgraded to version 7.0 and later. As a result, backups do not fail in the OpenShift Container Platform 4.16 cluster. OADP-3922

For a complete list of all issues resolved in this release, see the list of OADP 1.4.0 resolved issues in Jira.

4.2.1.2.2. Known issues

Backup fails when checksumAlgorithm is not set for MCG

While performing a backup of any application with Noobaa as the backup location, if the checksumAlgorithm configuration parameter is not set, backup fails. To fix this problem, if you do not provide a value for checksumAlgorithm in the Backup Storage Location (BSL) configuration, an empty value is added. The empty value is only added for BSLs that are created using Data Protection Application (DPA) custom resource (CR), and this value is not added if BSLs are created using any other method. OADP-4274

For a complete list of all known issues in this release, see the list of OADP 1.4.0 known issues in Jira.

4.2.1.2.3. Upgrade notes
Note

Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OpenShift API for Data Protection (OADP) 1.1 to 1.3, upgrade first to 1.2, and then to 1.3.

4.2.1.2.3.1. Changes from OADP 1.3 to 1.4

The Velero server has been updated from version 1.12 to 1.14. Note that there are no changes in the Data Protection Application (DPA).

This changes the following:

  • The velero-plugin-for-csi code is now available in the Velero code, which means an init container is no longer required for the plugin.
  • Velero changed client Burst and QPS defaults from 30 and 20 to 100 and 100, respectively.
  • The velero-plugin-for-aws plugin updated default value of the spec.config.checksumAlgorithm field in BackupStorageLocation objects (BSLs) from "" (no checksum calculation) to the CRC32 algorithm. The checksum algorithm types are known to work only with AWS. Several S3 providers require the md5sum to be disabled by setting the checksum algorithm to "". Confirm md5sum algorithm support and configuration with your storage provider.

    In OADP 1.4, the default value for BSLs created within DPA for this configuration is "". This default value means that the md5sum is not checked, which is consistent with OADP 1.3. For BSLs created within DPA, update it by using the spec.backupLocations[].velero.config.checksumAlgorithm field in the DPA. If your BSLs are created outside DPA, you can update this configuration by using spec.config.checksumAlgorithm in the BSLs.

4.2.1.2.3.2. Backing up the DPA configuration

You must back up your current DataProtectionApplication (DPA) configuration.

Procedure

  • Save your current DPA configuration by running the following command:

    Example command

    $ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup

4.2.1.2.3.3. Upgrading the OADP Operator

Use the following procedure when upgrading the OpenShift API for Data Protection (OADP) Operator.

Procedure

  1. Change your subscription channel for the OADP Operator from stable-1.3 to stable-1.4.
  2. Wait for the Operator and containers to update and restart.

Additional resources

4.2.1.2.4. Converting DPA to the new version

To upgrade from OADP 1.3 to 1.4, no Data Protection Application (DPA) changes are required.

4.2.1.2.5. Verifying the upgrade

Use the following procedure to verify the upgrade.

Procedure

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/restic-9cq4q                                         1/1     Running   0          94s
    pod/restic-m4lts                                         1/1     Running   0          94s
    pod/restic-pv4kr                                         1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    
    NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/restic   3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.3. OADP performance

4.4. OADP features and plugins

OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications.

The default plugins enable Velero to integrate with certain cloud providers and to back up and restore OpenShift Container Platform resources.

4.4.1. OADP features

OpenShift API for Data Protection (OADP) supports the following features:

Backup

You can use OADP to back up all applications on the OpenShift Platform, or you can filter the resources by type, namespace, or label.

OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.

Note

You must exclude Operators from the backup of an application for backup and restore to succeed.

Restore

You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the objects by namespace, PV, or label.

Note

You must exclude Operators from the backup of an application for backup and restore to succeed.

Schedule
You can schedule backups at specified intervals.
Hooks
You can use hooks to run commands in a container on a pod, for example, fsfreeze to freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.

4.4.2. OADP plugins

The OpenShift API for Data Protection (OADP) provides default Velero plugins that are integrated with storage providers to support backup and snapshot operations. You can create custom plugins based on the Velero plugins.

OADP also provides plugins for OpenShift Container Platform resource backups, OpenShift Virtualization resource backups, and Container Storage Interface (CSI) snapshots.

Table 4.3. OADP plugins
OADP pluginFunctionStorage location

aws

Backs up and restores Kubernetes objects.

AWS S3

Backs up and restores volumes with snapshots.

AWS EBS

azure

Backs up and restores Kubernetes objects.

Microsoft Azure Blob storage

Backs up and restores volumes with snapshots.

Microsoft Azure Managed Disks

gcp

Backs up and restores Kubernetes objects.

Google Cloud Storage

Backs up and restores volumes with snapshots.

Google Compute Engine Disks

openshift

Backs up and restores OpenShift Container Platform resources. [1]

Object store

kubevirt

Backs up and restores OpenShift Virtualization resources. [2]

Object store

csi

Backs up and restores volumes with CSI snapshots. [3]

Cloud storage that supports CSI snapshots

vsm

VolumeSnapshotMover relocates snapshots from the cluster into an object store to be used during a restore process to recover stateful applications, in situations such as cluster deletion. [4]

Object store

  1. Mandatory.
  2. Virtual machine disks are backed up with CSI snapshots or Restic.
  3. The csi plugin uses the Kubernetes CSI snapshot API.

    • OADP 1.1 or later uses snapshot.storage.k8s.io/v1
    • OADP 1.0 uses snapshot.storage.k8s.io/v1beta1
  4. OADP 1.2 only.

4.4.3. About OADP Velero plugins

You can configure two types of plugins when you install Velero:

  • Default cloud provider plugins
  • Custom plugins

Both types of plugin are optional, but most users configure at least one cloud provider plugin.

4.4.3.1. Default Velero cloud provider plugins

You can install any of the following default Velero cloud provider plugins when you configure the oadp_v1alpha1_dpa.yaml file during deployment:

  • aws (Amazon Web Services)
  • gcp (Google Cloud Platform)
  • azure (Microsoft Azure)
  • openshift (OpenShift Velero plugin)
  • csi (Container Storage Interface)
  • kubevirt (KubeVirt)

You specify the desired default plugins in the oadp_v1alpha1_dpa.yaml file during deployment.

Example file

The following .yaml file installs the openshift, aws, azure, and gcp plugins:

 apiVersion: oadp.openshift.io/v1alpha1
 kind: DataProtectionApplication
 metadata:
   name: dpa-sample
 spec:
   configuration:
     velero:
       defaultPlugins:
       - openshift
       - aws
       - azure
       - gcp
4.4.3.2. Custom Velero plugins

You can install a custom Velero plugin by specifying the plugin image and name when you configure the oadp_v1alpha1_dpa.yaml file during deployment.

You specify the desired custom plugins in the oadp_v1alpha1_dpa.yaml file during deployment.

Example file

The following .yaml file installs the default openshift, azure, and gcp plugins and a custom plugin that has the name custom-plugin-example and the image quay.io/example-repo/custom-velero-plugin:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-sample
spec:
 configuration:
   velero:
     defaultPlugins:
     - openshift
     - azure
     - gcp
     customPlugins:
     - name: custom-plugin-example
       image: quay.io/example-repo/custom-velero-plugin
4.4.3.3. Velero plugins returning "received EOF, stopping recv loop" message
Note

Velero plugins are started as separate processes. After the Velero operation has completed, either successfully or not, they exit. Receiving a received EOF, stopping recv loop message in the debug logs indicates that a plugin operation has completed. It does not mean that an error has occurred.

4.4.4. Supported architectures for OADP

OpenShift API for Data Protection (OADP) supports the following architectures:

  • AMD64
  • ARM64
  • PPC64le
  • s390x
Note

OADP 1.2.0 and later versions support the ARM64 architecture.

4.4.5. OADP support for IBM Power and IBM Z

OpenShift API for Data Protection (OADP) is platform neutral. The information that follows relates only to IBM Power® and to IBM Z®.

  • OADP 1.1.7 was tested successfully against OpenShift Container Platform 4.11 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.1.7 in terms of backup locations for these systems.
  • OADP 1.2.3 was tested successfully against OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.2.3 in terms of backup locations for these systems.
  • OADP 1.3.3 was tested successfully against OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.3.3 in terms of backup locations for these systems.
  • OADP 1.4.1 was tested successfully against OpenShift Container Platform 4.14, 4.15, and 4.16 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.4.1 in terms of backup locations for these systems.
4.4.5.1. OADP support for target backup locations using IBM Power
  • IBM Power® running with OpenShift Container Platform 4.11 and 4.12, and OpenShift API for Data Protection (OADP) 1.1.7 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.11 and 4.12, and OADP 1.1.7 against all S3 backup location targets, which are not AWS, as well.
  • IBM Power® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and OADP 1.2.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.12, 4.13. 4.14, and 4.15, and OADP 1.2.3 against all S3 backup location targets, which are not AWS, as well.
  • IBM Power® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and OADP 1.3.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.13, 4.14, and 4.15, and OADP 1.3.3 against all S3 backup location targets, which are not AWS, as well.
  • IBM Power® running with OpenShift Container Platform 4.14, 4.15, and 4.16, and OADP 1.4.1 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.14, 4.15, and 4.16, and OADP 1.4.1 against all S3 backup location targets, which are not AWS, as well.
4.4.5.2. OADP testing and support for target backup locations using IBM Z
  • IBM Z® running with OpenShift Container Platform 4.11 and 4.12, and OpenShift API for Data Protection (OADP) 1.1.7 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.11 and 4.12, and OADP 1.1.7 against all S3 backup location targets, which are not AWS, as well.
  • IBM Z® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and OADP 1.2.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.12, 4.13, 4.14 and 4.15, and OADP 1.2.3 against all S3 backup location targets, which are not AWS, as well.
  • IBM Z® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and 1.3.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.13 4.14, and 4.15, and 1.3.3 against all S3 backup location targets, which are not AWS, as well.
  • IBM Z® running with OpenShift Container Platform 4.14, 4.15, and 4.16, and 1.4.1 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.14, 4.15, and 4.16, and 1.4.1 against all S3 backup location targets, which are not AWS, as well.
4.4.5.2.1. Known issue of OADP using IBM Power(R) and IBM Z(R) platforms
  • Currently, there are backup method restrictions for Single-node OpenShift clusters deployed on IBM Power® and IBM Z® platforms. Only NFS storage is currently compatible with Single-node OpenShift clusters on these platforms. In addition, only the File System Backup (FSB) methods such as Kopia and Restic are supported for backup and restore operations. There is currently no workaround for this issue.

4.4.6. OADP plugins known issues

The following section describes known issues in OpenShift API for Data Protection (OADP) plugins:

4.4.6.1. Velero plugin panics during imagestream backups due to a missing secret

When the backup and the Backup Storage Location (BSL) are managed outside the scope of the Data Protection Application (DPA), the OADP controller, meaning the DPA reconciliation does not create the relevant oadp-<bsl_name>-<bsl_provider>-registry-secret.

When the backup is run, the OpenShift Velero plugin panics on the imagestream backup, with the following panic error:

024-02-27T10:46:50.028951744Z time="2024-02-27T10:46:50Z" level=error msg="Error backing up item"
backup=openshift-adp/<backup name> error="error executing custom action (groupResource=imagestreams.image.openshift.io,
namespace=<BSL Name>, name=postgres): rpc error: code = Aborted desc = plugin panicked:
runtime error: index out of range with length 1, stack trace: goroutine 94…
4.4.6.1.1. Workaround to avoid the panic error

To avoid the Velero plugin panic error, perform the following steps:

  1. Label the custom BSL with the relevant label:

    $ oc label BackupStorageLocation <bsl_name> app.kubernetes.io/component=bsl
  2. After the BSL is labeled, wait until the DPA reconciles.

    Note

    You can force the reconciliation by making any minor change to the DPA itself.

  3. When the DPA reconciles, confirm that the relevant oadp-<bsl_name>-<bsl_provider>-registry-secret has been created and that the correct registry data has been populated into it:

    $ oc -n openshift-adp get secret/oadp-<bsl_name>-<bsl_provider>-registry-secret -o json | jq -r '.data'
4.4.6.2. OpenShift ADP Controller segmentation fault

If you configure a DPA with both cloudstorage and restic enabled, the openshift-adp-controller-manager pod crashes and restarts indefinitely until the pod fails with a crash loop segmentation fault.

You can have either velero or cloudstorage defined, because they are mutually exclusive fields.

  • If you have both velero and cloudstorage defined, the openshift-adp-controller-manager fails.
  • If you have neither velero nor cloudstorage defined, the openshift-adp-controller-manager fails.

For more information about this issue, see OADP-1054.

4.4.6.2.1. OpenShift ADP Controller segmentation fault workaround

You must define either velero or cloudstorage when you configure a DPA. If you define both APIs in your DPA, the openshift-adp-controller-manager pod fails with a crash loop segmentation fault.

4.5. OADP use cases

4.5.1. Backup using OpenShift API for Data Protection and Red Hat OpenShift Data Foundation (ODF)

Following is a use case for using OADP and ODF to back up an application.

4.5.1.1. Backing up an application using OADP and ODF

In this use case, you back up an application by using OADP and store the backup in an object storage provided by Red Hat OpenShift Data Foundation (ODF).

  • You create a object bucket claim (OBC) to configure the backup storage location. You use ODF to configure an Amazon S3-compatible object storage bucket. ODF provides MultiCloud Object Gateway (NooBaa MCG) and Ceph Object Gateway, also known as RADOS Gateway (RGW), object storage service. In this use case, you use NooBaa MCG as the backup storage location.
  • You use the NooBaa MCG service with OADP by using the aws provider plugin.
  • You configure the Data Protection Application (DPA) with the backup storage location (BSL).
  • You create a backup custom resource (CR) and specify the application namespace to back up.
  • You create and verify the backup.

Prerequisites

  • You installed the OADP Operator.
  • You installed the ODF Operator.
  • You have an application with a database running in a separate namespace.

Procedure

  1. Create an OBC manifest file to request a NooBaa MCG bucket as shown in the following example:

    Example OBC

    apiVersion: objectbucket.io/v1alpha1
    kind: ObjectBucketClaim
    metadata:
      name: test-obc 1
      namespace: openshift-adp
    spec:
      storageClassName: openshift-storage.noobaa.io
      generateBucketName: test-backup-bucket 2

    1
    The name of the object bucket claim.
    2
    The name of the bucket.
  2. Create the OBC by running the following command:

    $ oc create -f <obc_file_name> 1
    1
    Specify the file name of the object bucket claim manifest.
  3. When you create an OBC, ODF creates a secret and a config map with the same name as the object bucket claim. The secret has the bucket credentials, and the config map has information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:

    $ oc extract --to=- cm/test-obc 1
    1
    test-obc is the name of the OBC.

    Example output

    # BUCKET_NAME
    backup-c20...41fd
    # BUCKET_PORT
    443
    # BUCKET_REGION
    
    # BUCKET_SUBREGION
    
    # BUCKET_HOST
    s3.openshift-storage.svc

  4. To get the bucket credentials from the generated secret, run the following command:

    $ oc extract --to=- secret/test-obc

    Example output

    # AWS_ACCESS_KEY_ID
    ebYR....xLNMc
    # AWS_SECRET_ACCESS_KEY
    YXf...+NaCkdyC3QPym

  5. Get the public URL for the S3 endpoint from the s3 route in the openshift-storage namespace by running the following command:

    $ oc get route s3 -n openshift-storage
  6. Create a cloud-credentials file with the object bucket credentials as shown in the following command:

    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
  7. Create the cloud-credentials secret with the cloud-credentials file content as shown in the following command:

    $ oc create secret generic \
      cloud-credentials \
      -n openshift-adp \
      --from-file cloud=cloud-credentials
  8. Configure the Data Protection Application (DPA) as shown in the following example:

    Example DPA

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: oadp-backup
      namespace: openshift-adp
    spec:
      configuration:
        nodeAgent:
          enable: true
          uploaderType: kopia
        velero:
          defaultPlugins:
            - aws
            - openshift
            - csi
          defaultSnapshotMoveData: true 1
      backupLocations:
        - velero:
            config:
              profile: "default"
              region: noobaa
              s3Url: https://s3.openshift-storage.svc 2
              s3ForcePathStyle: "true"
              insecureSkipTLSVerify: "true"
            provider: aws
            default: true
            credential:
              key: cloud
              name:  cloud-credentials
            objectStorage:
              bucket: <bucket_name> 3
              prefix: oadp

    1
    Set to true to use the OADP Data Mover to enable movement of Container Storage Interface (CSI) snapshots to a remote object storage.
    2
    This is the S3 URL of ODF storage.
    3
    Specify the bucket name.
  9. Create the DPA by running the following command:

    $ oc apply -f <dpa_filename>
  10. Verify that the DPA is created successfully by running the following command. In the example output, you can see the status object has type field set to Reconciled. This means, the DPA is successfully created.

    $ oc get dpa -o yaml

    Example output

    apiVersion: v1
    items:
    - apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        namespace: openshift-adp
        #...#
      spec:
        backupLocations:
        - velero:
            config:
              #...#
      status:
        conditions:
        - lastTransitionTime: "20....9:54:02Z"
          message: Reconcile complete
          reason: Complete
          status: "True"
          type: Reconciled
    kind: List
    metadata:
      resourceVersion: ""

  11. Verify that the backup storage location (BSL) is available by running the following command:

    $ oc get bsl -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE   DEFAULT
    dpa-sample-1   Available   3s               15s   true

  12. Configure a backup CR as shown in the following example:

    Example backup CR

    apiVersion: velero.io/v1
    kind: Backup
    metadata:
      name: test-backup
      namespace: openshift-adp
    spec:
      includedNamespaces:
      - <application_namespace> 1

    1
    Specify the namespace for the application to back up.
  13. Create the backup CR by running the following command:

    $ oc apply -f <backup_cr_filename>

Verification

  • Verify that the backup object is in the Completed phase by running the following command. For more details, see the example output.

    $ oc describe backup test-backup -n openshift-adp

    Example output

    Name:         test-backup
    Namespace:    openshift-adp
    # ....#
    Status:
      Backup Item Operations Attempted:  1
      Backup Item Operations Completed:  1
      Completion Timestamp:              2024-09-25T10:17:01Z
      Expiration:                        2024-10-25T10:16:31Z
      Format Version:                    1.1.0
      Hook Status:
      Phase:  Completed
      Progress:
        Items Backed Up:  34
        Total Items:      34
      Start Timestamp:    2024-09-25T10:16:31Z
      Version:            1
    Events:               <none>

4.5.2. OpenShift API for Data Protection (OADP) restore use case

Following is a use case for using OADP to restore a backup to a different namespace.

4.5.2.1. Restoring an application to a different namespace using OADP

Restore a backup of an application by using OADP to a new target namespace, test-restore-application. To restore a backup, you create a restore custom resource (CR) as shown in the following example. In the restore CR, the source namespace refers to the application namespace that you included in the backup. You then verify the restore by changing your project to the new restored namespace and verifying the resources.

Prerequisites

  • You installed the OADP Operator.
  • You have the backup of an application to be restored.

Procedure

  1. Create a restore CR as shown in the following example:

    Example restore CR

    apiVersion: velero.io/v1
    kind: Restore
    metadata:
      name: test-restore 1
      namespace: openshift-adp
    spec:
      backupName: <backup_name> 2
      restorePVs: true
      namespaceMapping:
        <application_namespace>: test-restore-application 3

    1
    The name of the restore CR.
    2
    Specify the name of the backup.
    3
    namespaceMapping maps the source application namespace to the target application namespace. Specify the application namespace that you backed up. test-restore-application is the target namespace where you want to restore the backup.
  2. Apply the restore CR by running the following command:

    $ oc apply -f <restore_cr_filename>

Verification

  1. Verify that the restore is in the Completed phase by running the following command:

    $ oc describe restore <restore_name> -n openshift-adp
  2. Change to the restored namespace test-restore-application by running the following command:

    $ oc project test-restore-application
  3. Verify the restored resources such as persistent volume claim (pvc), service (svc), deployment, secret, and config map by running the following command:

    $ oc get pvc,svc,deployment,secret,configmap

    Example output

    NAME                          STATUS   VOLUME
    persistentvolumeclaim/mysql   Bound    pvc-9b3583db-...-14b86
    
    NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
    service/mysql      ClusterIP   172....157     <none>        3306/TCP   2m56s
    service/todolist   ClusterIP   172.....15     <none>        8000/TCP   2m56s
    
    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/mysql   0/1     1            0           2m55s
    
    NAME                                         TYPE                      DATA   AGE
    secret/builder-dockercfg-6bfmd               kubernetes.io/dockercfg   1      2m57s
    secret/default-dockercfg-hz9kz               kubernetes.io/dockercfg   1      2m57s
    secret/deployer-dockercfg-86cvd              kubernetes.io/dockercfg   1      2m57s
    secret/mysql-persistent-sa-dockercfg-rgp9b   kubernetes.io/dockercfg   1      2m57s
    
    NAME                                 DATA   AGE
    configmap/kube-root-ca.crt           1      2m57s
    configmap/openshift-service-ca.crt   1      2m57s

4.5.3. Including a self-signed CA certificate during backup

You can include a self-signed Certificate Authority (CA) certificate in the Data Protection Application (DPA) and then back up an application. You store the backup in a NooBaa bucket provided by Red Hat OpenShift Data Foundation (ODF).

4.5.3.1. Backing up an application and its self-signed CA certificate

The s3.openshift-storage.svc service, provided by ODF, uses a Transport Layer Security protocol (TLS) certificate that is signed with the self-signed service CA.

To prevent a certificate signed by unknown authority error, you must include a self-signed CA certificate in the backup storage location (BSL) section of DataProtectionApplication custom resource (CR). For this situation, you must complete the following tasks:

  • Request a NooBaa bucket by creating an object bucket claim (OBC).
  • Extract the bucket details.
  • Include a self-signed CA certificate in the DataProtectionApplication CR.
  • Back up an application.

Prerequisites

  • You installed the OADP Operator.
  • You installed the ODF Operator.
  • You have an application with a database running in a separate namespace.

Procedure

  1. Create an OBC manifest to request a NooBaa bucket as shown in the following example:

    Example ObjectBucketClaim CR

    apiVersion: objectbucket.io/v1alpha1
    kind: ObjectBucketClaim
    metadata:
      name: test-obc 1
      namespace: openshift-adp
    spec:
      storageClassName: openshift-storage.noobaa.io
      generateBucketName: test-backup-bucket 2

    1
    Specifies the name of the object bucket claim.
    2
    Specifies the name of the bucket.
  2. Create the OBC by running the following command:

    $ oc create -f <obc_file_name>
  3. When you create an OBC, ODF creates a secret and a ConfigMap with the same name as the object bucket claim. The secret object contains the bucket credentials, and the ConfigMap object contains information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:

    $ oc extract --to=- cm/test-obc 1
    1
    The name of the OBC is test-obc.

    Example output

    # BUCKET_NAME
    backup-c20...41fd
    # BUCKET_PORT
    443
    # BUCKET_REGION
    
    # BUCKET_SUBREGION
    
    # BUCKET_HOST
    s3.openshift-storage.svc

  4. To get the bucket credentials from the secret object , run the following command:

    $ oc extract --to=- secret/test-obc

    Example output

    # AWS_ACCESS_KEY_ID
    ebYR....xLNMc
    # AWS_SECRET_ACCESS_KEY
    YXf...+NaCkdyC3QPym

  5. Create a cloud-credentials file with the object bucket credentials by using the following example configuration:

    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
  6. Create the cloud-credentials secret with the cloud-credentials file content by running the following command:

    $ oc create secret generic \
      cloud-credentials \
      -n openshift-adp \
      --from-file cloud=cloud-credentials
  7. Extract the service CA certificate from the openshift-service-ca.crt config map by running the following command. Ensure that you encode the certificate in Base64 format and note the value to use in the next step.

    $ oc get cm/openshift-service-ca.crt \
      -o jsonpath='{.data.service-ca\.crt}' | base64 -w0; echo

    Example output

    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0...
    ....gpwOHMwaG9CRmk5a3....FLS0tLS0K

  8. Configure the DataProtectionApplication CR manifest file with the bucket name and CA certificate as shown in the following example:

    Example DataProtectionApplication CR

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: oadp-backup
      namespace: openshift-adp
    spec:
      configuration:
        nodeAgent:
          enable: true
          uploaderType: kopia
        velero:
          defaultPlugins:
            - aws
            - openshift
            - csi
          defaultSnapshotMoveData: true
      backupLocations:
        - velero:
            config:
              profile: "default"
              region: noobaa
              s3Url: https://s3.openshift-storage.svc
              s3ForcePathStyle: "true"
              insecureSkipTLSVerify: "false" 1
            provider: aws
            default: true
            credential:
              key: cloud
              name:  cloud-credentials
            objectStorage:
              bucket: <bucket_name> 2
              prefix: oadp
              caCert: <ca_cert> 3

    1
    The insecureSkipTLSVerify flag can be set to either true or false. If set to "true", SSL/TLS security is disabled. If set to false, SSL/TLS security is enabled.
    2
    Specify the name of the bucket extracted in an earlier step.
    3
    Copy and paste the Base64 encoded certificate from the previous step.
  9. Create the DataProtectionApplication CR by running the following command:

    $ oc apply -f <dpa_filename>
  10. Verify that the DataProtectionApplication CR is created successfully by running the following command:

    $ oc get dpa -o yaml

    Example output

    apiVersion: v1
    items:
    - apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        namespace: openshift-adp
        #...#
      spec:
        backupLocations:
        - velero:
            config:
              #...#
      status:
        conditions:
        - lastTransitionTime: "20....9:54:02Z"
          message: Reconcile complete
          reason: Complete
          status: "True"
          type: Reconciled
    kind: List
    metadata:
      resourceVersion: ""

  11. Verify that the backup storage location (BSL) is available by running the following command:

    $ oc get bsl -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE   DEFAULT
    dpa-sample-1   Available   3s               15s   true

  12. Configure the Backup CR by using the following example:

    Example Backup CR

    apiVersion: velero.io/v1
    kind: Backup
    metadata:
      name: test-backup
      namespace: openshift-adp
    spec:
      includedNamespaces:
      - <application_namespace> 1

    1
    Specify the namespace for the application to back up.
  13. Create the Backup CR by running the following command:

    $ oc apply -f <backup_cr_filename>

Verification

  • Verify that the Backup object is in the Completed phase by running the following command:

    $ oc describe backup test-backup -n openshift-adp

    Example output

    Name:         test-backup
    Namespace:    openshift-adp
    # ....#
    Status:
      Backup Item Operations Attempted:  1
      Backup Item Operations Completed:  1
      Completion Timestamp:              2024-09-25T10:17:01Z
      Expiration:                        2024-10-25T10:16:31Z
      Format Version:                    1.1.0
      Hook Status:
      Phase:  Completed
      Progress:
        Items Backed Up:  34
        Total Items:      34
      Start Timestamp:    2024-09-25T10:16:31Z
      Version:            1
    Events:               <none>

4.6. Installing and configuring OADP

4.6.1. About installing OADP

As a cluster administrator, you install the OpenShift API for Data Protection (OADP) by installing the OADP Operator. The OADP Operator installs Velero 1.14.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

To back up Kubernetes resources and internal images, you must have object storage as a backup location, such as one of the following storage types:

You can configure multiple backup storage locations within the same namespace for each individual OADP deployment.

Note

Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.

For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.

Important

The CloudStorage API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Note

The CloudStorage API is a Technology Preview feature when you use a CloudStorage object and want OADP to use the CloudStorage API to automatically create an S3 bucket for use as a BackupStorageLocation.

The CloudStorage API supports manually creating a BackupStorageLocation object by specifying an existing S3 bucket. The CloudStorage API that creates an S3 bucket automatically is currently only enabled for AWS S3 storage.

You can back up persistent volumes (PVs) by using snapshots or a File System Backup (FSB).

To back up PVs with snapshots, you must have a cloud provider that supports either a native snapshot API or Container Storage Interface (CSI) snapshots, such as one of the following cloud providers:

Note

If you want to use CSI backup on OCP 4.11 and later, install OADP 1.1.x.

OADP 1.0.x does not support CSI backup on OCP 4.11 and later. OADP 1.0.x includes Velero 1.7.x and expects the API group snapshot.storage.k8s.io/v1beta1, which is not present on OCP 4.11 and later.

If your cloud provider does not support snapshots or if your storage is NFS, you can back up applications with Backing up applications with File System Backup: Kopia or Restic on object storage.

You create a default Secret and then you install the Data Protection Application.

4.6.1.1. AWS S3 compatible backup storage providers

OADP is compatible with many object storage providers for use with different backup and snapshot operations. Several object storage providers are fully supported, several are unsupported but known to work, and some have known limitations.

4.6.1.1.1. Supported backup storage providers

The following AWS S3 compatible object storage providers are fully supported by OADP through the AWS plugin for use as backup storage locations:

  • MinIO
  • Multicloud Object Gateway (MCG)
  • Amazon Web Services (AWS) S3
  • IBM Cloud® Object Storage S3
  • Ceph RADOS Gateway (Ceph Object Gateway)
  • Red Hat Container Storage
  • Red Hat OpenShift Data Foundation
Note

The following compatible object storage providers are supported and have their own Velero object store plugins:

  • Google Cloud Platform (GCP)
  • Microsoft Azure
4.6.1.1.2. Unsupported backup storage providers

The following AWS S3 compatible object storage providers, are known to work with Velero through the AWS plugin, for use as backup storage locations, however, they are unsupported and have not been tested by Red Hat:

  • Oracle Cloud
  • DigitalOcean
  • NooBaa, unless installed using Multicloud Object Gateway (MCG)
  • Tencent Cloud
  • Ceph RADOS v12.2.7
  • Quobyte
  • Cloudian HyperStore
Note

Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.

For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.

4.6.1.1.3. Backup storage providers with known limitations

The following AWS S3 compatible object storage providers are known to work with Velero through the AWS plugin with a limited feature set:

  • Swift - It works for use as a backup storage location for backup storage, but is not compatible with Restic for filesystem-based volume backup and restore.
4.6.1.2. Configuring Multicloud Object Gateway (MCG) for disaster recovery on OpenShift Data Foundation

If you use cluster storage for your MCG bucket backupStorageLocation on OpenShift Data Foundation, configure MCG as an external object store.

Warning

Failure to configure MCG as an external object store might lead to backups not being available.

Note

Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.

For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.

Procedure

4.6.1.3. About OADP update channels

When you install an OADP Operator, you choose an update channel. This channel determines which upgrades to the OADP Operator and to Velero you receive. You can switch channels at any time.

The following update channels are available:

  • The stable channel is now deprecated. The stable channel contains the patches (z-stream updates) of OADP ClusterServiceVersion for OADP.v1.1.z and older versions from OADP.v1.0.z.
  • The stable-1.0 channel is deprecated and is not supported.
  • The stable-1.1 channel is deprecated and is not supported.
  • The stable-1.2 channel is deprecated and is not supported.
  • The stable-1.3 channel contains OADP.v1.3.z, the most recent OADP 1.3 ClusterServiceVersion.
  • The stable-1.4 channel contains OADP.v1.4.z, the most recent OADP 1.4 ClusterServiceVersion.

For more information, see OpenShift Operator Life Cycles.

Which update channel is right for you?

  • The stable channel is now deprecated. If you are already using the stable channel, you will continue to get updates from OADP.v1.1.z.
  • Choose the stable-1.y update channel to install OADP 1.y and to continue receiving patches for it. If you choose this channel, you will receive all z-stream patches for version 1.y.z.

When must you switch update channels?

  • If you have OADP 1.y installed, and you want to receive patches only for that y-stream, you must switch from the stable update channel to the stable-1.y update channel. You will then receive all z-stream patches for version 1.y.z.
  • If you have OADP 1.0 installed, want to upgrade to OADP 1.1, and then receive patches only for OADP 1.1, you must switch from the stable-1.0 update channel to the stable-1.1 update channel. You will then receive all z-stream patches for version 1.1.z.
  • If you have OADP 1.y installed, with y greater than 0, and want to switch to OADP 1.0, you must uninstall your OADP Operator and then reinstall it using the stable-1.0 update channel. You will then receive all z-stream patches for version 1.0.z.
Note

You cannot switch from OADP 1.y to OADP 1.0 by switching update channels. You must uninstall the Operator and then reinstall it.

4.6.1.4. Installation of OADP on multiple namespaces

You can install OpenShift API for Data Protection (OADP) into multiple namespaces on the same cluster so that multiple project owners can manage their own OADP instance. This use case has been validated with File System Backup (FSB) and Container Storage Interface (CSI).

You install each instance of OADP as specified by the per-platform procedures contained in this document with the following additional requirements:

  • All deployments of OADP on the same cluster must be the same version, for example, 1.1.4. Installing different versions of OADP on the same cluster is not supported.
  • Each individual deployment of OADP must have a unique set of credentials and at least one BackupStorageLocation configuration. You can also use multiple BackupStorageLocation configurations within the same namespace.
  • By default, each OADP deployment has cluster-level access across namespaces. OpenShift Container Platform administrators need to review security and RBAC settings carefully and make any necessary changes to them to ensure that each OADP instance has the correct permissions.

Additional resources

4.6.1.5. Velero CPU and memory requirements based on collected data

The following recommendations are based on observations of performance made in the scale and performance lab. The backup and restore resources can be impacted by the type of plugin, the amount of resources required by that backup or restore, and the respective data contained in the persistent volumes (PVs) related to those resources.

4.6.1.5.1. CPU and memory requirement for configurations
Configuration types[1] Average usage[2] Large usageresourceTimeouts

CSI

Velero:

CPU- Request 200m, Limits 1000m

Memory - Request 256Mi, Limits 1024Mi

Velero:

CPU- Request 200m, Limits 2000m

Memory- Request 256Mi, Limits 2048Mi

N/A

Restic

[3] Restic:

CPU- Request 1000m, Limits 2000m

Memory - Request 16Gi, Limits 32Gi

[4] Restic:

CPU - Request 2000m, Limits 8000m

Memory - Request 16Gi, Limits 40Gi

900m

[5] Data Mover

N/A

N/A

10m - average usage

60m - large usage

  1. Average usage - use these settings for most usage situations.
  2. Large usage - use these settings for large usage situations, such as a large PV (500GB Usage), multiple namespaces (100+), or many pods within a single namespace (2000 pods+), and for optimal performance for backup and restore involving large datasets.
  3. Restic resource usage corresponds to the amount of data, and type of data. For example, many small files or large amounts of data can cause Restic to use large amounts of resources. The Velero documentation references 500m as a supplied default, for most of our testing we found a 200m request suitable with 1000m limit. As cited in the Velero documentation, exact CPU and memory usage is dependent on the scale of files and directories, in addition to environmental limitations.
  4. Increasing the CPU has a significant impact on improving backup and restore times.
  5. Data Mover - Data Mover default resourceTimeout is 10m. Our tests show that for restoring a large PV (500GB usage), it is required to increase the resourceTimeout to 60m.
Note

The resource requirements listed throughout the guide are for average usage only. For large usage, adjust the settings as described in the table above.

4.6.1.5.2. NodeAgent CPU for large usage

Testing shows that increasing NodeAgent CPU can significantly improve backup and restore times when using OpenShift API for Data Protection (OADP).

Important

It is not recommended to use Kopia without limits in production environments on nodes running production workloads due to Kopia’s aggressive consumption of resources. However, running Kopia with limits that are too low results in CPU limiting and slow backups and restore situations. Testing showed that running Kopia with 20 cores and 32 Gi memory supported backup and restore operations of over 100 GB of data, multiple namespaces, or over 2000 pods in a single namespace.

Testing detected no CPU limiting or memory saturation with these resource specifications.

You can set these limits in Ceph MDS pods by following the procedure in Changing the CPU and memory resources on the rook-ceph pods.

You need to add the following lines to the storage cluster Custom Resource (CR) to set the limits:

   resources:
     mds:
       limits:
         cpu: "3"
         memory: 128Gi
       requests:
         cpu: "3"
         memory: 8Gi

4.6.2. Installing the OADP Operator

You can install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.17 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.14.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click OperatorsOperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click OperatorsInstalled Operators to verify the installation.
4.6.2.1. OADP-Velero-OpenShift Container Platform version relationship
OADP versionVelero versionOpenShift Container Platform version

1.1.0

1.9

4.9 and later

1.1.1

1.9

4.9 and later

1.1.2

1.9

4.9 and later

1.1.3

1.9

4.9 and later

1.1.4

1.9

4.9 and later

1.1.5

1.9

4.9 and later

1.1.6

1.9

4.11 and later

1.1.7

1.9

4.11 and later

1.2.0

1.11

4.11 and later

1.2.1

1.11

4.11 and later

1.2.2

1.11

4.11 and later

1.2.3

1.11

4.11 and later

1.3.0

1.12

4.10 - 4.15

1.3.1

1.12

4.10 - 4.15

1.3.2

1.12

4.10 - 4.15

1.4.0

1.14

4.14 and later

1.4.1

1.14

4.14 and later

4.6.3. Configuring the OpenShift API for Data Protection with AWS S3 compatible storage

You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) S3 compatible storage by installing the OADP Operator. The Operator installs Velero 1.14.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You configure AWS for Velero, create a default Secret, and then install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager in disconnected environments for details.

4.6.3.1. Configuring Amazon Web Services

You configure Amazon Web Services (AWS) for the OpenShift API for Data Protection (OADP).

Prerequisites

  • You must have the AWS CLI installed.

Procedure

  1. Set the BUCKET variable:

    $ BUCKET=<your_bucket>
  2. Set the REGION variable:

    $ REGION=<your_region>
  3. Create an AWS S3 bucket:

    $ aws s3api create-bucket \
        --bucket $BUCKET \
        --region $REGION \
        --create-bucket-configuration LocationConstraint=$REGION 1
    1
    us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.
  4. Create an IAM user:

    $ aws iam create-user --user-name velero 1
    1
    If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.
  5. Create a velero-policy.json file:

    $ cat > velero-policy.json <<EOF
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeVolumes",
                    "ec2:DescribeSnapshots",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:DeleteObject",
                    "s3:PutObject",
                    "s3:AbortMultipartUpload",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation",
                    "s3:ListBucketMultipartUploads"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}"
                ]
            }
        ]
    }
    EOF
  6. Attach the policies to give the velero user the minimum necessary permissions:

    $ aws iam put-user-policy \
      --user-name velero \
      --policy-name velero \
      --policy-document file://velero-policy.json
  7. Create an access key for the velero user:

    $ aws iam create-access-key --user-name velero

    Example output

    {
      "AccessKey": {
            "UserName": "velero",
            "Status": "Active",
            "CreateDate": "2017-07-31T22:24:41.576Z",
            "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
            "AccessKeyId": <AWS_ACCESS_KEY_ID>
      }
    }

  8. Create a credentials-velero file:

    $ cat << EOF > ./credentials-velero
    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    EOF

    You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

4.6.3.2. About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

  • Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
  • Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.
Important

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

4.6.3.2.1. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.3.2.2. Creating profiles for different credentials

If your backup and snapshot locations use different credentials, you create separate profiles in the credentials-velero file.

Then, you create a Secret object and specify the profiles in the DataProtectionApplication custom resource (CR).

Procedure

  1. Create a credentials-velero file with separate profiles for the backup and snapshot locations, as in the following example:

    [backupStorage]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    
    [volumeSnapshot]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
  2. Create a Secret object with the credentials-velero file:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero 1
  3. Add the profiles to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
            config:
              region: us-east-1
              profile: "backupStorage"
            credential:
              key: cloud
              name: cloud-credentials
      snapshotLocations:
        - velero:
            provider: aws
            config:
              region: us-west-2
              profile: "volumeSnapshot"
4.6.3.2.3. Configuring the backup storage location using AWS

You can configure the AWS backup storage location (BSL) as shown in the following example procedure.

Prerequisites

  • You have created an object storage bucket using AWS.
  • You have installed the OADP Operator.

Procedure

  • Configure the BSL custom resource (CR) with values as applicable to your use case.

    Backup storage location

    apiVersion: oadp.openshift.io/v1alpha1
    kind: BackupStorageLocation
    metadata:
      name: default
      namespace: openshift-adp
    spec:
      provider: aws 1
      objectStorage:
        bucket: <bucket_name> 2
        prefix: <bucket_prefix> 3
      credential: 4
        key: cloud 5
        name: cloud-credentials 6
      config:
        region: <bucket_region> 7
        s3ForcePathStyle: "true" 8
        s3Url: <s3_url> 9
        publicUrl: <public_s3_url> 10
        serverSideEncryption: AES256 11
        kmsKeyId: "50..c-4da1-419f-a16e-ei...49f" 12
        customerKeyEncryptionFile: "/credentials/customer-key" 13
        signatureVersion: "1" 14
        profile: "default" 15
        insecureSkipTLSVerify: "true" 16
        enableSharedConfig: "true" 17
        tagging: "" 18
        checksumAlgorithm: "CRC32" 19

    1 1
    The name of the object store plugin. In this example, the plugin is aws. This field is required.
    2
    The name of the bucket in which to store backups. This field is required.
    3
    The prefix within the bucket in which to store backups. This field is optional.
    4
    The credentials for the backup storage location. You can set custom credentials. If custom credentials are not set, the default credentials' secret is used.
    5
    The key within the secret credentials' data.
    6
    The name of the secret containing the credentials.
    7
    The AWS region where the bucket is located. Optional if s3ForcePathStyle is false.
    8
    A boolean flag to decide whether to use path-style addressing instead of virtual hosted bucket addressing. Set to true if using a storage service such as MinIO or NooBaa. This is an optional field. The default value is false.
    9
    You can specify the AWS S3 URL here for explicitness. This field is primarily for storage services such as MinIO or NooBaa. This is an optional field.
    10
    This field is primarily used for storage services such as MinIO or NooBaa. This is an optional field.
    11
    The name of the server-side encryption algorithm to use for uploading objects, for example, AES256. This is an optional field.
    12
    Specify an AWS KMS key ID. You can format, as shown in the example, as an alias, such as alias/<KMS-key-alias-name>, or the full ARN to enable encryption of the backups stored in S3. Note that kmsKeyId cannot be used in with customerKeyEncryptionFile. This is an optional field.
    13
    Specify the file that has the SSE-C customer key to enable customer key encryption of the backups stored in S3. The file must contain a 32-byte string. The customerKeyEncryptionFile field points to a mounted secret within the velero container. Add the following key-value pair to the velero cloud-credentials secret: customer-key: <your_b64_encoded_32byte_string>. Note that the customerKeyEncryptionFile field cannot be used with the kmsKeyId field. The default value is an empty string (""), which means SSE-C is disabled. This is an optional field.
    14
    The version of the signature algorithm used to create signed URLs. You use signed URLs to download the backups, or fetch the logs. Valid values are 1 and 4. The default version is 4. This is an optional field.
    15
    The name of the AWS profile in the credentials file. The default value is default. This is an optional field.
    16
    Set the insecureSkipTLSVerify field to true if you do not want to verify the TLS certificate when connecting to the object store, for example, for self-signed certificates with MinIO. Setting to true is susceptible to man-in-the-middle attacks and is not recommended for production workloads. The default value is false. This is an optional field.
    17
    Set the enableSharedConfig field to true if you want to load the credentials file as a shared config file. The default value is false. This is an optional field.
    18
    Specify the tags to annotate the AWS S3 objects. Specify the tags in key-value pairs. The default value is an empty string (""). This is an optional field.
    19
    Specify the checksum algorithm to use for uploading objects to S3. The supported values are: CRC32, CRC32C, SHA1, and SHA256. If you set the field as an empty string (""), the checksum check will be skipped. The default value is CRC32. This is an optional field.
4.6.3.2.4. Creating an OADP SSE-C encryption key for additional data security

Amazon Web Services (AWS) S3 applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for every bucket in Amazon S3.

OpenShift API for Data Protection (OADP) encrypts data by using SSL/TLS, HTTPS, and the velero-repo-credentials secret when transferring the data from a cluster to storage. To protect backup data in case of lost or stolen AWS credentials, apply an additional layer of encryption.

The velero-plugin-for-aws plugin provides several additional encryption methods. You should review its configuration options and consider implementing additional encryption.

You can store your own encryption keys by using server-side encryption with customer-provided keys (SSE-C). This feature provides additional security if your AWS credentials become exposed.

Warning

Be sure to store cryptographic keys in a secure and safe manner. Encrypted data and backups cannot be recovered if you do not have the encryption key.

Prerequisites

  • To make OADP mount a secret that contains your SSE-C key to the Velero pod at /credentials, use the following default secret name for AWS: cloud-credentials, and leave at least one of the following labels empty:

Note

The following procedure contains an example of a spec:backupLocations block that does not specify credentials. This example would trigger an OADP secret mounting.

  • If you need the backup location to have credentials with a different name than cloud-credentials, you must add a snapshot location, such as the one in the following example, that does not contain a credential name. Because the example does not contain a credential name, the snapshot location will use cloud-credentials as its secret for taking snapshots.

Example snapshot location in a DPA without credentials specified

 snapshotLocations:
  - velero:
      config:
        profile: default
        region: <region>
      provider: aws
# ...

Procedure

  1. Create an SSE-C encryption key:

    1. Generate a random number and save it as a file named sse.key by running the following command:

      $ dd if=/dev/urandom bs=1 count=32 > sse.key
    2. Encode the sse.key by using Base64 and save the result as a file named sse_encoded.key by running the following command:

      $ cat sse.key | base64 > sse_encoded.key
    3. Link the file named sse_encoded.key to a new file named customer-key by running the following command:

      $ ln -s sse_encoded.key customer-key
  2. Create an OpenShift Container Platform secret:

    • If you are initially installing and configuring OADP, create the AWS credential and encryption key secret at the same time by running the following command:

      $ oc create secret generic cloud-credentials --namespace openshift-adp --from-file cloud=<path>/openshift_aws_credentials,customer-key=<path>/sse_encoded.key
    • If you are updating an existing installation, edit the values of the cloud-credential secret block of the DataProtectionApplication CR manifest, as in the following example:

      apiVersion: v1
      data:
        cloud: W2Rfa2V5X2lkPSJBS0lBVkJRWUIyRkQ0TlFHRFFPQiIKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5P<snip>rUE1mNWVSbTN5K2FpeWhUTUQyQk1WZHBOIgo=
        customer-key: v+<snip>TFIiq6aaXPbj8dhos=
      kind: Secret
      # ...
  3. Edit the value of the customerKeyEncryptionFile attribute in the backupLocations block of the DataProtectionApplication CR manifest, as in the following example:

    spec:
      backupLocations:
        - velero:
            config:
              customerKeyEncryptionFile: /credentials/customer-key
              profile: default
    # ...
    Warning

    You must restart the Velero pod to remount the secret credentials properly on an existing installation.

    The installation is complete, and you can back up and restore OpenShift Container Platform resources. The data saved in AWS S3 storage is encrypted with the new key, and you cannot download it from the AWS S3 console or API without the additional encryption key.

Verification

To verify that you cannot download the encrypted files without the inclusion of an additional key, create a test file, upload it, and then try to download it.

  1. Create a test file by running the following command:

    $ echo "encrypt me please" > test.txt
  2. Upload the test file by running the following command:

    $ aws s3api put-object \
      --bucket <bucket> \
      --key test.txt \
      --body test.txt \
      --sse-customer-key fileb://sse.key \
      --sse-customer-algorithm AES256
  3. Try to download the file. In either the Amazon web console or the terminal, run the following command:

    $ s3cmd get s3://<bucket>/test.txt test.txt

    The download fails because the file is encrypted with an additional key.

  4. Download the file with the additional encryption key by running the following command:

    $ aws s3api get-object \
        --bucket <bucket> \
        --key test.txt \
        --sse-customer-key fileb://sse.key \
        --sse-customer-algorithm AES256 \
        downloaded.txt
  5. Read the file contents by running the following command:

    $ cat downloaded.txt

    Example output

    encrypt me please

Additional resources

You can also download the file with the additional encryption key backed up with Velcro by running a different command. See Downloading a file with an SSE-C encryption key for files backed up by Velero.

4.6.3.2.4.1. Downloading a file with an SSE-C encryption key for files backed up by Velero

When you are verifying an SSE-C encryption key, you can also download the file with the additional encryption key for files that were backed up with Velcro.

Procedure

  • Download the file with the additional encryption key for files backed up by Velero by running the following command:
$ aws s3api get-object \
  --bucket <bucket> \
  --key velero/backups/mysql-persistent-customerkeyencryptionfile4/mysql-persistent-customerkeyencryptionfile4.tar.gz \
  --sse-customer-key fileb://sse.key \
  --sse-customer-algorithm AES256 \
  --debug \
  velero_download.tar.gz
4.6.3.3. Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

4.6.3.3.1. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

4.6.3.3.2. Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket>
              prefix: <prefix>
              caCert: <base64_encoded_cert_string> 1
            config:
              insecureSkipTLSVerify: "false" 2
    # ...
    1
    Specify the Base64-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.
4.6.3.3.2.1. Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

    1. To use an aliased Velero command, run the following command:

      $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    2. Check that the alias is working by running the following command:

      Example

      $ velero version
      Client:
      	Version: v1.12.1-OADP
      	Git commit: -
      Server:
      	Version: v1.12.1-OADP

    3. To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

      $ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')
      
      $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
      $ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
    4. To fetch the backup logs, run the following command:

      $ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>

      You can use these logs to view failures and warnings for the resources that you cannot back up.

    5. If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
    6. You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:

      $ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
      /tmp/your-cacert.txt

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

4.6.3.4. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.
  • If the backup and snapshot locations use different credentials, you must create a Secret with the default name, cloud-credentials, which contains separate profiles for the backup and snapshot location credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - openshift 2
            - aws
          resourceTimeout: 10m 3
        nodeAgent: 4
          enable: true 5
          uploaderType: kopia 6
          podConfig:
            nodeSelector: <node_selector> 7
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket_name> 8
              prefix: <prefix> 9
            config:
              region: <region>
              profile: "default"
              s3ForcePathStyle: "true" 10
              s3Url: <s3_url> 11
            credential:
              key: cloud
              name: cloud-credentials 12
      snapshotLocations: 13
        - name: default
          velero:
            provider: aws
            config:
              region: <region> 14
              profile: "default"
            credential:
              key: cloud
              name: cloud-credentials 15
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    The openshift plugin is mandatory.
    3
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    4
    The administrative agent that routes the administrative requests to servers.
    5
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    6
    Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    7
    Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
    8
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    9
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    10
    Specify whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.
    11
    Specify the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.
    12
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.
    13
    Specify a snapshot location, unless you use CSI snapshots or a File System Backup (FSB) to back up PVs.
    14
    The snapshot location must be in the same region as the PVs.
    15
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in the credentials-velero file.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.3.4.1. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.3.5. Configuring the backup storage location with a MD5 checksum algorithm

You can configure the Backup Storage Location (BSL) in the Data Protection Application (DPA) to use a MD5 checksum algorithm for both Amazon Simple Storage Service (Amazon S3) and S3-compatible storage providers. The checksum algorithm calculates the checksum for uploading and downloading objects to Amazon S3. You can use one of the following options to set the checksumAlgorithm field in the spec.backupLocations.velero.config.checksumAlgorithm section of the DPA.

  • CRC32
  • CRC32C
  • SHA1
  • SHA256
Note

You can also set the checksumAlgorithm field to an empty value to skip the MD5 checksum check.

If you do not set a value for the checksumAlgorithm field, then the default value is set to CRC32.

Prerequisites

  • You have installed the OADP Operator.
  • You have configured Amazon S3, or S3-compatible object storage as a backup location.

Procedure

  • Configure the BSL in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
      - name: default
        velero:
          config:
            checksumAlgorithm: "" 1
            insecureSkipTLSVerify: "true"
            profile: "default"
            region: <bucket_region>
            s3ForcePathStyle: "true"
            s3Url: <bucket_url>
          credential:
            key: cloud
            name: cloud-credentials
          default: true
          objectStorage:
            bucket: <bucket_name>
            prefix: velero
          provider: aws
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - aws
          - csi

    1
    Specify the checksumAlgorithm. In this example, the checksumAlgorithm field is set to an empty value. You can select an option from the following list: CRC32, CRC32C, SHA1, SHA256.
Important

If you are using Noobaa as the object storage provider, and you do not set the spec.backupLocations.velero.config.checksumAlgorithm field in the DPA, an empty value of checksumAlgorithm is added to the BSL configuration.

The empty value is only added for BSLs that are created using the DPA. This value is not added if you create the BSL by using any other method.

4.6.3.6. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.3.7. Configuring the DPA with more than one BSL

You can configure the DPA with more than one BSL and specify the credentials provided by the cloud provider.

Prerequisites

  • You must install the OADP Operator.
  • You must create the secrets by using the credentials provided by the cloud provider.

Procedure

  1. Configure the DPA with more than one BSL. See the following example.

    Example DPA

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    #...
    backupLocations:
      - name: aws 1
        velero:
          provider: aws
          default: true 2
          objectStorage:
            bucket: <bucket_name> 3
            prefix: <prefix> 4
          config:
            region: <region_name> 5
            profile: "default"
          credential:
            key: cloud
            name: cloud-credentials 6
      - name: odf 7
        velero:
          provider: aws
          default: false
          objectStorage:
            bucket: <bucket_name>
            prefix: <prefix>
          config:
            profile: "default"
            region: <region_name>
            s3Url: <url> 8
            insecureSkipTLSVerify: "true"
            s3ForcePathStyle: "true"
          credential:
            key: cloud
            name: <custom_secret_name_odf> 9
    #...

    1
    Specify a name for the first BSL.
    2
    This parameter indicates that this BSL is the default BSL. If a BSL is not set in the Backup CR, the default BSL is used. You can set only one BSL as the default.
    3
    Specify the bucket name.
    4
    Specify a prefix for Velero backups; for example, velero.
    5
    Specify the AWS region for the bucket.
    6
    Specify the name of the default Secret object that you created.
    7
    Specify a name for the second BSL.
    8
    Specify the URL of the S3 endpoint.
    9
    Specify the correct name for the Secret; for example, custom_secret_name_odf. If you do not specify a Secret name, the default name is used.
  2. Specify the BSL to be used in the backup CR. See the following example.

    Example backup CR

    apiVersion: velero.io/v1
    kind: Backup
    # ...
    spec:
      includedNamespaces:
      - <namespace> 1
      storageLocation: <backup_storage_location> 2
      defaultVolumesToFsBackup: true

    1
    Specify the namespace to back up.
    2
    Specify the storage location.
4.6.3.7.1. Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    ...
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - csi 1
    1
    Add the csi default plugin.
4.6.3.7.2. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.4. Configuring the OpenShift API for Data Protection with IBM Cloud

You install the OpenShift API for Data Protection (OADP) Operator on an IBM Cloud cluster to back up and restore applications on the cluster. You configure IBM Cloud Object Storage (COS) to store the backups.

4.6.4.1. Configuring the COS instance

You create an IBM Cloud Object Storage (COS) instance to store the OADP backup data. After you create the COS instance, configure the HMAC service credentials.

Prerequisites

  • You have an IBM Cloud Platform account.
  • You installed the IBM Cloud CLI.
  • You are logged in to IBM Cloud.

Procedure

  1. Install the IBM Cloud Object Storage (COS) plugin by running the following command:

    $ ibmcloud plugin install cos -f
  2. Set a bucket name by running the following command:

    $ BUCKET=<bucket_name>
  3. Set a bucket region by running the following command:

    $ REGION=<bucket_region> 1
    1
    Specify the bucket region, for example, eu-gb.
  4. Create a resource group by running the following command:

    $ ibmcloud resource group-create <resource_group_name>
  5. Set the target resource group by running the following command:

    $ ibmcloud target -g <resource_group_name>
  6. Verify that the target resource group is correctly set by running the following command:

    $ ibmcloud target

    Example output

    API endpoint:     https://cloud.ibm.com
    Region:
    User:             test-user
    Account:          Test Account (fb6......e95) <-> 2...122
    Resource group:   Default

    In the example output, the resource group is set to Default.

  7. Set a resource group name by running the following command:

    $ RESOURCE_GROUP=<resource_group> 1
    1
    Specify the resource group name, for example, "default".
  8. Create an IBM Cloud service-instance resource by running the following command:

    $ ibmcloud resource service-instance-create \
    <service_instance_name> \1
    <service_name> \2
    <service_plan> \3
    <region_name> 4
    1
    Specify a name for the service-instance resource.
    2
    Specify the service name. Alternatively, you can specify a service ID.
    3
    Specify the service plan for your IBM Cloud account.
    4
    Specify the region name.

    Example command

    $ ibmcloud resource service-instance-create test-service-instance cloud-object-storage \ 1
    standard \
    global \
    -d premium-global-deployment 2

    1
    The service name is cloud-object-storage.
    2
    The -d flag specifies the deployment name.
  9. Extract the service instance ID by running the following command:

    $ SERVICE_INSTANCE_ID=$(ibmcloud resource service-instance test-service-instance --output json | jq -r '.[0].id')
  10. Create a COS bucket by running the following command:

    $ ibmcloud cos bucket-create \//
    --bucket $BUCKET \//
    --ibm-service-instance-id $SERVICE_INSTANCE_ID \//
    --region $REGION

    Variables such as $BUCKET, $SERVICE_INSTANCE_ID, and $REGION are replaced by the values you set previously.

  11. Create HMAC credentials by running the following command.

    $ ibmcloud resource service-key-create test-key Writer --instance-name test-service-instance --parameters {\"HMAC\":true}
  12. Extract the access key ID and the secret access key from the HMAC credentials and save them in the credentials-velero file. You can use the credentials-velero file to create a secret for the backup storage location. Run the following command:

    $ cat > credentials-velero << __EOF__
    [default]
    aws_access_key_id=$(ibmcloud resource service-key test-key -o json  | jq -r '.[0].credentials.cos_hmac_keys.access_key_id')
    aws_secret_access_key=$(ibmcloud resource service-key test-key -o json  | jq -r '.[0].credentials.cos_hmac_keys.secret_access_key')
    __EOF__
4.6.4.2. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.4.3. Creating secrets for different credentials

If your backup and snapshot locations use different credentials, you must create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.
  2. Create a Secret for the snapshot location with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.
  4. Create a Secret for the backup location with a custom name:

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - velero:
            provider: <provider>
            default: true
            credential:
              key: cloud
              name: <custom_secret> 1
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
    1
    Backup location Secret with custom name.
4.6.4.4. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      namespace: openshift-adp
      name: <dpa_name>
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - aws
          - csi
      backupLocations:
        - velero:
            provider: aws 1
            default: true
            objectStorage:
              bucket: <bucket_name> 2
              prefix: velero
            config:
              insecureSkipTLSVerify: 'true'
              profile: default
              region: <region_name> 3
              s3ForcePathStyle: 'true'
              s3Url: <s3_url> 4
            credential:
              key: cloud
              name: cloud-credentials 5
    1
    The provider is aws when you use IBM Cloud as a backup storage location.
    2
    Specify the IBM Cloud Object Storage (COS) bucket name.
    3
    Specify the COS region name, for example, eu-gb.
    4
    Specify the S3 URL of the COS bucket. For example, http://s3.eu-gb.cloud-object-storage.appdomain.cloud. Here, eu-gb is the region name. Replace the region name according to your bucket region.
    5
    Defines the name of the secret you created by using the access key and the secret access key from the HMAC credentials.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.4.5. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

4.6.4.6. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.4.7. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.4.8. Configuring the DPA with more than one BSL

You can configure the DPA with more than one BSL and specify the credentials provided by the cloud provider.

Prerequisites

  • You must install the OADP Operator.
  • You must create the secrets by using the credentials provided by the cloud provider.

Procedure

  1. Configure the DPA with more than one BSL. See the following example.

    Example DPA

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    #...
    backupLocations:
      - name: aws 1
        velero:
          provider: aws
          default: true 2
          objectStorage:
            bucket: <bucket_name> 3
            prefix: <prefix> 4
          config:
            region: <region_name> 5
            profile: "default"
          credential:
            key: cloud
            name: cloud-credentials 6
      - name: odf 7
        velero:
          provider: aws
          default: false
          objectStorage:
            bucket: <bucket_name>
            prefix: <prefix>
          config:
            profile: "default"
            region: <region_name>
            s3Url: <url> 8
            insecureSkipTLSVerify: "true"
            s3ForcePathStyle: "true"
          credential:
            key: cloud
            name: <custom_secret_name_odf> 9
    #...

    1
    Specify a name for the first BSL.
    2
    This parameter indicates that this BSL is the default BSL. If a BSL is not set in the Backup CR, the default BSL is used. You can set only one BSL as the default.
    3
    Specify the bucket name.
    4
    Specify a prefix for Velero backups; for example, velero.
    5
    Specify the AWS region for the bucket.
    6
    Specify the name of the default Secret object that you created.
    7
    Specify a name for the second BSL.
    8
    Specify the URL of the S3 endpoint.
    9
    Specify the correct name for the Secret; for example, custom_secret_name_odf. If you do not specify a Secret name, the default name is used.
  2. Specify the BSL to be used in the backup CR. See the following example.

    Example backup CR

    apiVersion: velero.io/v1
    kind: Backup
    # ...
    spec:
      includedNamespaces:
      - <namespace> 1
      storageLocation: <backup_storage_location> 2
      defaultVolumesToFsBackup: true

    1
    Specify the namespace to back up.
    2
    Specify the storage location.
4.6.4.9. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.5. Configuring the OpenShift API for Data Protection with Microsoft Azure

You install the OpenShift API for Data Protection (OADP) with Microsoft Azure by installing the OADP Operator. The Operator installs Velero 1.14.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You configure Azure for Velero, create a default Secret, and then install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager in disconnected environments for details.

4.6.5.1. Configuring Microsoft Azure

You configure Microsoft Azure for OpenShift API for Data Protection (OADP).

Prerequisites

Tools that use Azure services should always have restricted permissions to make sure that Azure resources are safe. Therefore, instead of having applications sign in as a fully privileged user, Azure offers service principals. An Azure service principal is a name that can be used with applications, hosted services, or automated tools.

This identity is used for access to resources.

  • Create a service principal
  • Sign in using a service principal and password
  • Sign in using a service principal and certificate
  • Manage service principal roles
  • Create an Azure resource using a service principal
  • Reset service principal credentials

For more details, see Create an Azure service principal with Azure CLI.

4.6.5.2. About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

  • Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
  • Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.
Important

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

4.6.5.2.1. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials-azure.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.5.2.2. Creating secrets for different credentials

If your backup and snapshot locations use different credentials, you must create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials-azure. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.
  2. Create a Secret for the snapshot location with the default name:

    $ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.
  4. Create a Secret for the backup location with a custom name:

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - velero:
            config:
              resourceGroup: <azure_resource_group>
              storageAccount: <azure_storage_account_id>
              subscriptionId: <azure_subscription_id>
              storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
            credential:
              key: cloud
              name: <custom_secret> 1
            provider: azure
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
      snapshotLocations:
        - velero:
            config:
              resourceGroup: <azure_resource_group>
              subscriptionId: <azure_subscription_id>
              incremental: "true"
            provider: azure
    1
    Backup location Secret with custom name.
4.6.5.3. Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

4.6.5.3.1. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

4.6.5.3.2. Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket>
              prefix: <prefix>
              caCert: <base64_encoded_cert_string> 1
            config:
              insecureSkipTLSVerify: "false" 2
    # ...
    1
    Specify the Base64-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.
4.6.5.3.2.1. Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

    1. To use an aliased Velero command, run the following command:

      $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    2. Check that the alias is working by running the following command:

      Example

      $ velero version
      Client:
      	Version: v1.12.1-OADP
      	Git commit: -
      Server:
      	Version: v1.12.1-OADP

    3. To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

      $ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')
      
      $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
      $ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
    4. To fetch the backup logs, run the following command:

      $ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>

      You can use these logs to view failures and warnings for the resources that you cannot back up.

    5. If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
    6. You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:

      $ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
      /tmp/your-cacert.txt

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

4.6.5.4. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials-azure.
  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.
    • Secret with another custom name for the snapshot location. You add this Secret to the DataProtectionApplication CR.
    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - azure
            - openshift 2
          resourceTimeout: 10m 3
        nodeAgent: 4
          enable: true 5
          uploaderType: kopia 6
          podConfig:
            nodeSelector: <node_selector> 7
      backupLocations:
        - velero:
            config:
              resourceGroup: <azure_resource_group> 8
              storageAccount: <azure_storage_account_id> 9
              subscriptionId: <azure_subscription_id> 10
              storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
            credential:
              key: cloud
              name: cloud-credentials-azure  11
            provider: azure
            default: true
            objectStorage:
              bucket: <bucket_name> 12
              prefix: <prefix> 13
      snapshotLocations: 14
        - velero:
            config:
              resourceGroup: <azure_resource_group>
              subscriptionId: <azure_subscription_id>
              incremental: "true"
            name: default
            provider: azure
            credential:
              key: cloud
              name: cloud-credentials-azure 15
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    The openshift plugin is mandatory.
    3
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    4
    The administrative agent that routes the administrative requests to servers.
    5
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    6
    Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    7
    Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
    8
    Specify the Azure resource group.
    9
    Specify the Azure storage account ID.
    10
    Specify the Azure subscription ID.
    11
    If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.
    12
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    13
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    14
    You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
    15
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.5.5. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.5.5.1. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.5.5.2. Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    ...
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - csi 1
    1
    Add the csi default plugin.
4.6.5.5.3. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.6. Configuring the OpenShift API for Data Protection with Google Cloud Platform

You install the OpenShift API for Data Protection (OADP) with Google Cloud Platform (GCP) by installing the OADP Operator. The Operator installs Velero 1.14.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You configure GCP for Velero, create a default Secret, and then install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager in disconnected environments for details.

4.6.6.1. Configuring Google Cloud Platform

You configure Google Cloud Platform (GCP) for the OpenShift API for Data Protection (OADP).

Prerequisites

Procedure

  1. Log in to GCP:

    $ gcloud auth login
  2. Set the BUCKET variable:

    $ BUCKET=<bucket> 1
    1
    Specify your bucket name.
  3. Create the storage bucket:

    $ gsutil mb gs://$BUCKET/
  4. Set the PROJECT_ID variable to your active project:

    $ PROJECT_ID=$(gcloud config get-value project)
  5. Create a service account:

    $ gcloud iam service-accounts create velero \
        --display-name "Velero service account"
  6. List your service accounts:

    $ gcloud iam service-accounts list
  7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

    $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
        --filter="displayName:Velero service account" \
        --format 'value(email)')
  8. Attach the policies to give the velero user the minimum necessary permissions:

    $ ROLE_PERMISSIONS=(
        compute.disks.get
        compute.disks.create
        compute.disks.createSnapshot
        compute.snapshots.get
        compute.snapshots.create
        compute.snapshots.useReadOnly
        compute.snapshots.delete
        compute.zones.get
        storage.objects.create
        storage.objects.delete
        storage.objects.get
        storage.objects.list
        iam.serviceAccounts.signBlob
    )
  9. Create the velero.server custom role:

    $ gcloud iam roles create velero.server \
        --project $PROJECT_ID \
        --title "Velero Server" \
        --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
  10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server
  11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}
  12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

4.6.6.2. About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

  • Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
  • Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.
Important

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

4.6.6.2.1. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials-gcp.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.6.2.2. Creating secrets for different credentials

If your backup and snapshot locations use different credentials, you must create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials-gcp. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.
  2. Create a Secret for the snapshot location with the default name:

    $ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.
  4. Create a Secret for the backup location with a custom name:

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - velero:
            provider: gcp
            default: true
            credential:
              key: cloud
              name: <custom_secret> 1
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
      snapshotLocations:
        - velero:
            provider: gcp
            default: true
            config:
              project: <project>
              snapshotLocation: us-west1
    1
    Backup location Secret with custom name.
4.6.6.3. Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

4.6.6.3.1. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

4.6.6.3.2. Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket>
              prefix: <prefix>
              caCert: <base64_encoded_cert_string> 1
            config:
              insecureSkipTLSVerify: "false" 2
    # ...
    1
    Specify the Base64-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.
4.6.6.3.2.1. Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

    1. To use an aliased Velero command, run the following command:

      $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    2. Check that the alias is working by running the following command:

      Example

      $ velero version
      Client:
      	Version: v1.12.1-OADP
      	Git commit: -
      Server:
      	Version: v1.12.1-OADP

    3. To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

      $ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')
      
      $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
      $ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
    4. To fetch the backup logs, run the following command:

      $ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>

      You can use these logs to view failures and warnings for the resources that you cannot back up.

    5. If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
    6. You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:

      $ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
      /tmp/your-cacert.txt

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

4.6.6.4. Google workload identity federation cloud authentication

Applications running outside Google Cloud use service account keys, such as usernames and passwords, to gain access to Google Cloud resources. These service account keys might become a security risk if they are not properly managed.

With Google’s workload identity federation, you can use Identity and Access Management (IAM) to offer IAM roles, including the ability to impersonate service accounts, to external identities. This eliminates the maintenance and security risks associated with service account keys.

Workload identity federation handles encrypting and decrypting certificates, extracting user attributes, and validation. Identity federation externalizes authentication, passing it over to Security Token Services (STS), and reduces the demands on individual developers. Authorization and controlling access to resources remain the responsibility of the application.

Note

Google workload identity federation is available for OADP 1.3.x and later.

When backing up volumes, OADP on GCP with Google workload identity federation authentication only supports CSI snapshots.

OADP on GCP with Google workload identity federation authentication does not support Volume Snapshot Locations (VSL) backups. For more details, see Google workload identity federation known issues.

If you do not use Google workload identity federation cloud authentication, continue to Installing the Data Protection Application.

Prerequisites

  • You have installed a cluster in manual mode with GCP Workload Identity configured.
  • You have access to the Cloud Credential Operator utility (ccoctl) and to the associated workload identity pool.

Procedure

  1. Create an oadp-credrequest directory by running the following command:

    $ mkdir -p oadp-credrequest
  2. Create a CredentialsRequest.yaml file as following:

    echo 'apiVersion: cloudcredential.openshift.io/v1
    kind: CredentialsRequest
    metadata:
      name: oadp-operator-credentials
      namespace: openshift-cloud-credential-operator
    spec:
      providerSpec:
        apiVersion: cloudcredential.openshift.io/v1
        kind: GCPProviderSpec
        permissions:
        - compute.disks.get
        - compute.disks.create
        - compute.disks.createSnapshot
        - compute.snapshots.get
        - compute.snapshots.create
        - compute.snapshots.useReadOnly
        - compute.snapshots.delete
        - compute.zones.get
        - storage.objects.create
        - storage.objects.delete
        - storage.objects.get
        - storage.objects.list
        - iam.serviceAccounts.signBlob
        skipServiceCheck: true
      secretRef:
        name: cloud-credentials-gcp
        namespace: <OPERATOR_INSTALL_NS>
      serviceAccountNames:
      - velero
    ' > oadp-credrequest/credrequest.yaml
  3. Use the ccoctl utility to process the CredentialsRequest objects in the oadp-credrequest directory by running the following command:

    $ ccoctl gcp create-service-accounts \
        --name=<name> \
        --project=<gcp_project_id> \
        --credentials-requests-dir=oadp-credrequest \
        --workload-identity-pool=<pool_id> \
        --workload-identity-provider=<provider_id>

    The manifests/openshift-adp-cloud-credentials-gcp-credentials.yaml file is now available to use in the following steps.

  4. Create a namespace by running the following command:

    $ oc create namespace <OPERATOR_INSTALL_NS>
  5. Apply the credentials to the namespace by running the following command:

    $ oc apply -f manifests/openshift-adp-cloud-credentials-gcp-credentials.yaml
4.6.6.4.1. Google workload identity federation known issues
  • Volume Snapshot Location (VSL) backups finish with a PartiallyFailed phase when GCP workload identity federation is configured. Google workload identity federation authentication does not support VSL backups.
4.6.6.5. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials-gcp.
  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.
    • Secret with another custom name for the snapshot location. You add this Secret to the DataProtectionApplication CR.
    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: <OPERATOR_INSTALL_NS> 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - gcp
            - openshift 2
          resourceTimeout: 10m 3
        nodeAgent: 4
          enable: true 5
          uploaderType: kopia 6
          podConfig:
            nodeSelector: <node_selector> 7
      backupLocations:
        - velero:
            provider: gcp
            default: true
            credential:
              key: cloud 8
              name: cloud-credentials-gcp 9
            objectStorage:
              bucket: <bucket_name> 10
              prefix: <prefix> 11
      snapshotLocations: 12
        - velero:
            provider: gcp
            default: true
            config:
              project: <project>
              snapshotLocation: us-west1 13
            credential:
              key: cloud
              name: cloud-credentials-gcp 14
      backupImages: true 15
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    The openshift plugin is mandatory.
    3
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    4
    The administrative agent that routes the administrative requests to servers.
    5
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    6
    Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    7
    Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
    8
    Secret key that contains credentials. For Google workload identity federation cloud authentication use service_account.json.
    9
    Secret name that contains credentials. If you do not specify this value, the default name, cloud-credentials-gcp, is used.
    10
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    11
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    12
    Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.
    13
    The snapshot location must be in the same region as the PVs.
    14
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.
    15
    Google workload identity federation supports internal image backup. Set this field to false if you do not want to use image backup.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.6.6. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.6.6.1. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.6.6.2. Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    ...
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - csi 1
    1
    Add the csi default plugin.
4.6.6.6.3. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.7. Configuring the OpenShift API for Data Protection with Multicloud Object Gateway

You install the OpenShift API for Data Protection (OADP) with Multicloud Object Gateway (MCG) by installing the OADP Operator. The Operator installs Velero 1.14.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You configure Multicloud Object Gateway as a backup location. MCG is a component of OpenShift Data Foundation. You configure MCG as a backup location in the DataProtectionApplication custom resource (CR).

Important

The CloudStorage API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You create a Secret for the backup location and then you install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager in disconnected environments.

4.6.7.1. Retrieving Multicloud Object Gateway credentials

You must retrieve the Multicloud Object Gateway (MCG) credentials, which you need to create a Secret custom resource (CR) for the OpenShift API for Data Protection (OADP).

Note

Although the MCG Operator is deprecated, the MCG plugin is still available for OpenShift Data Foundation. To download the plugin, browse to Download Red Hat OpenShift Data Foundation and download the appropriate MCG plugin for your operating system.

Prerequisites

Procedure

  1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.
  2. Create a credentials-velero file:

    $ cat << EOF > ./credentials-velero
    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    EOF

    You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

4.6.7.2. About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

  • Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
  • Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.
Important

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

4.6.7.2.1. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.7.2.2. Creating secrets for different credentials

If your backup and snapshot locations use different credentials, you must create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.
  2. Create a Secret for the snapshot location with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.
  4. Create a Secret for the backup location with a custom name:

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - velero:
            config:
              profile: "default"
              region: <region_name> 1
              s3Url: <url>
              insecureSkipTLSVerify: "true"
              s3ForcePathStyle: "true"
            provider: aws
            default: true
            credential:
              key: cloud
              name:  <custom_secret> 2
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
    1
    Specify the region, following the naming convention of the documentation of your object storage server.
    2
    Backup location Secret with custom name.
4.6.7.3. Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

4.6.7.3.1. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

4.6.7.3.2. Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket>
              prefix: <prefix>
              caCert: <base64_encoded_cert_string> 1
            config:
              insecureSkipTLSVerify: "false" 2
    # ...
    1
    Specify the Base64-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.
4.6.7.3.2.1. Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

    1. To use an aliased Velero command, run the following command:

      $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    2. Check that the alias is working by running the following command:

      Example

      $ velero version
      Client:
      	Version: v1.12.1-OADP
      	Git commit: -
      Server:
      	Version: v1.12.1-OADP

    3. To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

      $ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')
      
      $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
      $ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
    4. To fetch the backup logs, run the following command:

      $ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>

      You can use these logs to view failures and warnings for the resources that you cannot back up.

    5. If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
    6. You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:

      $ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
      /tmp/your-cacert.txt

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

4.6.7.4. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.
  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.
    • Secret with another custom name for the snapshot location. You add this Secret to the DataProtectionApplication CR.
    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - aws 2
            - openshift 3
          resourceTimeout: 10m 4
        nodeAgent: 5
          enable: true 6
          uploaderType: kopia 7
          podConfig:
            nodeSelector: <node_selector> 8
      backupLocations:
        - velero:
            config:
              profile: "default"
              region: <region_name> 9
              s3Url: <url> 10
              insecureSkipTLSVerify: "true"
              s3ForcePathStyle: "true"
            provider: aws
            default: true
            credential:
              key: cloud
              name: cloud-credentials 11
            objectStorage:
              bucket: <bucket_name> 12
              prefix: <prefix> 13
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is aws. For Azure and GCP object stores, the azure or gcp plugin is required.
    3
    The openshift plugin is mandatory.
    4
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    5
    The administrative agent that routes the administrative requests to servers.
    6
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    7
    Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    8
    Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
    9
    Specify the region, following the naming convention of the documentation of your object storage server.
    10
    Specify the URL of the S3 endpoint.
    11
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.
    12
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    13
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.7.5. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.7.5.1. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.7.5.2. Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    ...
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - csi 1
    1
    Add the csi default plugin.
4.6.7.5.3. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.8. Configuring the OpenShift API for Data Protection with OpenShift Data Foundation

You install the OpenShift API for Data Protection (OADP) with OpenShift Data Foundation by installing the OADP Operator and configuring a backup location and a snapshot location. Then, you install the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You can configure Multicloud Object Gateway or any AWS S3-compatible object storage as a backup location.

Important

The CloudStorage API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You create a Secret for the backup location and then you install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager in disconnected environments.

4.6.8.1. About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

  • Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
  • Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.
Important

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

4.6.8.1.1. Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials, unless your backup storage provider has a default plugin, such as aws, azure, or gcp. In that case, the default name is specified in the provider-specific OADP installation procedure.

Note

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

  • Your object storage and cloud storage, if any, must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.6.8.1.2. Creating secrets for different credentials

If your backup and snapshot locations use different credentials, you must create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.
  2. Create a Secret for the snapshot location with the default name:

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.
  4. Create a Secret for the backup location with a custom name:

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
    ...
      backupLocations:
        - velero:
            provider: <provider>
            default: true
            credential:
              key: cloud
              name: <custom_secret> 1
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>
    1
    Backup location Secret with custom name.
4.6.8.2. Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

4.6.8.2.1. Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      configuration:
        velero:
          podConfig:
            nodeSelector: <node_selector> 1
            resourceAllocations: 2
              limits:
                cpu: "1"
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 256Mi
    1
    Specify the node selector to be supplied to Velero podSpec.
    2
    The resourceAllocations listed are for average usage.
Note

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

4.6.8.2.1.1. Adjusting Ceph CPU and memory requirements based on collected data

The following recommendations are based on observations of performance made in the scale and performance lab. The changes are specifically related to Red Hat OpenShift Data Foundation (ODF). If working with ODF, consult the appropriate tuning guides for official recommendations.

4.6.8.2.1.1.1. CPU and memory requirement for configurations

Backup and restore operations require large amounts of CephFS PersistentVolumes (PVs). To avoid Ceph MDS pods restarting with an out-of-memory (OOM) error, the following configuration is suggested:

Configuration typesRequestMax limit

CPU

Request changed to 3

Max limit to 3

Memory

Request changed to 8 Gi

Max limit to 128 Gi

4.6.8.2.2. Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
    spec:
    # ...
      backupLocations:
        - name: default
          velero:
            provider: aws
            default: true
            objectStorage:
              bucket: <bucket>
              prefix: <prefix>
              caCert: <base64_encoded_cert_string> 1
            config:
              insecureSkipTLSVerify: "false" 2
    # ...
    1
    Specify the Base64-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.
4.6.8.2.2.1. Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

    1. To use an aliased Velero command, run the following command:

      $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    2. Check that the alias is working by running the following command:

      Example

      $ velero version
      Client:
      	Version: v1.12.1-OADP
      	Git commit: -
      Server:
      	Version: v1.12.1-OADP

    3. To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

      $ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')
      
      $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
      $ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
    4. To fetch the backup logs, run the following command:

      $ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>

      You can use these logs to view failures and warnings for the resources that you cannot back up.

    5. If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
    6. You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:

      $ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
      /tmp/your-cacert.txt

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

4.6.8.3. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.
  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.
    • Secret with another custom name for the snapshot location. You add this Secret to the DataProtectionApplication CR.
    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - aws 2
            - kubevirt 3
            - csi 4
            - openshift 5
          resourceTimeout: 10m 6
        nodeAgent: 7
          enable: true 8
          uploaderType: kopia 9
          podConfig:
            nodeSelector: <node_selector> 10
      backupLocations:
        - velero:
            provider: gcp 11
            default: true
            credential:
              key: cloud
              name: <default_secret> 12
            objectStorage:
              bucket: <bucket_name> 13
              prefix: <prefix> 14
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is aws. For Azure and GCP object stores, the azure or gcp plugin is required.
    3
    Optional: The kubevirt plugin is used with OpenShift Virtualization.
    4
    Specify the csi default plugin if you use CSI snapshots to back up PVs. The csi plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location.
    5
    The openshift plugin is mandatory.
    6
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    7
    The administrative agent that routes the administrative requests to servers.
    8
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    9
    Enter kopia or restic as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    10
    Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
    11
    Specify the backup provider.
    12
    Specify the correct default name for the Secret, for example, cloud-credentials-gcp, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify a Secret name, the default name is used.
    13
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    14
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

4.6.8.4. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1
    Specify the client-burst value. In this example, the client-burst field is set to 500.
    2
    Specify the client-qps value. In this example, the client-qps field is set to 300.
4.6.8.4.1. Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""
4.6.8.4.2. Creating an Object Bucket Claim for disaster recovery on OpenShift Data Foundation

If you use cluster storage for your Multicloud Object Gateway (MCG) bucket backupStorageLocation on OpenShift Data Foundation, create an Object Bucket Claim (OBC) using the OpenShift Web Console.

Warning

Failure to configure an Object Bucket Claim (OBC) might lead to backups not being available.

Note

Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.

For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.

Procedure

4.6.8.4.3. Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    ...
    spec:
      configuration:
        velero:
          defaultPlugins:
          - openshift
          - csi 1
    1
    Add the csi default plugin.
4.6.8.4.4. Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

  1. To disable the nodeAgent, set the enable flag to false. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: false  1
        uploaderType: kopia
    # ...

    1
    Disables the node agent.
  2. To enable the nodeAgent, set the enable flag to true. See the following example:

    Example DataProtectionApplication CR

    # ...
    configuration:
      nodeAgent:
        enable: true  1
        uploaderType: kopia
    # ...

    1
    Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

4.6.9. Configuring the OpenShift API for Data Protection with OpenShift Virtualization

You can install the OpenShift API for Data Protection (OADP) with OpenShift Virtualization by installing the OADP Operator and configuring a backup location. Then, you can install the Data Protection Application.

Back up and restore virtual machines by using the OpenShift API for Data Protection.

Note

OpenShift API for Data Protection with OpenShift Virtualization supports the following backup and restore storage options:

  • Container Storage Interface (CSI) backups
  • Container Storage Interface (CSI) backups with DataMover

The following storage options are excluded:

  • File system backup and restore
  • Volume snapshot backups and restores

For more information, see Backing up applications with File System Backup: Kopia or Restic.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager in disconnected environments for details.

4.6.9.1. Installing and configuring OADP with OpenShift Virtualization

As a cluster administrator, you install OADP by installing the OADP Operator.

The latest version of the OADP Operator installs Velero 1.14.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

Procedure

  1. Install the OADP Operator according to the instructions for your storage provider.
  2. Install the Data Protection Application (DPA) with the kubevirt and openshift OADP plugins.
  3. Back up virtual machines by creating a Backup custom resource (CR).

    Warning

    Red Hat support is limited to only the following options:

    • CSI backups
    • CSI backups with DataMover.

You restore the Backup CR by creating a Restore CR.

4.6.9.2. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click OperatorsInstalled Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp 1
    spec:
      configuration:
        velero:
          defaultPlugins:
            - kubevirt 2
            - gcp 3
            - csi 4
            - openshift 5
          resourceTimeout: 10m 6
        nodeAgent: 7
          enable: true 8
          uploaderType: kopia 9
          podConfig:
            nodeSelector: <node_selector> 10
      backupLocations:
        - velero:
            provider: gcp 11
            default: true
            credential:
              key: cloud
              name: <default_secret> 12
            objectStorage:
              bucket: <bucket_name> 13
              prefix: <prefix> 14
    1
    The default namespace for OADP is openshift-adp. The namespace is a variable and is configurable.
    2
    The kubevirt plugin is mandatory for OpenShift Virtualization.
    3
    Specify the plugin for the backup provider, for example, gcp, if it exists.
    4
    The csi plugin is mandatory for backing up PVs with CSI snapshots. The csi plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location.
    5
    The openshift plugin is mandatory.
    6
    Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
    7
    The administrative agent that routes the administrative requests to servers.
    8
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    9
    Enter kopia as your uploader to use the Built-in DataMover. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    10
    Specify the nodes on which Kopia are available. By default, Kopia runs on all nodes.
    11
    Specify the backup provider.
    12
    Specify the correct default name for the Secret, for example, cloud-credentials-gcp, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify a Secret name, the default name is used.
    13
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    14
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s

  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

    Example output

    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupStorageLocation -n openshift-adp

    Example output

    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

Warning

If you run a backup of a Microsoft Windows virtual machine (VM) immediately after the VM reboots, the backup might fail with a PartiallyFailed error. This is because, immediately after a VM boots, the Microsoft Windows Volume Shadow Copy Service (VSS) and Guest Agent (GA) service are not ready. The VSS and GA service being unready causes the backup to fail. In such a case, retry the backup a few minutes after the VM boots.

4.6.9.3. Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

  • You have installed the OADP Operator.

Procedure

  • Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

    Example Data Protection Application

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: test-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - name: default
          velero:
            config:
              insecureSkipTLSVerify: "true"
              profile: "default"
              region: <bucket_region>
              s3ForcePathStyle: "true"
              s3Url: <bucket_url>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: aws
      configuration:
        nodeAgent:
          enable: true
          uploaderType: restic
        velero:
          client-burst: 500 1
          client-qps: 300 2
          defaultPlugins:
            - openshift
            - aws
            - kubevirt

    1