Backup and restore
Backing up and restoring your OpenShift Container Platform cluster
Abstract
Chapter 1. Backup and restore
1.1. Control plane backup and restore operations
As a cluster administrator, you might need to stop an OpenShift Container Platform cluster for a period and restart it later. Some reasons for restarting a cluster are that you need to perform maintenance on a cluster or want to reduce resource costs. In OpenShift Container Platform, you can perform a graceful shutdown of a cluster so that you can easily restart the cluster later.
You must back up etcd data before shutting down a cluster; etcd is the key-value store for OpenShift Container Platform, which persists the state of all resource objects. An etcd backup plays a crucial role in disaster recovery. In OpenShift Container Platform, you can also replace an unhealthy etcd member.
When you want to get your cluster running again, restart the cluster gracefully.
A cluster’s certificates expire one year after the installation date. You can shut down a cluster and expect it to restart gracefully while the certificates are still valid. Although the cluster automatically retrieves the expired control plane certificates, you must still approve the certificate signing requests (CSRs).
You might run into several situations where OpenShift Container Platform does not work as expected, such as:
- You have a cluster that is not functional after the restart because of unexpected conditions, such as node failure or network connectivity issues.
- You have deleted something critical in the cluster by mistake.
- You have lost the majority of your control plane hosts, leading to etcd quorum loss.
You can always recover from a disaster situation by restoring your cluster to its previous state using the saved etcd snapshots.
Additional resources
1.2. Application backup and restore operations
As a cluster administrator, you can back up and restore applications running on OpenShift Container Platform by using the OpenShift API for Data Protection (OADP).
OADP backs up and restores Kubernetes resources and internal images, at the granularity of a namespace, by using the version of Velero that is appropriate for the version of OADP you install, according to the table in Downloading the Velero CLI tool. OADP backs up and restores persistent volumes (PVs) by using snapshots or Restic. For details, see OADP features.
1.2.1. OADP requirements
OADP has the following requirements:
-
You must be logged in as a user with a
cluster-admin
role. You must have object storage for storing backups, such as one of the following storage types:
- OpenShift Data Foundation
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- S3-compatible object storage
- IBM Cloud® Object Storage S3
If you want to use CSI backup on OCP 4.11 and later, install OADP 1.1.x.
OADP 1.0.x does not support CSI backup on OCP 4.11 and later. OADP 1.0.x includes Velero 1.7.x and expects the API group snapshot.storage.k8s.io/v1beta1
, which is not present on OCP 4.11 and later.
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
To back up PVs with snapshots, you must have cloud storage that has a native snapshot API or supports Container Storage Interface (CSI) snapshots, such as the following providers:
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- CSI snapshot-enabled cloud storage, such as Ceph RBD or Ceph FS
If you do not want to back up PVs by using snapshots, you can use Restic, which is installed by the OADP Operator by default.
1.2.2. Backing up and restoring applications
You back up applications by creating a Backup
custom resource (CR). See Creating a Backup CR. You can configure the following backup options:
- Creating backup hooks to run commands before or after the backup operation
- Scheduling backups
- Backing up applications with File System Backup: Kopia or Restic
-
You restore application backups by creating a
Restore
(CR). See Creating a Restore CR. - You can configure restore hooks to run commands in init containers or in the application container during the restore operation.
Chapter 2. Shutting down the cluster gracefully
This document describes the process to gracefully shut down your cluster. You might need to temporarily shut down your cluster for maintenance reasons, or to save on resource costs.
2.1. Prerequisites
Take an etcd backup prior to shutting down the cluster.
ImportantIt is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues when restarting the cluster.
For example, the following conditions can cause the restarted cluster to malfunction:
- etcd data corruption during shutdown
- Node failure due to hardware
- Network connectivity issues
If your cluster fails to recover, follow the steps to restore to a previous cluster state.
2.2. Shutting down the cluster
You can shut down your cluster in a graceful manner so that it can be restarted at a later date.
You can shut down a cluster until a year from the installation date and expect it to restart gracefully. After a year from the installation date, the cluster certificates expire.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role. - You have taken an etcd backup.
Procedure
If you are shutting the cluster down for an extended period, determine the date on which certificates expire and run the following command:
$ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}'
Example output
2022-08-05T14:37:50Zuser@user:~ $ 1
- 1
- To ensure that the cluster can restart gracefully, plan to restart it on or before the specified date. As the cluster restarts, the process might require you to manually approve the pending certificate signing requests (CSRs) to recover kubelet certificates.
Mark all the nodes in the cluster as unschedulable. You can do this from your cloud provider’s web console, or by running the following loop:
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm cordon ${node} ; done
Example output
ci-ln-mgdnf4b-72292-n547t-master-0 node/ci-ln-mgdnf4b-72292-n547t-master-0 cordoned ci-ln-mgdnf4b-72292-n547t-master-1 node/ci-ln-mgdnf4b-72292-n547t-master-1 cordoned ci-ln-mgdnf4b-72292-n547t-master-2 node/ci-ln-mgdnf4b-72292-n547t-master-2 cordoned ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl node/ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl cordoned ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k node/ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k cordoned ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn node/ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn cordoned
Evacuate the pods using the following method:
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm drain ${node} --delete-emptydir-data --ignore-daemonsets=true --timeout=15s --force ; done
Shut down all of the nodes in the cluster. You can do this from the web console for your cloud provider web console, or by running the following loop. Shutting down the nodes by using one of these methods allows pods to terminate gracefully, which reduces the chance for data corruption.
NoteEnsure that the control plane node with the API VIP assigned is the last node processed in the loop. Otherwise, the shutdown command fails.
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 1; done 1
- 1
-h 1
indicates how long, in minutes, this process lasts before the control plane nodes are shut down. For large-scale clusters with 10 nodes or more, set to-h 10
or longer to make sure all the compute nodes have time to shut down first.
Example output
Starting pod/ip-10-0-130-169us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Shutdown scheduled for Mon 2021-09-13 09:36:17 UTC, use 'shutdown -c' to cancel. Removing debug pod ... Starting pod/ip-10-0-150-116us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Shutdown scheduled for Mon 2021-09-13 09:36:29 UTC, use 'shutdown -c' to cancel.
NoteIt is not necessary to drain control plane nodes of the standard pods that ship with OpenShift Container Platform prior to shutdown. Cluster administrators are responsible for ensuring a clean restart of their own workloads after the cluster is restarted. If you drained control plane nodes prior to shutdown because of custom workloads, you must mark the control plane nodes as schedulable before the cluster will be functional again after restart.
Shut off any cluster dependencies that are no longer needed, such as external storage or an LDAP server. Be sure to consult your vendor’s documentation before doing so.
ImportantIf you deployed your cluster on a cloud-provider platform, do not shut down, suspend, or delete the associated cloud resources. If you delete the cloud resources of a suspended virtual machine, OpenShift Container Platform might not restore successfully.
2.3. Additional resources
Chapter 3. Restarting the cluster gracefully
This document describes the process to restart your cluster after a graceful shutdown.
Even though the cluster is expected to be functional after the restart, the cluster might not recover due to unexpected conditions, for example:
- etcd data corruption during shutdown
- Node failure due to hardware
- Network connectivity issues
If your cluster fails to recover, follow the steps to restore to a previous cluster state.
3.1. Prerequisites
- You have gracefully shut down your cluster.
3.2. Restarting the cluster
You can restart your cluster after it has been shut down gracefully.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role. - This procedure assumes that you gracefully shut down the cluster.
Procedure
- Power on any cluster dependencies, such as external storage or an LDAP server.
Start all cluster machines.
Use the appropriate method for your cloud environment to start the machines, for example, from your cloud provider’s web console.
Wait approximately 10 minutes before continuing to check the status of control plane nodes.
Verify that all control plane nodes are ready.
$ oc get nodes -l node-role.kubernetes.io/master
The control plane nodes are ready if the status is
Ready
, as shown in the following output:NAME STATUS ROLES AGE VERSION ip-10-0-168-251.ec2.internal Ready master 75m v1.28.5 ip-10-0-170-223.ec2.internal Ready master 75m v1.28.5 ip-10-0-211-16.ec2.internal Ready master 75m v1.28.5
If the control plane nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.
Get the list of current CSRs:
$ oc get csr
Review the details of a CSR to verify that it is valid:
$ oc describe csr <csr_name> 1
- 1
<csr_name>
is the name of a CSR from the list of current CSRs.
Approve each valid CSR:
$ oc adm certificate approve <csr_name>
After the control plane nodes are ready, verify that all worker nodes are ready.
$ oc get nodes -l node-role.kubernetes.io/worker
The worker nodes are ready if the status is
Ready
, as shown in the following output:NAME STATUS ROLES AGE VERSION ip-10-0-179-95.ec2.internal Ready worker 64m v1.28.5 ip-10-0-182-134.ec2.internal Ready worker 64m v1.28.5 ip-10-0-250-100.ec2.internal Ready worker 64m v1.28.5
If the worker nodes are not ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.
Get the list of current CSRs:
$ oc get csr
Review the details of a CSR to verify that it is valid:
$ oc describe csr <csr_name> 1
- 1
<csr_name>
is the name of a CSR from the list of current CSRs.
Approve each valid CSR:
$ oc adm certificate approve <csr_name>
Verify that the cluster started properly.
Check that there are no degraded cluster Operators.
$ oc get clusteroperators
Check that there are no cluster Operators with the
DEGRADED
condition set toTrue
.NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.15.0 True False False 59m cloud-credential 4.15.0 True False False 85m cluster-autoscaler 4.15.0 True False False 73m config-operator 4.15.0 True False False 73m console 4.15.0 True False False 62m csi-snapshot-controller 4.15.0 True False False 66m dns 4.15.0 True False False 76m etcd 4.15.0 True False False 76m ...
Check that all nodes are in the
Ready
state:$ oc get nodes
Check that the status for all nodes is
Ready
.NAME STATUS ROLES AGE VERSION ip-10-0-168-251.ec2.internal Ready master 82m v1.28.5 ip-10-0-170-223.ec2.internal Ready master 82m v1.28.5 ip-10-0-179-95.ec2.internal Ready worker 70m v1.28.5 ip-10-0-182-134.ec2.internal Ready worker 70m v1.28.5 ip-10-0-211-16.ec2.internal Ready master 82m v1.28.5 ip-10-0-250-100.ec2.internal Ready worker 69m v1.28.5
If the cluster did not start properly, you might need to restore your cluster using an etcd backup.
After the control plane and worker nodes are ready, mark all the nodes in the cluster as schedulable. Run the following command:
for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm uncordon ${node} ; done
Additional resources
- See Restoring to a previous cluster state for how to use an etcd backup to restore if your cluster failed to recover after restarting.
Chapter 4. OADP Application backup and restore
4.1. Introduction to OpenShift API for Data Protection
The OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs).
However, OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.
4.1.1. OpenShift API for Data Protection APIs
OpenShift API for Data Protection (OADP) provides APIs that enable multiple approaches to customizing backups and preventing the inclusion of unnecessary or inappropriate resources.
OADP provides the following APIs:
Additional resources
4.2. OADP release notes
4.2.1. OADP 1.4 release notes
The release notes for OpenShift API for Data Protection (OADP) describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.
4.2.1.1. OADP 1.4.1 release notes
The OpenShift API for Data Protection (OADP) 1.4.1 release notes lists new features, resolved issues and bugs, and known issues.
4.2.1.1.1. New features
New DPA fields to update client qps and burst
You can now change Velero Server Kubernetes API queries per second and burst values by using the new Data Protection Application (DPA) fields. The new DPA fields are spec.configuration.velero.client-qps
and spec.configuration.velero.client-burst
, which both default to 100. OADP-4076
Enabling non-default algorithms with Kopia
With this update, you can now configure the hash, encryption, and splitter algorithms in Kopia to select non-default options to optimize performance for different backup workloads.
To configure these algorithms, set the env
variable of a velero
pod in the podConfig
section of the DataProtectionApplication (DPA) configuration. If this variable is not set, or an unsupported algorithm is chosen, Kopia will default to its standard algorithms. OADP-4640
4.2.1.1.2. Resolved issues
Restoring a backup without pods is now successful
Previously, restoring a backup without pods and having StorageClass VolumeBindingMode
set as WaitForFirstConsumer
, resulted in the PartiallyFailed
status with an error: fail to patch dynamic PV, err: context deadline exceeded
. With this update, patching dynamic PV is skipped and restoring a backup is successful without any PartiallyFailed
status. OADP-4231
PodVolumeBackup CR now displays correct message
Previously, the PodVolumeBackup
custom resource (CR) generated an incorrect message, which was: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
. With this update, the message produced is now:
found a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed".
Overriding imagePullPolicy is now possible with DPA
Previously, OADP set the imagePullPolicy
parameter to Always
for all images. With this update, OADP checks if each image contains sha256
or sha512
digest, then it sets imagePullPolicy
to IfNotPresent
; otherwise imagePullPolicy
is set to Always
. You can now override this policy by using the new spec.containerImagePullPolicy
DPA field. OADP-4172
OADP Velero can now retry updating the restore status if initial update fails
Previously, OADP Velero failed to update the restored CR status. This left the status at InProgress
indefinitely. Components which relied on the backup and restore CR status to determine the completion would fail. With this update, the restore CR status for a restore correctly proceeds to the Completed
or Failed
status. OADP-3227
Restoring BuildConfig Build from a different cluster is successful without any errors
Previously, when performing a restore of the BuildConfig
Build resource from a different cluster, the application generated an error on TLS verification to the internal image registry. The resulting error was failed to verify certificate: x509: certificate signed by unknown authority
error. With this update, the restore of the BuildConfig
build resources to a different cluster can proceed successfully without generating the failed to verify certificate
error. OADP-4692
Restoring an empty PVC is successful
Previously, downloading data failed while restoring an empty persistent volume claim (PVC). It failed with the following error:
data path restore failed: Failed to run kopia restore: Unable to load snapshot : snapshot not found
With this update, the downloading of data proceeds to correct conclusion when restoring an empty PVC and the error message is not generated. OADP-3106
There is no Velero memory leak in CSI and DataMover plugins
Previously, a Velero memory leak was caused by using the CSI and DataMover plugins. When the backup ended, the Velero plugin instance was not deleted and the memory leak consumed memory until an Out of Memory
(OOM) condition was generated in the Velero pod. With this update, there is no resulting Velero memory leak when using the CSI and DataMover plugins. OADP-4448
Post-hook operation does not start before the related PVs are released
Previously, due to the asynchronous nature of the Data Mover operation, a post-hook might be attempted before the Data Mover persistent volume claim (PVC) releases the persistent volumes (PVs) of the related pods. This problem would cause the backup to fail with a PartiallyFailed
status. With this update, the post-hook operation is not started until the related PVs are released by the Data Mover PVC, eliminating the PartiallyFailed
backup status. OADP-3140
Deploying a DPA works as expected in namespaces with more than 37 characters
When you install the OADP Operator in a namespace with more than 37 characters to create a new DPA, labeling the "cloud-credentials" Secret fails and the DPA reports the following error:
The generated label name is too long.
With this update, creating a DPA does not fail in namespaces with more than 37 characters in the name. OADP-3960
Restore is successfully completed by overriding the timeout error
Previously, in a large scale environment, the restore operation would result in a Partiallyfailed
status with the error: fail to patch dynamic PV, err: context deadline exceeded
. With this update, the resourceTimeout
Velero server argument is used to override this timeout error resulting in a successful restore. OADP-4344
For a complete list of all issues resolved in this release, see the list of OADP 1.4.1 resolved issues in Jira.
4.2.1.1.3. Known issues
Cassandra application pods enter into the CrashLoopBackoff
status after restoring OADP
After OADP restores, the Cassandra application pods might enter CrashLoopBackoff
status. To work around this problem, delete the StatefulSet
pods that are returning the error CrashLoopBackoff
state after restoring OADP. The StatefulSet
controller then recreates these pods and it runs normally. OADP-4407
Deployment referencing ImageStream is not restored properly leading to corrupted pod and volume contents
During a File System Backup (FSB) restore operation, a Deployment
resource referencing an ImageStream
is not restored properly. The restored pod that runs the FSB, and the postHook
is terminated prematurely.
During the restore operation, the OpenShift Container Platform controller updates the spec.template.spec.containers[0].image
field in the Deployment
resource with an updated ImageStreamTag
hash. The update triggers the rollout of a new pod, terminating the pod on which velero
runs the FSB along with the post-hook. For more information about image stream trigger, see Triggering updates on image stream changes.
The workaround for this behavior is a two-step restore process:
Perform a restore excluding the
Deployment
resources, for example:$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --exclude-resources=deployment.apps
Once the first restore is successful, perform a second restore by including these resources, for example:
$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --include-resources=deployment.apps
4.2.1.2. OADP 1.4.0 release notes
The OpenShift API for Data Protection (OADP) 1.4.0 release notes lists resolved issues and known issues.
4.2.1.2.1. Resolved issues
Restore works correctly in OpenShift Container Platform 4.16
Previously, while restoring the deleted application namespace, the restore operation partially failed with the resource name may not be empty
error in OpenShift Container Platform 4.16. With this update, restore works as expected in OpenShift Container Platform 4.16. OADP-4075
Data Mover backups work properly in the OpenShift Container Platform 4.16 cluster
Previously, Velero was using the earlier version of SDK where the Spec.SourceVolumeMode
field did not exist. As a consequence, Data Mover backups failed in the OpenShift Container Platform 4.16 cluster on the external snapshotter with version 4.2. With this update, external snapshotter is upgraded to version 7.0 and later. As a result, backups do not fail in the OpenShift Container Platform 4.16 cluster. OADP-3922
For a complete list of all issues resolved in this release, see the list of OADP 1.4.0 resolved issues in Jira.
4.2.1.2.2. Known issues
Backup fails when checksumAlgorithm is not set for MCG
While performing a backup of any application with Noobaa as the backup location, if the checksumAlgorithm
configuration parameter is not set, backup fails. To fix this problem, if you do not provide a value for checksumAlgorithm
in the Backup Storage Location (BSL) configuration, an empty value is added. The empty value is only added for BSLs that are created using Data Protection Application (DPA) custom resource (CR), and this value is not added if BSLs are created using any other method. OADP-4274
For a complete list of all known issues in this release, see the list of OADP 1.4.0 known issues in Jira.
4.2.1.2.3. Upgrade notes
Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OpenShift API for Data Protection (OADP) 1.1 to 1.3, upgrade first to 1.2, and then to 1.3.
4.2.1.2.3.1. Changes from OADP 1.3 to 1.4
The Velero server has been updated from version 1.12 to 1.14. Note that there are no changes in the Data Protection Application (DPA).
This changes the following:
-
The
velero-plugin-for-csi
code is now available in the Velero code, which means aninit
container is no longer required for the plugin. - Velero changed client Burst and QPS defaults from 30 and 20 to 100 and 100, respectively.
The
velero-plugin-for-aws
plugin updated default value of thespec.config.checksumAlgorithm
field inBackupStorageLocation
objects (BSLs) from""
(no checksum calculation) to theCRC32
algorithm. For more information, see Velero plugins for AWS Backup Storage Location. The checksum algorithm types are known to work only with AWS. Several S3 providers require themd5sum
to be disabled by setting the checksum algorithm to""
. Confirmmd5sum
algorithm support and configuration with your storage provider.In OADP 1.4, the default value for BSLs created within DPA for this configuration is
""
. This default value means that themd5sum
is not checked, which is consistent with OADP 1.3. For BSLs created within DPA, update it by using thespec.backupLocations[].velero.config.checksumAlgorithm
field in the DPA. If your BSLs are created outside DPA, you can update this configuration by usingspec.config.checksumAlgorithm
in the BSLs.
4.2.1.2.3.2. Backing up the DPA configuration
You must back up your current DataProtectionApplication
(DPA) configuration.
Procedure
Save your current DPA configuration by running the following command:
Example command
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
4.2.1.2.3.3. Upgrading the OADP Operator
Use the following procedure when upgrading the OpenShift API for Data Protection (OADP) Operator.
Procedure
-
Change your subscription channel for the OADP Operator from
stable-1.3
tostable-1.4
. - Wait for the Operator and containers to update and restart.
Additional resources
4.2.1.2.4. Converting DPA to the new version
To upgrade from OADP 1.3 to 1.4, no Data Protection Application (DPA) changes are required.
4.2.1.2.5. Verifying the upgrade
Verify the installation by following steps from the Verifying the upgrade section.
4.2.2. OADP 1.3 release notes
The release notes for OpenShift API for Data Protection (OADP) 1.3 describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.
4.2.2.1. OADP 1.3.3 release notes
The OpenShift API for Data Protection (OADP) 1.3.3 release notes list resolved issues and known issues.
4.2.2.1.1. Resolved issues
OADP fails when its namespace name is longer than 37 characters
When installing the OADP Operator in a namespace with more than 37 characters and when creating a new DPA, labeling the cloud-credentials
secret fails. With this release, the issue has been fixed. OADP-4211
OADP image PullPolicy set to Always
In previous versions of OADP, the image PullPolicy of the adp-controller-manager and Velero pods was set to Always
. This was problematic in edge scenarios where there could be limited network bandwidth to the registry, resulting in slow recovery time following a pod restart. In OADP 1.3.3, the image PullPolicy of the openshift-adp-controller-manager
and Velero pods is set to IfNotPresent
.
The list of security fixes that are included in this release is documented in the RHSA-2024:4982 advisory.
For a complete list of all issues resolved in this release, see the list of OADP 1.3.3 resolved issues in Jira.
4.2.2.1.2. Known issues
Cassandra application pods enter into the CrashLoopBackoff
status after restoring OADP
After OADP restores, the Cassandra application pods might enter in the CrashLoopBackoff
status. To work around this problem, delete the StatefulSet
pods that are returning an error or the CrashLoopBackoff
state after restoring OADP. The StatefulSet
controller recreates these pods and it runs normally.
4.2.2.2. OADP 1.3.2 release notes
The OpenShift API for Data Protection (OADP) 1.3.2 release notes list resolved issues and known issues.
4.2.2.2.1. Resolved issues
DPA fails to reconcile if a valid custom secret is used for BSL
DPA fails to reconcile if a valid custom secret is used for Backup Storage Location (BSL), but the default secret is missing. The workaround is to create the required default cloud-credentials
initially. When the custom secret is re-created, it can be used and checked for its existence.
CVE-2023-45290: oadp-velero-container
: Golang net/http
: Memory exhaustion in Request.ParseMultipartForm
A flaw was found in the net/http
Golang standard library package, which impacts previous versions of OADP. When parsing a multipart
form, either explicitly with Request.ParseMultipartForm
or implicitly with Request.FormValue
, Request.PostFormValue
, or Request.FormFile
, limits on the total size of the parsed form are not applied to the memory consumed while reading a single form line. This permits a maliciously crafted input containing long lines to cause the allocation of arbitrarily large amounts of memory, potentially leading to memory exhaustion. This flaw has been resolved in OADP 1.3.2.
For more details, see CVE-2023-45290.
CVE-2023-45289: oadp-velero-container
: Golang net/http/cookiejar
: Incorrect forwarding of sensitive headers and cookies on HTTP redirect
A flaw was found in the net/http/cookiejar
Golang standard library package, which impacts previous versions of OADP. When following an HTTP redirect to a domain that is not a subdomain match or exact match of the initial domain, an http.Client
does not forward sensitive headers such as Authorization
or Cookie
. A maliciously crafted HTTP redirect could cause sensitive headers to be unexpectedly forwarded. This flaw has been resolved in OADP 1.3.2.
For more details, see CVE-2023-45289.
CVE-2024-24783: oadp-velero-container
: Golang crypto/x509
: Verify panics on certificates with an unknown public key algorithm
A flaw was found in the crypto/x509
Golang standard library package, which impacts previous versions of OADP. Verifying a certificate chain that contains a certificate with an unknown public key algorithm causes Certificate.Verify
to panic. This affects all crypto/tls
clients and servers that set Config.ClientAuth
to VerifyClientCertIfGiven
or RequireAndVerifyClientCert
. The default behavior is for TLS servers to not verify client certificates. This flaw has been resolved in OADP 1.3.2.
For more details, see CVE-2024-24783.
CVE-2024-24784: oadp-velero-plugin-container
: Golang net/mail
: Comments in display names are incorrectly handled
A flaw was found in the net/mail
Golang standard library package, which impacts previous versions of OADP. The ParseAddressList
function incorrectly handles comments, text in parentheses, and display names. Because this is a misalignment with conforming address parsers, it can result in different trust decisions being made by programs using different parsers. This flaw has been resolved in OADP 1.3.2.
For more details, see CVE-2024-24784.
CVE-2024-24785: oadp-velero-container
: Golang: html/template: errors returned from MarshalJSON
methods may break template escaping
A flaw was found in the html/template
Golang standard library package, which impacts previous versions of OADP. If errors returned from MarshalJSON
methods contain user-controlled data, they may be used to break the contextual auto-escaping behavior of the HTML/template package, allowing subsequent actions to inject unexpected content into the templates. This flaw has been resolved in OADP 1.3.2.
For more details, see CVE-2024-24785.
For a complete list of all issues resolved in this release, see the list of OADP 1.3.2 resolved issues in Jira.
4.2.2.2.2. Known issues
Cassandra application pods enter into the CrashLoopBackoff
status after restoring OADP
After OADP restores, the Cassandra application pods might enter in the CrashLoopBackoff
status. To work around this problem, delete the StatefulSet
pods that are returning an error or the CrashLoopBackoff
state after restoring OADP. The StatefulSet
controller recreates these pods and it runs normally.
4.2.2.3. OADP 1.3.1 release notes
The OpenShift API for Data Protection (OADP) 1.3.1 release notes lists new features and resolved issues.
4.2.2.3.1. New features
OADP 1.3.0 Data Mover is now fully supported
The OADP built-in Data Mover, introduced in OADP 1.3.0 as a Technology Preview, is now fully supported for both containerized and virtual machine workloads.
4.2.2.3.2. Resolved issues
IBM Cloud(R) Object Storage is now supported as a backup storage provider
IBM Cloud® Object Storage is one of the AWS S3 compatible backup storage providers, which was unsupported previously. With this update, IBM Cloud® Object Storage is now supported as an AWS S3 compatible backup storage provider.
OADP operator now correctly reports the missing region error
Previously, when you specified profile:default
without specifying the region
in the AWS Backup Storage Location (BSL) configuration, the OADP operator failed to report the missing region
error on the Data Protection Application (DPA) custom resource (CR). This update corrects validation of DPA BSL specification for AWS. As a result, the OADP Operator reports the missing region
error.
Custom labels are not removed from the openshift-adp namespace
Previously, the openshift-adp-controller-manager
pod would reset the labels attached to the openshift-adp
namespace. This caused synchronization issues for applications requiring custom labels such as Argo CD, leading to improper functionality. With this update, this issue is fixed and custom labels are not removed from the openshift-adp
namespace.
OADP must-gather image collects CRDs
Previously, the OADP must-gather
image did not collect the custom resource definitions (CRDs) shipped by OADP. Consequently, you could not use the omg
tool to extract data in the support shell. With this fix, the must-gather
image now collects CRDs shipped by OADP and can use the omg
tool to extract data.
Garbage collection has the correct description for the default frequency value
Previously, the garbage-collection-frequency
field had a wrong description for the default frequency value. With this update, garbage-collection-frequency
has a correct value of one hour for the gc-controller
reconciliation default frequency.
FIPS Mode flag is available in OperatorHub
By setting the fips-compliant
flag to true
, the FIPS mode flag is now added to the OADP Operator listing in OperatorHub. This feature was enabled in OADP 1.3.0 but did not show up in the Red Hat Container catalog as being FIPS enabled.
CSI plugin does not panic with a nil pointer when csiSnapshotTimeout is set to a short duration
Previously, when the csiSnapshotTimeout
parameter was set to a short duration, the CSI plugin encountered the following error: plugin panicked: runtime error: invalid memory address or nil pointer dereference
.
With this fix, the backup fails with the following error: Timed out awaiting reconciliation of volumesnapshot
.
For a complete list of all issues resolved in this release, see the list of OADP 1.3.1 resolved issues in Jira.
4.2.2.3.3. Known issues
Backup and storage restrictions for Single-node OpenShift clusters deployed on IBM Power(R) and IBM Z(R) platforms
Review the following backup and storage related restrictions for Single-node OpenShift clusters that are deployed on IBM Power® and IBM Z® platforms:
- Storage
- Only NFS storage is currently compatible with single-node OpenShift clusters deployed on IBM Power® and IBM Z® platforms.
- Backup
-
Only the backing up applications with File System Backup such as
kopia
andrestic
are supported for backup and restore operations.
Cassandra application pods enter in the CrashLoopBackoff status after restoring OADP
After OADP restores, the Cassandra application pods might enter in the CrashLoopBackoff
status. To work around this problem, delete the StatefulSet
pods with any error or the CrashLoopBackoff
state after restoring OADP. The StatefulSet
controller recreates these pods and it runs normally.
4.2.2.4. OADP 1.3.0 release notes
The OpenShift API for Data Protection (OADP) 1.3.0 release notes lists new features, resolved issues and bugs, and known issues.
4.2.2.4.1. New features
Velero built-in DataMover is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
OADP 1.3 includes a built-in Data Mover that you can use to move Container Storage Interface (CSI) volume snapshots to a remote object store. The built-in Data Mover allows you to restore stateful applications from the remote object store if a failure, accidental deletion, or corruption of the cluster occurs. It uses Kopia as the uploader mechanism to read the snapshot data and to write to the Unified Repository.
Backing up applications with File System Backup: Kopia or Restic
Velero’s File System Backup (FSB) supports two backup libraries: the Restic path and the Kopia path.
Velero allows users to select between the two paths.
For backup, specify the path during the installation through the uploader-type
flag. The valid value is either restic
or kopia
. This field defaults to kopia
if the value is not specified. The selection cannot be changed after the installation.
GCP Cloud authentication
Google Cloud Platform (GCP) authentication enables you to use short-lived Google credentials.
GCP with Workload Identity Federation enables you to use Identity and Access Management (IAM) to grant external identities IAM roles, including the ability to impersonate service accounts. This eliminates the maintenance and security risks associated with service account keys.
AWS ROSA STS authentication
You can use OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS (ROSA) clusters to backup and restore application data.
ROSA provides seamless integration with a wide range of AWS compute, database, analytics, machine learning, networking, mobile, and other services to speed up the building and delivering of differentiating experiences to your customers.
You can subscribe to the service directly from your AWS account.
After the clusters are created, you can operate your clusters by using the OpenShift web console. The ROSA service also uses OpenShift APIs and command-line interface (CLI) tools.
4.2.2.4.2. Resolved issues
ACM applications were removed and re-created on managed clusters after restore
Applications on managed clusters were deleted and re-created upon restore activation. OpenShift API for Data Protection (OADP 1.2) backup and restore process is faster than the older versions. The OADP performance change caused this behavior when restoring ACM resources. Therefore, some resources were restored before other resources, which caused the removal of the applications from managed clusters. OADP-2686
Restic restore was partially failing due to Pod Security standard
During interoperability testing, OpenShift Container Platform 4.14 had the pod Security mode set to enforce
, which caused the pod to be denied. This was caused due to the restore order. The pod was getting created before the security context constraints (SCC) resource, since the pod violated the podSecurity
standard, it denied the pod. When setting the restore priority field on the Velero server, restore is successful. OADP-2688
Possible pod volume backup failure if Velero is installed in several namespaces
There was a regression in Pod Volume Backup (PVB) functionality when Velero was installed in several namespaces. The PVB controller was not properly limiting itself to PVBs in its own namespace. OADP-2308
OADP Velero plugins returning "received EOF, stopping recv loop" message
In OADP, Velero plugins were started as separate processes. When the Velero operation completes, either successfully or not, they exit. Therefore, if you see a received EOF, stopping recv loop
messages in debug logs, it does not mean an error occurred, it means that a plugin operation has completed. OADP-2176
CVE-2023-39325 Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)
In previous releases of OADP, the HTTP/2 protocol was susceptible to a denial of service attack because request cancellation could reset multiple streams quickly. The server had to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This resulted in a denial of service due to server resource consumption.
For more information, see CVE-2023-39325 (Rapid Reset Attack)
For a complete list of all issues resolved in this release, see the list of OADP 1.3.0 resolved issues in Jira.
4.2.2.4.3. Known issues
CSI plugin errors on nil pointer when csiSnapshotTimeout is set to a short duration
The CSI plugin errors on nil pointer when csiSnapshotTimeout
is set to a short duration. Sometimes it succeeds to complete the snapshot within a short duration, but often it panics with the backup PartiallyFailed
with the following error: plugin panicked: runtime error: invalid memory address or nil pointer dereference
.
Backup is marked as PartiallyFailed when volumeSnapshotContent CR has an error
If any of the VolumeSnapshotContent
CRs have an error related to removing the VolumeSnapshotBeingCreated
annotation, it moves the backup to the WaitingForPluginOperationsPartiallyFailed
phase. OADP-2871
Performance issues when restoring 30,000 resources for the first time
When restoring 30,000 resources for the first time, without an existing-resource-policy, it takes twice as long to restore them, than it takes during the second and third try with an existing-resource-policy set to update
. OADP-3071
Post restore hooks might start running before Datadownload operation has released the related PV
Due to the asynchronous nature of the Data Mover operation, a post-hook might be attempted before the related pods persistent volumes (PVs) are released by the Data Mover persistent volume claim (PVC).
GCP-Workload Identity Federation VSL backup PartiallyFailed
VSL backup PartiallyFailed
when GCP workload identity is configured on GCP.
For a complete list of all known issues in this release, see the list of OADP 1.3.0 known issues in Jira.
4.2.2.4.4. Upgrade notes
Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OpenShift API for Data Protection (OADP) 1.1 to 1.3, upgrade first to 1.2, and then to 1.3.
4.2.2.4.4.1. Changes from OADP 1.2 to 1.3
The Velero server has been updated from version 1.11 to 1.12.
OpenShift API for Data Protection (OADP) 1.3 uses the Velero built-in Data Mover instead of the VolumeSnapshotMover (VSM) or the Volsync Data Mover.
This changes the following:
-
The
spec.features.dataMover
field and the VSM plugin are not compatible with OADP 1.3, and you must remove the configuration from theDataProtectionApplication
(DPA) configuration. - The Volsync Operator is no longer required for Data Mover functionality, and you can remove it.
-
The custom resource definitions
volumesnapshotbackups.datamover.oadp.openshift.io
andvolumesnapshotrestores.datamover.oadp.openshift.io
are no longer required, and you can remove them. - The secrets used for the OADP-1.2 Data Mover are no longer required, and you can remove them.
OADP 1.3 supports Kopia, which is an alternative file system backup tool to Restic.
To employ Kopia, use the new
spec.configuration.nodeAgent
field as shown in the following example:Example
spec: configuration: nodeAgent: enable: true uploaderType: kopia # ...
The
spec.configuration.restic
field is deprecated in OADP 1.3 and will be removed in a future version of OADP. To avoid seeing deprecation warnings, remove therestic
key and its values, and use the following new syntax:Example
spec: configuration: nodeAgent: enable: true uploaderType: restic # ...
In a future OADP release, it is planned that the kopia
tool will become the default uploaderType
value.
4.2.2.4.4.2. Upgrading from OADP 1.2 Technology Preview Data Mover
OpenShift API for Data Protection (OADP) 1.2 Data Mover backups cannot be restored with OADP 1.3. To prevent a gap in the data protection of your applications, complete the following steps before upgrading to OADP 1.3:
Procedure
- If your cluster backups are sufficient and Container Storage Interface (CSI) storage is available, back up the applications with a CSI backup.
If you require off cluster backups:
-
Back up the applications with a file system backup that uses the
--default-volumes-to-fs-backup=true or backup.spec.defaultVolumesToFsBackup
options. -
Back up the applications with your object storage plugins, for example,
velero-plugin-for-aws
.
-
Back up the applications with a file system backup that uses the
The default timeout value for the Restic file system backup is one hour. In OADP 1.3.1 and later, the default timeout value for Restic and Kopia is four hours.
To restore OADP 1.2 Data Mover backup, you must uninstall OADP, and install and configure OADP 1.2.
4.2.2.4.4.3. Backing up the DPA configuration
You must back up your current DataProtectionApplication
(DPA) configuration.
Procedure
Save your current DPA configuration by running the following command:
Example
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
4.2.2.4.4.4. Upgrading the OADP Operator
Use the following sequence when upgrading the OpenShift API for Data Protection (OADP) Operator.
Procedure
-
Change your subscription channel for the OADP Operator from
stable-1.2
tostable-1.3
. - Allow time for the Operator and containers to update and restart.
Additional resources
4.2.2.4.4.5. Converting DPA to the new version
If you need to move backups off cluster with the Data Mover, reconfigure the DataProtectionApplication
(DPA) manifest as follows.
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- In the Provided APIs section, click View more.
- Click Create instance in the DataProtectionApplication box.
Click YAML View to display the current DPA parameters.
Example current DPA
spec: configuration: features: dataMover: enable: true credentialName: dm-credentials velero: defaultPlugins: - vsm - csi - openshift # ...
Update the DPA parameters:
-
Remove the
features.dataMover
key and values from the DPA. - Remove the VolumeSnapshotMover (VSM) plugin.
Add the
nodeAgent
key and values.Example updated DPA
spec: configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - csi - openshift # ...
-
Remove the
- Wait for the DPA to reconcile successfully.
4.2.2.4.4.6. Verifying the upgrade
Use the following procedure to verify the upgrade.
Procedure
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
In OADP 1.3 you can start data movement off cluster per backup versus creating a DataProtectionApplication
(DPA) configuration.
Example
$ velero backup create example-backup --include-namespaces mysql-persistent --snapshot-move-data=true
Example
apiVersion: velero.io/v1 kind: Backup metadata: name: example-backup namespace: openshift-adp spec: snapshotMoveData: true includedNamespaces: - mysql-persistent storageLocation: dpa-sample-1 ttl: 720h0m0s # ...
4.2.3. OADP 1.2 release notes
The release notes for OpenShift API for Data Protection (OADP) 1.2 describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.
4.2.3.1. OADP 1.2.5 release notes
OpenShift API for Data Protection (OADP) 1.2.5 is a Container Grade Only (CGO) release, released to refresh the health grades of the containers, with no changes to any code in the product itself compared to that of OADP 1.2.4.
4.2.3.1.1. Resolved issues
CVE-2023-2431: oadp-velero-plugin-for-microsoft-azure-container
: Bypass of seccomp profile enforcement
A flaw was found in Kubernetes, which impacts earlier versions of OADP. This flaw arises when Kubernetes allows a local authenticated attacker to bypass security restrictions, caused by a flaw when using the localhost type for a seccomp
profile but specifying an empty profile field. An attacker can bypass the seccomp
profile enforcement by sending a specially crafted request. This flaw has been resolved in OADP 1.2.5.
For more details, see (CVE-2023-2431).
CSI restore ended with 'PartiallyFailed' status and PVCs not created
CSI restore ended with PartiallyFailed
status. PVCs are not created, pod are in Pending
status. This issue has been resolved in OADP 1.2.5.
PodVolumeBackup fails on completed pod volumes
In earlier versions of OADP 1.2, when there is a completed pod that mounted volumes in a namespace used by the Restic podvolumebackup
or Velero backup, the backup does not complete successfully. This occurs when defaultVolumesToFsBackup
is set to true
. This issue has been resolved in OADP 1.2.5.
4.2.3.1.2. Known issues
Data Protection Application (DPA) does not reconcile when the credentials secret is updated
Currently, the OADP Operator does not reconcile when you update the cloud-credentials
secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials
secret. If you create a cloud-credentials
secret with incorrect credentials, such as empty data, the Operator reconciles and creates a backup storage location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials
secret with the correct credentials, the OADP Operator does not immediately reconcile to catch the new credentials.
Workaround: Update to OADP 1.3.
4.2.3.2. OADP 1.2.4 release notes
OpenShift API for Data Protection (OADP) 1.2.4 is a Container Grade Only (CGO) release, released to refresh the health grades of the containers, with no changes to any code in the product itself compared to that of OADP 1.2.3.
4.2.3.2.1. Resolved issues
There are no resolved issues in OADP 1.2.4.
4.2.3.2.2. Known issues
The OADP 1.2.4 has the following known issue:
Data Protection Application (DPA) does not reconcile when the credentials secret is updated
Currently, the OADP Operator does not reconcile when you update the cloud-credentials
secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials
secret. If you create a cloud-credentials
secret with incorrect credentials, such as empty data, the Operator reconciles and creates a Backup Storage Location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials
secret with the correct credentials, the Operator does not immediately reconcile to catch the new credentials.
Workaround: Update to OADP 1.3.
4.2.3.3. OADP 1.2.3 release notes
4.2.3.3.1. New features
There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.3.
4.2.3.3.2. Resolved issues
The following highlighted issues are resolved in OADP 1.2.3:
Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)
In previous releases of OADP 1.2, the HTTP/2 protocol was susceptible to a denial of service attack because request cancellation could reset multiple streams quickly. The server had to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This resulted in a denial of service due to server resource consumption. For a list of all OADP issues associated with this CVE, see the following Jira list.
For more information, see CVE-2023-39325 (Rapid Reset Attack).
For a complete list of all issues resolved in the release of OADP 1.2.3, see the list of OADP 1.2.3 resolved issues in Jira.
4.2.3.3.3. Known issues
The OADP 1.2.3 has the following known issue:
Data Protection Application (DPA) does not reconcile when the credentials secret is updated
Currently, the OADP Operator does not reconcile when you update the cloud-credentials
secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials
secret. If you create a cloud-credentials
secret with incorrect credentials, such as empty data, the Operator reconciles and creates a Backup Storage Location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials
secret with the correct credentials, the Operator does not immediately reconcile to catch the new credentials.
Workaround: Update to OADP 1.3.
4.2.3.4. OADP 1.2.2 release notes
4.2.3.4.1. New features
There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.2.
4.2.3.4.2. Resolved issues
The following highlighted issues are resolved in OADP 1.2.2:
Restic restore partially failed due to a Pod Security standard
In previous releases of OADP 1.2, OpenShift Container Platform 4.14 enforced a pod security admission (PSA) policy that hindered the readiness of pods during a Restic restore process.
This issue has been resolved in the release of OADP 1.2.2, and also OADP 1.1.6. Therefore, it is recommended that users upgrade to these releases.
For more information, see Restic restore partially failing on OCP 4.14 due to changed PSA policy. (OADP-2094)
Backup of an app with internal images partially failed with plugin panicked error
In previous releases of OADP 1.2, the backup of an application with internal images partially failed with plugin panicked error returned. The backup partially fails with this error in the Velero logs:
time="2022-11-23T15:40:46Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/django-persistent-67a5b83d-6b44-11ed-9cba-902e163f806c logSource="/remote-source/velero/app/pkg/backup/backup.go:413" name=django-psql-persistent time="2022-11-23T15:40:46Z" level=error msg="Error backing up item" backup=openshift-adp/django-persistent-67a5b83d-6b44-11ed-9cba-902e163f8
This issue has been resolved in OADP 1.2.2. (OADP-1057).
ACM cluster restore was not functioning as expected due to restore order
In previous releases of OADP 1.2, ACM cluster restore was not functioning as expected due to restore order. ACM applications were removed and re-created on managed clusters after restore activation. (OADP-2505)
VM’s using filesystemOverhead failed when backing up and restoring due to volume size mismatch
In previous releases of OADP 1.2, due to storage provider implementation choices, whenever there was a difference between the application persistent volume claims (PVCs) storage request and the snapshot size of the same PVC, VM’s using filesystemOverhead failed when backing up and restoring. This issue has been resolved in the Data Mover of OADP 1.2.2. (OADP-2144)
OADP did not contain an option to set VolSync replication source prune interval
In previous releases of OADP 1.2, there was no option to set the VolSync replication source pruneInterval
. (OADP-2052)
Possible pod volume backup failure if Velero was installed in multiple namespaces
In previous releases of OADP 1.2, there was a possibility of pod volume backup failure if Velero was installed in multiple namespaces. (OADP-2409)
Backup Storage Locations moved to unavailable phase when VSL uses custom secret
In previous releases of OADP 1.2, Backup Storage Locations moved to unavailable phase when Volume Snapshot Location used custom secret. (OADP-1737)
For a complete list of all issues resolved in the release of OADP 1.2.2, see the list of OADP 1.2.2 resolved issues in Jira.
4.2.3.4.3. Known issues
The following issues have been highlighted as known issues in the release of OADP 1.2.2:
Must-gather command fails to remove ClusterRoleBinding resources
The oc adm must-gather
command fails to remove ClusterRoleBinding
resources, which are left on cluster due to admission webhook. Therefore, requests for the removal of the ClusterRoleBinding
resources are denied. (OADP-27730)
admission webhook "clusterrolebindings-validation.managed.openshift.io" denied the request: Deleting ClusterRoleBinding must-gather-p7vwj is not allowed
For a complete list of all known issues in this release, see the list of OADP 1.2.2 known issues in Jira.
4.2.3.5. OADP 1.2.1 release notes
4.2.3.5.1. New features
There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.1.
4.2.3.5.2. Resolved issues
For a complete list of all issues resolved in the release of OADP 1.2.1, see the list of OADP 1.2.1 resolved issues in Jira.
4.2.3.5.3. Known issues
The following issues have been highlighted as known issues in the release of OADP 1.2.1:
DataMover Restic retain and prune policies do not work as expected
The retention and prune features provided by VolSync and Restic are not working as expected. Because there is no working option to set the prune interval on VolSync replication, you have to manage and prune remotely stored backups on S3 storage outside of OADP. For more details, see:
OADP Data Mover is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
For a complete list of all known issues in this release, see the list of OADP 1.2.1 known issues in Jira.
4.2.3.6. OADP 1.2.0 release notes
The OADP 1.2.0 release notes include information about new features, bug fixes, and known issues.
4.2.3.6.1. New features
Resource timeouts
The new resourceTimeout
option specifies the timeout duration in minutes for waiting on various Velero resources. This option applies to resources such as Velero CRD availability, volumeSnapshot
deletion, and backup repository availability. The default duration is 10 minutes.
AWS S3 compatible backup storage providers
You can back up objects and snapshots on AWS S3 compatible providers.
4.2.3.6.1.1. Technical preview features
Data Mover
The OADP Data Mover enables you to back up Container Storage Interface (CSI) volume snapshots to a remote object store. When you enable Data Mover, you can restore stateful applications using CSI volume snapshots pulled from the object store in case of accidental cluster deletion, cluster failure, or data corruption.
OADP Data Mover is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
4.2.3.6.2. Resolved issues
For a complete list of all issues resolved in this release, see the list of OADP 1.2.0 resolved issues in Jira.
4.2.3.6.3. Known issues
The following issues have been highlighted as known issues in the release of OADP 1.2.0:
Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)
The HTTP/2 protocol is susceptible to a denial of service attack because request cancellation can reset multiple streams quickly. The server has to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This results in a denial of service due to server resource consumption.
It is advised to upgrade to OADP 1.2.3, which resolves this issue.
For more information, see CVE-2023-39325 (Rapid Reset Attack).
An incorrect hostname can be created when changing a hostname in a generated route.
By default, the OpenShift Container Platform cluster makes sure that the openshift.io/host.generated: true
annotation is turned on and fills in the field for both the routes that are generated and those that are not generated.
You cannot modify the value for the .spec.host
field based on the base domain name of your cluster in the generated and non-generated routes.
If you modify the value for the .spec.host
field, it is not possible to restore the default value that was generated by the OpenShift Container Platform cluster. After you restore your OpenShift Container Platform cluster, the Operator resets the value for the field.
4.2.3.6.4. Upgrade notes
Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OpenShift API for Data Protection (OADP) 1.1 to 1.3, upgrade first to 1.2, then to 1.3.
4.2.3.6.4.1. Changes from OADP 1.1 to 1.2
The Velero server was updated from version 1.9 to 1.11.
In OADP 1.2, the DataProtectionApplication
(DPA) configuration dpa.spec.configuration.velero.args
has the following changes:
-
The
default-volumes-to-restic
field was renamed todefault-volumes-to-fs-backup
. If you usedpa.spec.configuration.velero.args
, you must add it again with the new name to your DPA after upgrading OADP. -
The
restic-timeout
field was renamed tofs-backup-timeout
. If you usedpa.spec.configuration.velero.args
, you must add it again with the new name to your DPA after upgrading OADP. -
The
restic
daemon set was renamed tonode-agent
. OADP automatically updates the name of the daemon set. -
The custom resource definition
resticrepositories.velero.io
was renamed tobackuprepositories.velero.io
. -
The custom resource definition
resticrepositories.velero.io
can be removed from the cluster.
4.2.3.6.5. Upgrading steps
4.2.3.6.5.1. Backing up the DPA configuration
You must back up your current DataProtectionApplication
(DPA) configuration.
Procedure
Save your current DPA configuration by running the following command:
Example
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
4.2.3.6.5.2. Upgrading the OADP Operator
Use the following sequence when upgrading the OpenShift API for Data Protection (OADP) Operator.
Procedure
-
Change your subscription channel for the OADP Operator from
stable-1.1
tostable-1.2
. - Allow time for the Operator and containers to update and restart.
4.2.3.6.5.3. Converting DPA to the new version
If you use the fields that were updated in the spec.configuration.velero.args
stanza, you must configure your DataProtectionApplication
(DPA) manifest to use the new parameter names.
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Select Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View to display the current DPA parameters.
Example current DPA
spec: configuration: velero: args: default-volumes-to-fs-backup: true default-restic-prune-frequency: 6000 fs-backup-timeout: 600 # ...
- Update the DPA parameters:
Update the DPA parameter names without changing their values:
-
Change the
default-volumes-to-restic
key todefault-volumes-to-fs-backup
. -
Change the
default-restic-prune-frequency
key todefault-repo-maintain-frequency
. -
Change the
restic-timeout
key tofs-backup-timeout
.
.Example updated DPA
spec: configuration: velero: args: default-volumes-to-fs-backup: true default-repo-maintain-frequency: 6000 fs-backup-timeout: 600 # ...
-
Change the
- Wait for the DPA to reconcile successfully.
The default timeout value for the Restic file system backup is one hour. In OADP 1.3.1 and later, the default timeout value for Restic and Kopia is four hours.
4.2.3.6.5.4. Verifying the upgrade
Use the following procedure to verify the upgrade.
Procedure
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.2.4. OADP 1.1 release notes
The release notes for OpenShift API for Data Protection (OADP) 1.1 describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.
4.2.4.1. OADP 1.1.8 release notes
The OpenShift API for Data Protection (OADP) 1.1.8 release notes lists any known issues. There are no resolved issues in this release.
4.2.4.1.1. Known issues
For a complete list of all known issues in OADP 1.1.8, see the list of OADP 1.1.8 known issues in Jira.
4.2.4.2. OADP 1.1.7 release notes
The OADP 1.1.7 release notes lists any resolved issues and known issues.
4.2.4.2.1. Resolved issues
The following highlighted issues are resolved in OADP 1.1.7:
Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)
In previous releases of OADP 1.1, the HTTP/2 protocol was susceptible to a denial of service attack because request cancellation could reset multiple streams quickly. The server had to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This resulted in a denial of service due to server resource consumption. For a list of all OADP issues associated with this CVE, see the following Jira list.
For more information, see CVE-2023-39325 (Rapid Reset Attack).
For a complete list of all issues resolved in the release of OADP 1.1.7, see the list of OADP 1.1.7 resolved issues in Jira.
4.2.4.2.2. Known issues
There are no known issues in the release of OADP 1.1.7.
4.2.4.3. OADP 1.1.6 release notes
The OADP 1.1.6 release notes lists any new features, resolved issues and bugs, and known issues.
4.2.4.3.1. Resolved issues
Restic restore partially failing due to Pod Security standard
OCP 4.14 introduced pod security standards that meant the privileged
profile is enforced
. In previous releases of OADP, this profile caused the pod to receive permission denied
errors. This issue was caused because of the restore order. The pod was created before the security context constraints (SCC) resource. As this pod violated the pod security standard, the pod was denied and subsequently failed. OADP-2420
Restore partially failing for job resource
In previous releases of OADP, the restore of job resource was partially failing in OCP 4.14. This issue was not seen in older OCP versions. The issue was caused by an additional label being to the job resource, which was not present in older OCP versions. OADP-2530
For a complete list of all issues resolved in this release, see the list of OADP 1.1.6 resolved issues in Jira.
4.2.4.3.2. Known issues
For a complete list of all known issues in this release, see the list of OADP 1.1.6 known issues in Jira.
4.2.4.4. OADP 1.1.5 release notes
The OADP 1.1.5 release notes lists any new features, resolved issues and bugs, and known issues.
4.2.4.4.1. New features
This version of OADP is a service release. No new features are added to this version.
4.2.4.4.2. Resolved issues
For a complete list of all issues resolved in this release, see the list of OADP 1.1.5 resolved issues in Jira.
4.2.4.4.3. Known issues
For a complete list of all known issues in this release, see the list of OADP 1.1.5 known issues in Jira.
4.2.4.5. OADP 1.1.4 release notes
The OADP 1.1.4 release notes lists any new features, resolved issues and bugs, and known issues.
4.2.4.5.1. New features
This version of OADP is a service release. No new features are added to this version.
4.2.4.5.2. Resolved issues
Add support for all the velero deployment server arguments
In previous releases of OADP, OADP did not facilitate the support of all the upstream Velero server arguments. This issue has been resolved in OADP 1.1.4 and all the upstream Velero server arguments are supported. OADP-1557
Data Mover can restore from an incorrect snapshot when there was more than one VSR for the restore name and pvc name
In previous releases of OADP, OADP Data Mover could restore from an incorrect snapshot if there was more than one Volume Snapshot Restore (VSR) resource in the cluster for the same Velero restore
name and PersistentVolumeClaim (pvc) name. OADP-1822
Cloud Storage API BSLs need OwnerReference
In previous releases of OADP, ACM BackupSchedules failed validation because of a missing OwnerReference
on Backup Storage Locations (BSLs) created with dpa.spec.backupLocations.bucket
. OADP-1511
For a complete list of all issues resolved in this release, see the list of OADP 1.1.4 resolved issues in Jira.
4.2.4.5.3. Known issues
This release has the following known issues:
OADP backups might fail because a UID/GID range might have changed on the cluster
OADP backups might fail because a UID/GID range might have changed on the cluster where the application has been restored, with the result that OADP does not back up and restore OpenShift Container Platform UID/GID range metadata. To avoid the issue, if the backed application requires a specific UUID, ensure the range is available when restored. An additional workaround is to allow OADP to create the namespace in the restore operation.
A restoration might fail if ArgoCD is used during the process due to a label used by ArgoCD
A restoration might fail if ArgoCD is used during the process due to a label used by ArgoCD, app.kubernetes.io/instance
. This label identifies which resources ArgoCD needs to manage, which can create a conflict with OADP’s procedure for managing resources on restoration. To work around this issue, set .spec.resourceTrackingMethod
on the ArgoCD YAML to annotation+label
or annotation
. If the issue continues to persist, then disable ArgoCD before beginning to restore, and enable it again when restoration is finished.
OADP Velero plugins returning "received EOF, stopping recv loop" message
Velero plugins are started as separate processes. When the Velero operation has completed, either successfully or not, they exit. Therefore if you see a received EOF, stopping recv loop
messages in debug logs, it does not mean an error occurred. The message indicates that a plugin operation has completed. OADP-2176
For a complete list of all known issues in this release, see the list of OADP 1.1.4 known issues in Jira.
4.2.4.6. OADP 1.1.3 release notes
The OADP 1.1.3 release notes lists any new features, resolved issues and bugs, and known issues.
4.2.4.6.1. New features
This version of OADP is a service release. No new features are added to this version.
4.2.4.6.2. Resolved issues
For a complete list of all issues resolved in this release, see the list of OADP 1.1.3 resolved issues in Jira.
4.2.4.6.3. Known issues
For a complete list of all known issues in this release, see the list of OADP 1.1.3 known issues in Jira.
4.2.4.7. OADP 1.1.2 release notes
The OADP 1.1.2 release notes include product recommendations, a list of fixed bugs and descriptions of known issues.
4.2.4.7.1. Product recommendations
VolSync
To prepare for the upgrade from VolSync 0.5.1 to the latest version available from the VolSync stable channel, you must add this annotation in the openshift-adp
namespace by running the following command:
$ oc annotate --overwrite namespace/openshift-adp volsync.backube/privileged-movers='true'
Velero
In this release, Velero has been upgraded from version 1.9.2 to version 1.9.5.
Restic
In this release, Restic has been upgraded from version 0.13.1 to version 0.14.0.
4.2.4.7.2. Resolved issues
The following issues have been resolved in this release:
4.2.4.7.3. Known issues
This release has the following known issues:
- OADP currently does not support backup and restore of AWS EFS volumes using restic in Velero (OADP-778).
CSI backups might fail due to a Ceph limitation of
VolumeSnapshotContent
snapshots per PVC.You can create many snapshots of the same persistent volume claim (PVC) but cannot schedule periodic creation of snapshots:
For more information, see Volume Snapshots.
4.2.4.8. OADP 1.1.1 release notes
The OADP 1.1.1 release notes include product recommendations and descriptions of known issues.
4.2.4.8.1. Product recommendations
Before you install OADP 1.1.1, it is recommended to either install VolSync 0.5.1 or to upgrade to it.
4.2.4.8.2. Known issues
This release has the following known issues:
Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)
The HTTP/2 protocol is susceptible to a denial of service attack because request cancellation can reset multiple streams quickly. The server has to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This results in a denial of service due to server resource consumption. For a list of all OADP issues associated with this CVE, see the following Jira list.
It is advised to upgrade to OADP 1.1.7 or 1.2.3, which resolve this issue.
For more information, see CVE-2023-39325 (Rapid Reset Attack).
- OADP currently does not support backup and restore of AWS EFS volumes using restic in Velero (OADP-778).
CSI backups might fail due to a Ceph limitation of
VolumeSnapshotContent
snapshots per PVC.You can create many snapshots of the same persistent volume claim (PVC) but cannot schedule periodic creation of snapshots:
- For CephFS, you can create up to 100 snapshots per PVC.
For RADOS Block Device (RBD), you can create up to 512 snapshots for each PVC. (OADP-804) and (OADP-975)
For more information, see Volume Snapshots.
4.3. OADP features and plugins
OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications.
The default plugins enable Velero to integrate with certain cloud providers and to back up and restore OpenShift Container Platform resources.
4.3.1. OADP features
OpenShift API for Data Protection (OADP) supports the following features:
- Backup
You can use OADP to back up all applications on the OpenShift Platform, or you can filter the resources by type, namespace, or label.
OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Restore
You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the objects by namespace, PV, or label.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Schedule
- You can schedule backups at specified intervals.
- Hooks
-
You can use hooks to run commands in a container on a pod, for example,
fsfreeze
to freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.
4.3.2. OADP plugins
The OpenShift API for Data Protection (OADP) provides default Velero plugins that are integrated with storage providers to support backup and snapshot operations. You can create custom plugins based on the Velero plugins.
OADP also provides plugins for OpenShift Container Platform resource backups, OpenShift Virtualization resource backups, and Container Storage Interface (CSI) snapshots.
OADP plugin | Function | Storage location |
---|---|---|
| Backs up and restores Kubernetes objects. | AWS S3 |
Backs up and restores volumes with snapshots. | AWS EBS | |
| Backs up and restores Kubernetes objects. | Microsoft Azure Blob storage |
Backs up and restores volumes with snapshots. | Microsoft Azure Managed Disks | |
| Backs up and restores Kubernetes objects. | Google Cloud Storage |
Backs up and restores volumes with snapshots. | Google Compute Engine Disks | |
| Backs up and restores OpenShift Container Platform resources. [1] | Object store |
| Backs up and restores OpenShift Virtualization resources. [2] | Object store |
| Backs up and restores volumes with CSI snapshots. [3] | Cloud storage that supports CSI snapshots |
| VolumeSnapshotMover relocates snapshots from the cluster into an object store to be used during a restore process to recover stateful applications, in situations such as cluster deletion. [4] | Object store |
- Mandatory.
- Virtual machine disks are backed up with CSI snapshots or Restic.
The
csi
plugin uses the Kubernetes CSI snapshot API.-
OADP 1.1 or later uses
snapshot.storage.k8s.io/v1
-
OADP 1.0 uses
snapshot.storage.k8s.io/v1beta1
-
OADP 1.1 or later uses
- OADP 1.2 only.
4.3.3. About OADP Velero plugins
You can configure two types of plugins when you install Velero:
- Default cloud provider plugins
- Custom plugins
Both types of plugin are optional, but most users configure at least one cloud provider plugin.
4.3.3.1. Default Velero cloud provider plugins
You can install any of the following default Velero cloud provider plugins when you configure the oadp_v1alpha1_dpa.yaml
file during deployment:
-
aws
(Amazon Web Services) -
gcp
(Google Cloud Platform) -
azure
(Microsoft Azure) -
openshift
(OpenShift Velero plugin) -
csi
(Container Storage Interface) -
kubevirt
(KubeVirt)
You specify the desired default plugins in the oadp_v1alpha1_dpa.yaml
file during deployment.
Example file
The following .yaml
file installs the openshift
, aws
, azure
, and gcp
plugins:
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample spec: configuration: velero: defaultPlugins: - openshift - aws - azure - gcp
4.3.3.2. Custom Velero plugins
You can install a custom Velero plugin by specifying the plugin image
and name
when you configure the oadp_v1alpha1_dpa.yaml
file during deployment.
You specify the desired custom plugins in the oadp_v1alpha1_dpa.yaml
file during deployment.
Example file
The following .yaml
file installs the default openshift
, azure
, and gcp
plugins and a custom plugin that has the name custom-plugin-example
and the image quay.io/example-repo/custom-velero-plugin
:
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample spec: configuration: velero: defaultPlugins: - openshift - azure - gcp customPlugins: - name: custom-plugin-example image: quay.io/example-repo/custom-velero-plugin
4.3.3.3. Velero plugins returning "received EOF, stopping recv loop" message
Velero plugins are started as separate processes. After the Velero operation has completed, either successfully or not, they exit. Receiving a received EOF, stopping recv loop
message in the debug logs indicates that a plugin operation has completed. It does not mean that an error has occurred.
4.3.4. Supported architectures for OADP
OpenShift API for Data Protection (OADP) supports the following architectures:
- AMD64
- ARM64
- PPC64le
- s390x
OADP 1.2.0 and later versions support the ARM64 architecture.
4.3.5. OADP support for IBM Power and IBM Z
OpenShift API for Data Protection (OADP) is platform neutral. The information that follows relates only to IBM Power® and to IBM Z®.
- OADP 1.1.7 was tested successfully against OpenShift Container Platform 4.11 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.1.7 in terms of backup locations for these systems.
- OADP 1.2.3 was tested successfully against OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.2.3 in terms of backup locations for these systems.
- OADP 1.3.3 was tested successfully against OpenShift Container Platform 4.13, 4.14, and 4.15 for both IBM Power® and IBM Z®. The sections that follow give testing and support information for OADP 1.3.3 in terms of backup locations for these systems.
4.3.5.1. OADP support for target backup locations using IBM Power
- IBM Power® running with OpenShift Container Platform 4.11 and 4.12, and OpenShift API for Data Protection (OADP) 1.1.7 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.11 and 4.12, and OADP 1.1.7 against all S3 backup location targets, which are not AWS, as well.
- IBM Power® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and OADP 1.2.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.12, 4.13. 4.14, and 4.15, and OADP 1.2.3 against all S3 backup location targets, which are not AWS, as well.
- IBM Power® running with OpenShift Container Platform 4.13, 4.14, and 4.15, and OADP 1.3.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Power® with OpenShift Container Platform 4.13, 4.14, and 4.15, and OADP 1.3.3 against all S3 backup location targets, which are not AWS, as well.
4.3.5.2. OADP testing and support for target backup locations using IBM Z
- IBM Z® running with OpenShift Container Platform 4.11 and 4.12, and OpenShift API for Data Protection (OADP) 1.1.7 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.11 and 4.12, and OADP 1.1.7 against all S3 backup location targets, which are not AWS, as well.
- IBM Z® running with OpenShift Container Platform 4.12, 4.13, 4.14, and 4.15, and OADP 1.2.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.12, 4.13, 4.14 and 4.15, and OADP 1.2.3 against all S3 backup location targets, which are not AWS, as well.
- IBM Z® running with OpenShift Container Platform 4.13, 4.14, and 4.15, and 1.3.3 was tested successfully against an AWS S3 backup location target. Although the test involved only an AWS S3 target, Red Hat supports running IBM Z® with OpenShift Container Platform 4.13 4.14, and 4.15, and 1.3.3 against all S3 backup location targets, which are not AWS, as well.
4.3.5.2.1. Known issue of OADP using IBM Power(R) and IBM Z(R) platforms
- Currently, there are backup method restrictions for Single-node OpenShift clusters deployed on IBM Power® and IBM Z® platforms. Only NFS storage is currently compatible with Single-node OpenShift clusters on these platforms. In addition, only the File System Backup (FSB) methods such as Kopia and Restic are supported for backup and restore operations. There is currently no workaround for this issue.
4.3.6. OADP plugins known issues
The following section describes known issues in OpenShift API for Data Protection (OADP) plugins:
4.3.6.1. Velero plugin panics during imagestream backups due to a missing secret
When the backup and the Backup Storage Location (BSL) are managed outside the scope of the Data Protection Application (DPA), the OADP controller, meaning the DPA reconciliation does not create the relevant oadp-<bsl_name>-<bsl_provider>-registry-secret
.
When the backup is run, the OpenShift Velero plugin panics on the imagestream backup, with the following panic error:
024-02-27T10:46:50.028951744Z time="2024-02-27T10:46:50Z" level=error msg="Error backing up item" backup=openshift-adp/<backup name> error="error executing custom action (groupResource=imagestreams.image.openshift.io, namespace=<BSL Name>, name=postgres): rpc error: code = Aborted desc = plugin panicked: runtime error: index out of range with length 1, stack trace: goroutine 94…
4.3.6.1.1. Workaround to avoid the panic error
To avoid the Velero plugin panic error, perform the following steps:
Label the custom BSL with the relevant label:
$ oc label BackupStorageLocation <bsl_name> app.kubernetes.io/component=bsl
After the BSL is labeled, wait until the DPA reconciles.
NoteYou can force the reconciliation by making any minor change to the DPA itself.
When the DPA reconciles, confirm that the relevant
oadp-<bsl_name>-<bsl_provider>-registry-secret
has been created and that the correct registry data has been populated into it:$ oc -n openshift-adp get secret/oadp-<bsl_name>-<bsl_provider>-registry-secret -o json | jq -r '.data'
4.3.6.2. OpenShift ADP Controller segmentation fault
If you configure a DPA with both cloudstorage
and restic
enabled, the openshift-adp-controller-manager
pod crashes and restarts indefinitely until the pod fails with a crash loop segmentation fault.
You can have either velero
or cloudstorage
defined, because they are mutually exclusive fields.
-
If you have both
velero
andcloudstorage
defined, theopenshift-adp-controller-manager
fails. -
If you have neither
velero
norcloudstorage
defined, theopenshift-adp-controller-manager
fails.
For more information about this issue, see OADP-1054.
4.3.6.2.1. OpenShift ADP Controller segmentation fault workaround
You must define either velero
or cloudstorage
when you configure a DPA. If you define both APIs in your DPA, the openshift-adp-controller-manager
pod fails with a crash loop segmentation fault.
4.4. Installing and configuring OADP
4.4.1. About installing OADP
As a cluster administrator, you install the OpenShift API for Data Protection (OADP) by installing the OADP Operator. The OADP Operator installs Velero 1.14.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
To back up Kubernetes resources and internal images, you must have object storage as a backup location, such as one of the following storage types:
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- Multicloud Object Gateway
- IBM Cloud® Object Storage S3
- AWS S3 compatible object storage, such as Multicloud Object Gateway or MinIO
You can configure multiple backup storage locations within the same namespace for each individual OADP deployment.
Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.
For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.
The CloudStorage
API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The CloudStorage
API is a Technology Preview feature when you use a CloudStorage
object and want OADP to use the CloudStorage
API to automatically create an S3 bucket for use as a BackupStorageLocation
.
The CloudStorage
API supports manually creating a BackupStorageLocation
object by specifying an existing S3 bucket. The CloudStorage
API that creates an S3 bucket automatically is currently only enabled for AWS S3 storage.
You can back up persistent volumes (PVs) by using snapshots or a File System Backup (FSB).
To back up PVs with snapshots, you must have a cloud provider that supports either a native snapshot API or Container Storage Interface (CSI) snapshots, such as one of the following cloud providers:
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- CSI snapshot-enabled cloud provider, such as OpenShift Data Foundation
If you want to use CSI backup on OCP 4.11 and later, install OADP 1.1.x.
OADP 1.0.x does not support CSI backup on OCP 4.11 and later. OADP 1.0.x includes Velero 1.7.x and expects the API group snapshot.storage.k8s.io/v1beta1
, which is not present on OCP 4.11 and later.
If your cloud provider does not support snapshots or if your storage is NFS, you can back up applications with Backing up applications with File System Backup: Kopia or Restic on object storage.
You create a default Secret
and then you install the Data Protection Application.
4.4.1.1. AWS S3 compatible backup storage providers
OADP is compatible with many object storage providers for use with different backup and snapshot operations. Several object storage providers are fully supported, several are unsupported but known to work, and some have known limitations.
4.4.1.1.1. Supported backup storage providers
The following AWS S3 compatible object storage providers are fully supported by OADP through the AWS plugin for use as backup storage locations:
- MinIO
- Multicloud Object Gateway (MCG)
- Amazon Web Services (AWS) S3
- IBM Cloud® Object Storage S3
- Ceph RADOS Gateway (Ceph Object Gateway)
- Red Hat Container Storage
- Red Hat OpenShift Data Foundation
The following compatible object storage providers are supported and have their own Velero object store plugins:
- Google Cloud Platform (GCP)
- Microsoft Azure
4.4.1.1.2. Unsupported backup storage providers
The following AWS S3 compatible object storage providers, are known to work with Velero through the AWS plugin, for use as backup storage locations, however, they are unsupported and have not been tested by Red Hat:
- Oracle Cloud
- DigitalOcean
- NooBaa, unless installed using Multicloud Object Gateway (MCG)
- Tencent Cloud
- Ceph RADOS v12.2.7
- Quobyte
- Cloudian HyperStore
Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.
For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.
4.4.1.1.3. Backup storage providers with known limitations
The following AWS S3 compatible object storage providers are known to work with Velero through the AWS plugin with a limited feature set:
- Swift - It works for use as a backup storage location for backup storage, but is not compatible with Restic for filesystem-based volume backup and restore.
4.4.1.2. Configuring Multicloud Object Gateway (MCG) for disaster recovery on OpenShift Data Foundation
If you use cluster storage for your MCG bucket backupStorageLocation
on OpenShift Data Foundation, configure MCG as an external object store.
Failure to configure MCG as an external object store might lead to backups not being available.
Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.
For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.
Procedure
- Configure MCG as an external object store as described in Adding storage resources for hybrid or Multicloud.
Additional resources
4.4.1.3. About OADP update channels
When you install an OADP Operator, you choose an update channel. This channel determines which upgrades to the OADP Operator and to Velero you receive. You can switch channels at any time.
The following update channels are available:
-
The stable channel is now deprecated. The stable channel contains the patches (z-stream updates) of OADP
ClusterServiceVersion
forOADP.v1.1.z
and older versions fromOADP.v1.0.z
. - The stable-1.0 channel is deprecated and is not supported.
- The stable-1.1 channel is deprecated and is not supported.
- The stable-1.2 channel is deprecated and is not supported.
-
The stable-1.3 channel contains
OADP.v1.3.z
, the most recent OADP 1.3ClusterServiceVersion
. -
The stable-1.4 channel contains
OADP.v1.4.z
, the most recent OADP 1.4ClusterServiceVersion
.
For more information, see OpenShift Operator Life Cycles.
Which update channel is right for you?
-
The stable channel is now deprecated. If you are already using the stable channel, you will continue to get updates from
OADP.v1.1.z
. - Choose the stable-1.y update channel to install OADP 1.y and to continue receiving patches for it. If you choose this channel, you will receive all z-stream patches for version 1.y.z.
When must you switch update channels?
- If you have OADP 1.y installed, and you want to receive patches only for that y-stream, you must switch from the stable update channel to the stable-1.y update channel. You will then receive all z-stream patches for version 1.y.z.
- If you have OADP 1.0 installed, want to upgrade to OADP 1.1, and then receive patches only for OADP 1.1, you must switch from the stable-1.0 update channel to the stable-1.1 update channel. You will then receive all z-stream patches for version 1.1.z.
- If you have OADP 1.y installed, with y greater than 0, and want to switch to OADP 1.0, you must uninstall your OADP Operator and then reinstall it using the stable-1.0 update channel. You will then receive all z-stream patches for version 1.0.z.
You cannot switch from OADP 1.y to OADP 1.0 by switching update channels. You must uninstall the Operator and then reinstall it.
4.4.1.4. Installation of OADP on multiple namespaces
You can install OpenShift API for Data Protection (OADP) into multiple namespaces on the same cluster so that multiple project owners can manage their own OADP instance. This use case has been validated with File System Backup (FSB) and Container Storage Interface (CSI).
You install each instance of OADP as specified by the per-platform procedures contained in this document with the following additional requirements:
- All deployments of OADP on the same cluster must be the same version, for example, 1.1.4. Installing different versions of OADP on the same cluster is not supported.
-
Each individual deployment of OADP must have a unique set of credentials and at least one
BackupStorageLocation
configuration. You can also use multipleBackupStorageLocation
configurations within the same namespace. - By default, each OADP deployment has cluster-level access across namespaces. OpenShift Container Platform administrators need to review security and RBAC settings carefully and make any necessary changes to them to ensure that each OADP instance has the correct permissions.
Additional resources
4.4.1.5. Velero CPU and memory requirements based on collected data
The following recommendations are based on observations of performance made in the scale and performance lab. The backup and restore resources can be impacted by the type of plugin, the amount of resources required by that backup or restore, and the respective data contained in the persistent volumes (PVs) related to those resources.
4.4.1.5.1. CPU and memory requirement for configurations
Configuration types | [1] Average usage | [2] Large usage | resourceTimeouts |
---|---|---|---|
CSI | Velero: CPU- Request 200m, Limits 1000m Memory - Request 256Mi, Limits 1024Mi | Velero: CPU- Request 200m, Limits 2000m Memory- Request 256Mi, Limits 2048Mi | N/A |
Restic | [3] Restic: CPU- Request 1000m, Limits 2000m Memory - Request 16Gi, Limits 32Gi | [4] Restic: CPU - Request 2000m, Limits 8000m Memory - Request 16Gi, Limits 40Gi | 900m |
[5] Data Mover | N/A | N/A | 10m - average usage 60m - large usage |
- Average usage - use these settings for most usage situations.
- Large usage - use these settings for large usage situations, such as a large PV (500GB Usage), multiple namespaces (100+), or many pods within a single namespace (2000 pods+), and for optimal performance for backup and restore involving large datasets.
- Restic resource usage corresponds to the amount of data, and type of data. For example, many small files or large amounts of data can cause Restic to use large amounts of resources. The Velero documentation references 500m as a supplied default, for most of our testing we found a 200m request suitable with 1000m limit. As cited in the Velero documentation, exact CPU and memory usage is dependent on the scale of files and directories, in addition to environmental limitations.
- Increasing the CPU has a significant impact on improving backup and restore times.
- Data Mover - Data Mover default resourceTimeout is 10m. Our tests show that for restoring a large PV (500GB usage), it is required to increase the resourceTimeout to 60m.
The resource requirements listed throughout the guide are for average usage only. For large usage, adjust the settings as described in the table above.
4.4.1.5.2. NodeAgent CPU for large usage
Testing shows that increasing NodeAgent
CPU can significantly improve backup and restore times when using OpenShift API for Data Protection (OADP).
It is not recommended to use Kopia without limits in production environments on nodes running production workloads due to Kopia’s aggressive consumption of resources. However, running Kopia with limits that are too low results in CPU limiting and slow backups and restore situations. Testing showed that running Kopia with 20 cores and 32 Gi memory supported backup and restore operations of over 100 GB of data, multiple namespaces, or over 2000 pods in a single namespace.
Testing detected no CPU limiting or memory saturation with these resource specifications.
You can set these limits in Ceph MDS pods by following the procedure in Changing the CPU and memory resources on the rook-ceph pods.
You need to add the following lines to the storage cluster Custom Resource (CR) to set the limits:
resources: mds: limits: cpu: "3" memory: 128Gi requests: cpu: "3" memory: 8Gi
4.4.2. Installing the OADP Operator
You can install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.15 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.14.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
- In the OpenShift Container Platform web console, click Operators → OperatorHub.
- Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. - Click Operators → Installed Operators to verify the installation.
4.4.2.1. OADP-Velero-OpenShift Container Platform version relationship
OADP version | Velero version | OpenShift Container Platform version |
---|---|---|
1.1.0 | 4.9 and later | |
1.1.1 | 4.9 and later | |
1.1.2 | 4.9 and later | |
1.1.3 | 4.9 and later | |
1.1.4 | 4.9 and later | |
1.1.5 | 4.9 and later | |
1.1.6 | 4.11 and later | |
1.1.7 | 4.11 and later | |
1.2.0 | 4.11 and later | |
1.2.1 | 4.11 and later | |
1.2.2 | 4.11 and later | |
1.2.3 | 4.11 and later | |
1.3.0 | 4.10 - 4.15 | |
1.3.1 | 4.10 - 4.15 | |
1.3.2 | 4.10 - 4.15 | |
1.3.3 | 4.10 - 4.15 | |
1.4.0 | 4.14 and later | |
1.4.1 | 4.14 and later |
4.4.3. Configuring the OpenShift API for Data Protection with AWS S3 compatible storage
You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) S3 compatible storage by installing the OADP Operator. The Operator installs Velero 1.14.
IBM Cloud® S3 is supported as an AWS S3 compatible backup storage provider.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You configure AWS for Velero, create a default Secret
, and then install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.4.3.1. Configuring Amazon Web Services
You configure Amazon Web Services (AWS) for the OpenShift API for Data Protection (OADP).
Prerequisites
- You must have the AWS CLI installed.
Procedure
Set the
BUCKET
variable:$ BUCKET=<your_bucket>
Set the
REGION
variable:$ REGION=<your_region>
Create an AWS S3 bucket:
$ aws s3api create-bucket \ --bucket $BUCKET \ --region $REGION \ --create-bucket-configuration LocationConstraint=$REGION 1
- 1
us-east-1
does not support aLocationConstraint
. If your region isus-east-1
, omit--create-bucket-configuration LocationConstraint=$REGION
.
Create an IAM user:
$ aws iam create-user --user-name velero 1
- 1
- If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.
Create a
velero-policy.json
file:$ cat > velero-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:DeleteObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::${BUCKET}/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::${BUCKET}" ] } ] } EOF
Attach the policies to give the
velero
user the minimum necessary permissions:$ aws iam put-user-policy \ --user-name velero \ --policy-name velero \ --policy-document file://velero-policy.json
Create an access key for the
velero
user:$ aws iam create-access-key --user-name velero
Example output
{ "AccessKey": { "UserName": "velero", "Status": "Active", "CreateDate": "2017-07-31T22:24:41.576Z", "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, "AccessKeyId": <AWS_ACCESS_KEY_ID> } }
Create a
credentials-velero
file:$ cat << EOF > ./credentials-velero [default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> EOF
You use the
credentials-velero
file to create aSecret
object for AWS before you install the Data Protection Application.
4.4.3.2. About backup and snapshot locations and their secrets
You specify backup and snapshot locations and their secrets in the DataProtectionApplication
custom resource (CR).
Backup locations
You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.
Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.
Snapshot locations
If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.
If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass
CR to register the CSI driver.
If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.
Secrets
If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret
.
If the backup and snapshot locations use different credentials, you create two secret objects:
-
Custom
Secret
for the backup location, which you specify in theDataProtectionApplication
CR. -
Default
Secret
for the snapshot location, which is not referenced in theDataProtectionApplication
CR.
The Data Protection Application requires a default Secret
. Otherwise, the installation will fail.
If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret
with an empty credentials-velero
file.
4.4.3.2.1. Creating a default Secret
You create a default Secret
if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
The default name of the Secret
is cloud-credentials
.
The DataProtectionApplication
custom resource (CR) requires a default Secret
. Otherwise, the installation will fail. If the name of the backup location Secret
is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret
with the default name by using an empty credentials-velero
file.
Prerequisites
- Your object storage and cloud storage, if any, must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.4.3.2.2. Creating profiles for different credentials
If your backup and snapshot locations use different credentials, you create separate profiles in the credentials-velero
file.
Then, you create a Secret
object and specify the profiles in the DataProtectionApplication
custom resource (CR).
Procedure
Create a
credentials-velero
file with separate profiles for the backup and snapshot locations, as in the following example:[backupStorage] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> [volumeSnapshot] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
Create a
Secret
object with thecredentials-velero
file:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero 1
Add the profiles to the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket_name> prefix: <prefix> config: region: us-east-1 profile: "backupStorage" credential: key: cloud name: cloud-credentials snapshotLocations: - velero: provider: aws config: region: us-west-2 profile: "volumeSnapshot"
4.4.3.2.3. Creating an OADP SSE-C encryption key for additional data security
Amazon Web Services (AWS) S3 applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for every bucket in Amazon S3.
OpenShift API for Data Protection (OADP) encrypts data by using SSL/TLS, HTTPS, and the velero-repo-credentials
secret when transferring the data from a cluster to storage. To protect backup data in case of lost or stolen AWS credentials, apply an additional layer of encryption.
The velero-plugin-for-aws plugin provides several additional encryption methods. You should review its configuration options and consider implementing additional encryption.
You can store your own encryption keys by using server-side encryption with customer-provided keys (SSE-C). This feature provides additional security if your AWS credentials become exposed.
Be sure to store cryptographic keys in a secure and safe manner. Encrypted data and backups cannot be recovered if you do not have the encryption key.
Prerequisites
To make OADP mount a secret that contains your SSE-C key to the Velero pod at
/credentials
, use the following default secret name for AWS:cloud-credentials
, and leave at least one of the following labels empty:-
dpa.spec.backupLocations[].velero.credential
dpa.spec.snapshotLocations[].velero.credential
This is a workaround for a known issue: https://issues.redhat.com/browse/OADP-3971.
-
The following procedure contains an example of a spec:backupLocations
block that does not specify credentials. This example would trigger an OADP secret mounting.
-
If you need the backup location to have credentials with a different name than
cloud-credentials
, you must add a snapshot location, such as the one in the following example, that does not contain a credential name. Because the example does not contain a credential name, the snapshot location will usecloud-credentials
as its secret for taking snapshots.
Example snapshot location in a DPA without credentials specified
snapshotLocations: - velero: config: profile: default region: <region> provider: aws # ...
Procedure
Create an SSE-C encryption key:
Generate a random number and save it as a file named
sse.key
by running the following command:$ dd if=/dev/urandom bs=1 count=32 > sse.key
Encode the
sse.key
by using Base64 and save the result as a file namedsse_encoded.key
by running the following command:$ cat sse.key | base64 > sse_encoded.key
Link the file named
sse_encoded.key
to a new file namedcustomer-key
by running the following command:$ ln -s sse_encoded.key customer-key
Create an OpenShift Container Platform secret:
If you are initially installing and configuring OADP, create the AWS credential and encryption key secret at the same time by running the following command:
$ oc create secret generic cloud-credentials --namespace openshift-adp --from-file cloud=<path>/openshift_aws_credentials,customer-key=<path>/sse_encoded.key
If you are updating an existing installation, edit the values of the
cloud-credential
secret
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: v1 data: cloud: W2Rfa2V5X2lkPSJBS0lBVkJRWUIyRkQ0TlFHRFFPQiIKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5P<snip>rUE1mNWVSbTN5K2FpeWhUTUQyQk1WZHBOIgo= customer-key: v+<snip>TFIiq6aaXPbj8dhos= kind: Secret # ...
Edit the value of the
customerKeyEncryptionFile
attribute in thebackupLocations
block of theDataProtectionApplication
CR manifest, as in the following example:spec: backupLocations: - velero: config: customerKeyEncryptionFile: /credentials/customer-key profile: default # ...
WarningYou must restart the Velero pod to remount the secret credentials properly on an existing installation.
The installation is complete, and you can back up and restore OpenShift Container Platform resources. The data saved in AWS S3 storage is encrypted with the new key, and you cannot download it from the AWS S3 console or API without the additional encryption key.
Verification
To verify that you cannot download the encrypted files without the inclusion of an additional key, create a test file, upload it, and then try to download it.
Create a test file by running the following command:
$ echo "encrypt me please" > test.txt
Upload the test file by running the following command:
$ aws s3api put-object \ --bucket <bucket> \ --key test.txt \ --body test.txt \ --sse-customer-key fileb://sse.key \ --sse-customer-algorithm AES256
Try to download the file. In either the Amazon web console or the terminal, run the following command:
$ s3cmd get s3://<bucket>/test.txt test.txt
The download fails because the file is encrypted with an additional key.
Download the file with the additional encryption key by running the following command:
$ aws s3api get-object \ --bucket <bucket> \ --key test.txt \ --sse-customer-key fileb://sse.key \ --sse-customer-algorithm AES256 \ downloaded.txt
Read the file contents by running the following command:
$ cat downloaded.txt
Example output
encrypt me please
Additional resources
You can also download the file with the additional encryption key backed up with Velcro by running a different command. See Downloading a file with an SSE-C encryption key for files backed up by Velero.
4.4.3.2.3.1. Downloading a file with an SSE-C encryption key for files backed up by Velero
When you are verifying an SSE-C encryption key, you can also download the file with the additional encryption key for files that were backed up with Velcro.
Procedure
- Download the file with the additional encryption key for files backed up by Velero by running the following command:
$ aws s3api get-object \ --bucket <bucket> \ --key velero/backups/mysql-persistent-customerkeyencryptionfile4/mysql-persistent-customerkeyencryptionfile4.tar.gz \ --sse-customer-key fileb://sse.key \ --sse-customer-algorithm AES256 \ --debug \ velero_download.tar.gz
4.4.3.3. Configuring the Data Protection Application
You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.
4.4.3.3.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Use the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.
For more details, see Configuring node agents and node labels.
4.4.3.3.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 # ...
4.4.3.3.2.1. Using CA certificates with the velero command aliased for Velero deployment
You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. You must have the OpenShift CLI (
oc
) installed.To use an aliased Velero command, run the following command:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
Check that the alias is working by running the following command:
Example
$ velero version Client: Version: v1.12.1-OADP Git commit: - Server: Version: v1.12.1-OADP
To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:
$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}') $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
To fetch the backup logs, run the following command:
$ velero backup logs <backup_name> --cacert /tmp/<your_cacert.txt>
You can use these logs to view failures and warnings for the resources that you cannot back up.
-
If the Velero pod restarts, the
/tmp/your-cacert.txt
file disappears, and you must re-create the/tmp/your-cacert.txt
file by re-running the commands from the previous step. You can check if the
/tmp/your-cacert.txt
file still exists, in the file location where you stored it, by running the following command:$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt" /tmp/your-cacert.txt
In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.
4.4.3.4. Installing the Data Protection Application 1.2 and earlier
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create a
Secret
with the default name,cloud-credentials
, which contains separate profiles for the backup and snapshot location credentials.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.NoteVelero creates a secret named
velero-repo-credentials
in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update isData[repository-password]
.After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is
velero-repo-credentials
, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password invelero-repo-credentials
, and therefore, Velero will not be able to connect with the older backups.
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - openshift 1 - aws resourceTimeout: 10m 2 restic: enable: true 3 podConfig: nodeSelector: <node_selector> 4 backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket_name> 5 prefix: <prefix> 6 config: region: <region> profile: "default" s3ForcePathStyle: "true" 7 s3Url: <s3_url> 8 credential: key: cloud name: cloud-credentials 9 snapshotLocations: 10 - velero: provider: aws config: region: <region> 11 profile: "default" credential: key: cloud name: cloud-credentials 12
- 1
- The
openshift
plugin is mandatory. - 2
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 3
- Set this value to
false
if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. In OADP version 1.1, addspec.defaultVolumesToRestic: true
to theBackup
CR. - 4
- Specify on which nodes Restic is available. By default, Restic runs on all nodes.
- 5
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 6
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 7
- Specify whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.
- 8
- Specify the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.
- 9
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. - 10
- Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.
- 11
- The snapshot location must be in the same region as the PVs.
- 12
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in thecredentials-velero
file.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.3.5. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create a
Secret
with the default name,cloud-credentials
, which contains separate profiles for the backup and snapshot location credentials.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp 1 spec: configuration: velero: defaultPlugins: - openshift 2 - aws resourceTimeout: 10m 3 nodeAgent: 4 enable: true 5 uploaderType: kopia 6 podConfig: nodeSelector: <node_selector> 7 backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket_name> 8 prefix: <prefix> 9 config: region: <region> profile: "default" s3ForcePathStyle: "true" 10 s3Url: <s3_url> 11 credential: key: cloud name: cloud-credentials 12 snapshotLocations: 13 - name: default velero: provider: aws config: region: <region> 14 profile: "default" credential: key: cloud name: cloud-credentials 15
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- The
openshift
plugin is mandatory. - 3
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 4
- The administrative agent that routes the administrative requests to servers.
- 5
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 6
- Enter
kopia
orrestic
as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 7
- Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
- 8
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 9
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 10
- Specify whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.
- 11
- Specify the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.
- 12
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. - 13
- Specify a snapshot location, unless you use CSI snapshots or a File System Backup (FSB) to back up PVs.
- 14
- The snapshot location must be in the same region as the PVs.
- 15
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in thecredentials-velero
file.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.3.5.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.3.6. Configuring the backup storage location with a MD5 checksum algorithm
You can configure the Backup Storage Location (BSL) in the Data Protection Application (DPA) to use a MD5 checksum algorithm for both Amazon Simple Storage Service (Amazon S3) and S3-compatible storage providers. The checksum algorithm calculates the checksum for uploading and downloading objects to Amazon S3. You can use one of the following options to set the checksumAlgorithm
field in the spec.backupLocations.velero.config.checksumAlgorithm
section of the DPA.
-
CRC32
-
CRC32C
-
SHA1
-
SHA256
You can also set the checksumAlgorithm
field to an empty value to skip the MD5 checksum check.
If you do not set a value for the checksumAlgorithm
field, then the default value is set to CRC32
.
Prerequisites
- You have installed the OADP Operator.
- You have configured Amazon S3, or S3-compatible object storage as a backup location.
Procedure
Configure the BSL in the DPA as shown in the following example:
Example Data Protection Application
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-dpa namespace: openshift-adp spec: backupLocations: - name: default velero: config: checksumAlgorithm: "" 1 insecureSkipTLSVerify: "true" profile: "default" region: <bucket_region> s3ForcePathStyle: "true" s3Url: <bucket_url> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: aws configuration: velero: defaultPlugins: - openshift - aws - csi
- 1
- Specify the
checksumAlgorithm
. In this example, thechecksumAlgorithm
field is set to an empty value. You can select an option from the following list:CRC32
,CRC32C
,SHA1
,SHA256
.
If you are using Noobaa as the object storage provider, and you do not set the spec.backupLocations.velero.config.checksumAlgorithm
field in the DPA, an empty value of checksumAlgorithm
is added to the BSL configuration.
The empty value is only added for BSLs that are created using the DPA. This value is not added if you create the BSL by using any other method.
4.4.3.7. Configuring the DPA with more than one BSL
You can configure the DPA with more than one BSL and specify the credentials provided by the cloud provider.
Prerequisites
- You must install the OADP Operator.
- You must create the secrets by using the credentials provided by the cloud provider.
Procedure
Configure the DPA with more than one BSL. See the following example.
Example DPA
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication #... backupLocations: - name: aws 1 velero: provider: aws default: true 2 objectStorage: bucket: <bucket_name> 3 prefix: <prefix> 4 config: region: <region_name> 5 profile: "default" credential: key: cloud name: cloud-credentials 6 - name: odf 7 velero: provider: aws default: false objectStorage: bucket: <bucket_name> prefix: <prefix> config: profile: "default" region: <region_name> s3Url: <url> 8 insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" credential: key: cloud name: <custom_secret_name_odf> 9 #...
- 1
- Specify a name for the first BSL.
- 2
- This parameter indicates that this BSL is the default BSL. If a BSL is not set in the
Backup CR
, the default BSL is used. You can set only one BSL as the default. - 3
- Specify the bucket name.
- 4
- Specify a prefix for Velero backups; for example,
velero
. - 5
- Specify the AWS region for the bucket.
- 6
- Specify the name of the default
Secret
object that you created. - 7
- Specify a name for the second BSL.
- 8
- Specify the URL of the S3 endpoint.
- 9
- Specify the correct name for the
Secret
; for example,custom_secret_name_odf
. If you do not specify aSecret
name, the default name is used.
Specify the BSL to be used in the backup CR. See the following example.
Example backup CR
apiVersion: velero.io/v1 kind: Backup # ... spec: includedNamespaces: - <namespace> 1 storageLocation: <backup_storage_location> 2 defaultVolumesToFsBackup: true
4.4.3.7.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1
- 1
- Add the
csi
default plugin.
4.4.3.7.2. Disabling the node agent in DataProtectionApplication
If you are not using Restic
, Kopia
, or DataMover
for your backups, you can disable the nodeAgent
field in the DataProtectionApplication
custom resource (CR). Before you disable nodeAgent
, ensure the OADP Operator is idle and not running any backups.
Procedure
To disable the
nodeAgent
, set theenable
flag tofalse
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: false 1 uploaderType: kopia # ...
- 1
- Disables the node agent.
To enable the
nodeAgent
, set theenable
flag totrue
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: true 1 uploaderType: kopia # ...
- 1
- Enables the node agent.
You can set up a job to enable and disable the nodeAgent
field in the DataProtectionApplication
CR. For more information, see "Running tasks in pods using jobs".
4.4.4. Configuring the OpenShift API for Data Protection with Microsoft Azure
You install the OpenShift API for Data Protection (OADP) with Microsoft Azure by installing the OADP Operator. The Operator installs Velero 1.14.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You configure Azure for Velero, create a default Secret
, and then install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.4.4.1. Configuring Microsoft Azure
You configure Microsoft Azure for OpenShift API for Data Protection (OADP).
Prerequisites
- You must have the Azure CLI installed.
Tools that use Azure services should always have restricted permissions to make sure that Azure resources are safe. Therefore, instead of having applications sign in as a fully privileged user, Azure offers service principals. An Azure service principal is a name that can be used with applications, hosted services, or automated tools.
This identity is used for access to resources.
- Create a service principal
- Sign in using a service principal and password
- Sign in using a service principal and certificate
- Manage service principal roles
- Create an Azure resource using a service principal
- Reset service principal credentials
For more details, see Create an Azure service principal with Azure CLI.
4.4.4.2. About backup and snapshot locations and their secrets
You specify backup and snapshot locations and their secrets in the DataProtectionApplication
custom resource (CR).
Backup locations
You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.
Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.
Snapshot locations
If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.
If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass
CR to register the CSI driver.
If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.
Secrets
If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret
.
If the backup and snapshot locations use different credentials, you create two secret objects:
-
Custom
Secret
for the backup location, which you specify in theDataProtectionApplication
CR. -
Default
Secret
for the snapshot location, which is not referenced in theDataProtectionApplication
CR.
The Data Protection Application requires a default Secret
. Otherwise, the installation will fail.
If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret
with an empty credentials-velero
file.
4.4.4.2.1. Creating a default Secret
You create a default Secret
if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
The default name of the Secret
is cloud-credentials-azure
.
The DataProtectionApplication
custom resource (CR) requires a default Secret
. Otherwise, the installation will fail. If the name of the backup location Secret
is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret
with the default name by using an empty credentials-velero
file.
Prerequisites
- Your object storage and cloud storage, if any, must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.4.4.2.2. Creating secrets for different credentials
If your backup and snapshot locations use different credentials, you must create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials-azure
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: config: resourceGroup: <azure_resource_group> storageAccount: <azure_storage_account_id> subscriptionId: <azure_subscription_id> storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY credential: key: cloud name: <custom_secret> 1 provider: azure default: true objectStorage: bucket: <bucket_name> prefix: <prefix> snapshotLocations: - velero: config: resourceGroup: <azure_resource_group> subscriptionId: <azure_subscription_id> incremental: "true" provider: azure
- 1
- Backup location
Secret
with custom name.
4.4.4.3. Configuring the Data Protection Application
You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.
4.4.4.3.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Use the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.
For more details, see Configuring node agents and node labels.
4.4.4.3.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 # ...
4.4.4.3.2.1. Using CA certificates with the velero command aliased for Velero deployment
You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. You must have the OpenShift CLI (
oc
) installed.To use an aliased Velero command, run the following command:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
Check that the alias is working by running the following command:
Example
$ velero version Client: Version: v1.12.1-OADP Git commit: - Server: Version: v1.12.1-OADP
To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:
$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}') $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
To fetch the backup logs, run the following command:
$ velero backup logs <backup_name> --cacert /tmp/<your_cacert.txt>
You can use these logs to view failures and warnings for the resources that you cannot back up.
-
If the Velero pod restarts, the
/tmp/your-cacert.txt
file disappears, and you must re-create the/tmp/your-cacert.txt
file by re-running the commands from the previous step. You can check if the
/tmp/your-cacert.txt
file still exists, in the file location where you stored it, by running the following command:$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt" /tmp/your-cacert.txt
In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.
4.4.4.4. Installing the Data Protection Application 1.2 and earlier
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-azure
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.NoteVelero creates a secret named
velero-repo-credentials
in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update isData[repository-password]
.After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is
velero-repo-credentials
, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password invelero-repo-credentials
, and therefore, Velero will not be able to connect with the older backups.
-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - azure - openshift 1 resourceTimeout: 10m 2 restic: enable: true 3 podConfig: nodeSelector: <node_selector> 4 backupLocations: - velero: config: resourceGroup: <azure_resource_group> 5 storageAccount: <azure_storage_account_id> 6 subscriptionId: <azure_subscription_id> 7 storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY credential: key: cloud name: cloud-credentials-azure 8 provider: azure default: true objectStorage: bucket: <bucket_name> 9 prefix: <prefix> 10 snapshotLocations: 11 - velero: config: resourceGroup: <azure_resource_group> subscriptionId: <azure_subscription_id> incremental: "true" name: default provider: azure credential: key: cloud name: cloud-credentials-azure 12
- 1
- The
openshift
plugin is mandatory. - 2
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 3
- Set this value to
false
if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. In OADP version 1.1, addspec.defaultVolumesToRestic: true
to theBackup
CR. - 4
- Specify on which nodes Restic is available. By default, Restic runs on all nodes.
- 5
- Specify the Azure resource group.
- 6
- Specify the Azure storage account ID.
- 7
- Specify the Azure subscription ID.
- 8
- If you do not specify this value, the default name,
cloud-credentials-azure
, is used. If you specify a custom name, the custom name is used for the backup location. - 9
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 10
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 11
- You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
- 12
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials-azure
, is used. If you specify a custom name, the custom name is used for the backup location.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.4.5. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-azure
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. -
Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.
NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp 1 spec: configuration: velero: defaultPlugins: - azure - openshift 2 resourceTimeout: 10m 3 nodeAgent: 4 enable: true 5 uploaderType: kopia 6 podConfig: nodeSelector: <node_selector> 7 backupLocations: - velero: config: resourceGroup: <azure_resource_group> 8 storageAccount: <azure_storage_account_id> 9 subscriptionId: <azure_subscription_id> 10 storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY credential: key: cloud name: cloud-credentials-azure 11 provider: azure default: true objectStorage: bucket: <bucket_name> 12 prefix: <prefix> 13 snapshotLocations: 14 - velero: config: resourceGroup: <azure_resource_group> subscriptionId: <azure_subscription_id> incremental: "true" name: default provider: azure credential: key: cloud name: cloud-credentials-azure 15
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- The
openshift
plugin is mandatory. - 3
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 4
- The administrative agent that routes the administrative requests to servers.
- 5
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 6
- Enter
kopia
orrestic
as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 7
- Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
- 8
- Specify the Azure resource group.
- 9
- Specify the Azure storage account ID.
- 10
- Specify the Azure subscription ID.
- 11
- If you do not specify this value, the default name,
cloud-credentials-azure
, is used. If you specify a custom name, the custom name is used for the backup location. - 12
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 13
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 14
- You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
- 15
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials-azure
, is used. If you specify a custom name, the custom name is used for the backup location.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.4.5.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.4.5.2. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1
- 1
- Add the
csi
default plugin.
4.4.4.5.3. Disabling the node agent in DataProtectionApplication
If you are not using Restic
, Kopia
, or DataMover
for your backups, you can disable the nodeAgent
field in the DataProtectionApplication
custom resource (CR). Before you disable nodeAgent
, ensure the OADP Operator is idle and not running any backups.
Procedure
To disable the
nodeAgent
, set theenable
flag tofalse
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: false 1 uploaderType: kopia # ...
- 1
- Disables the node agent.
To enable the
nodeAgent
, set theenable
flag totrue
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: true 1 uploaderType: kopia # ...
- 1
- Enables the node agent.
You can set up a job to enable and disable the nodeAgent
field in the DataProtectionApplication
CR. For more information, see "Running tasks in pods using jobs".
4.4.5. Configuring the OpenShift API for Data Protection with Google Cloud Platform
You install the OpenShift API for Data Protection (OADP) with Google Cloud Platform (GCP) by installing the OADP Operator. The Operator installs Velero 1.14.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You configure GCP for Velero, create a default Secret
, and then install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.4.5.1. Configuring Google Cloud Platform
You configure Google Cloud Platform (GCP) for the OpenShift API for Data Protection (OADP).
Prerequisites
-
You must have the
gcloud
andgsutil
CLI tools installed. See the Google cloud documentation for details.
Procedure
Log in to GCP:
$ gcloud auth login
Set the
BUCKET
variable:$ BUCKET=<bucket> 1
- 1
- Specify your bucket name.
Create the storage bucket:
$ gsutil mb gs://$BUCKET/
Set the
PROJECT_ID
variable to your active project:$ PROJECT_ID=$(gcloud config get-value project)
Create a service account:
$ gcloud iam service-accounts create velero \ --display-name "Velero service account"
List your service accounts:
$ gcloud iam service-accounts list
Set the
SERVICE_ACCOUNT_EMAIL
variable to match itsemail
value:$ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \ --filter="displayName:Velero service account" \ --format 'value(email)')
Attach the policies to give the
velero
user the minimum necessary permissions:$ ROLE_PERMISSIONS=( compute.disks.get compute.disks.create compute.disks.createSnapshot compute.snapshots.get compute.snapshots.create compute.snapshots.useReadOnly compute.snapshots.delete compute.zones.get storage.objects.create storage.objects.delete storage.objects.get storage.objects.list iam.serviceAccounts.signBlob )
Create the
velero.server
custom role:$ gcloud iam roles create velero.server \ --project $PROJECT_ID \ --title "Velero Server" \ --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
Add IAM policy binding to the project:
$ gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \ --role projects/$PROJECT_ID/roles/velero.server
Update the IAM service account:
$ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}
Save the IAM service account keys to the
credentials-velero
file in the current directory:$ gcloud iam service-accounts keys create credentials-velero \ --iam-account $SERVICE_ACCOUNT_EMAIL
You use the
credentials-velero
file to create aSecret
object for GCP before you install the Data Protection Application.
4.4.5.2. About backup and snapshot locations and their secrets
You specify backup and snapshot locations and their secrets in the DataProtectionApplication
custom resource (CR).
Backup locations
You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.
Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.
Snapshot locations
If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.
If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass
CR to register the CSI driver.
If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.
Secrets
If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret
.
If the backup and snapshot locations use different credentials, you create two secret objects:
-
Custom
Secret
for the backup location, which you specify in theDataProtectionApplication
CR. -
Default
Secret
for the snapshot location, which is not referenced in theDataProtectionApplication
CR.
The Data Protection Application requires a default Secret
. Otherwise, the installation will fail.
If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret
with an empty credentials-velero
file.
4.4.5.2.1. Creating a default Secret
You create a default Secret
if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
The default name of the Secret
is cloud-credentials-gcp
.
The DataProtectionApplication
custom resource (CR) requires a default Secret
. Otherwise, the installation will fail. If the name of the backup location Secret
is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret
with the default name by using an empty credentials-velero
file.
Prerequisites
- Your object storage and cloud storage, if any, must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.4.5.2.2. Creating secrets for different credentials
If your backup and snapshot locations use different credentials, you must create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials-gcp
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: provider: gcp default: true credential: key: cloud name: <custom_secret> 1 objectStorage: bucket: <bucket_name> prefix: <prefix> snapshotLocations: - velero: provider: gcp default: true config: project: <project> snapshotLocation: us-west1
- 1
- Backup location
Secret
with custom name.
4.4.5.3. Configuring the Data Protection Application
You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.
4.4.5.3.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Use the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.
For more details, see Configuring node agents and node labels.
4.4.5.3.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 # ...
4.4.5.3.2.1. Using CA certificates with the velero command aliased for Velero deployment
You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. You must have the OpenShift CLI (
oc
) installed.To use an aliased Velero command, run the following command:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
Check that the alias is working by running the following command:
Example
$ velero version Client: Version: v1.12.1-OADP Git commit: - Server: Version: v1.12.1-OADP
To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:
$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}') $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
To fetch the backup logs, run the following command:
$ velero backup logs <backup_name> --cacert /tmp/<your_cacert.txt>
You can use these logs to view failures and warnings for the resources that you cannot back up.
-
If the Velero pod restarts, the
/tmp/your-cacert.txt
file disappears, and you must re-create the/tmp/your-cacert.txt
file by re-running the commands from the previous step. You can check if the
/tmp/your-cacert.txt
file still exists, in the file location where you stored it, by running the following command:$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt" /tmp/your-cacert.txt
In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.
4.4.5.4. Installing the Data Protection Application 1.2 and earlier
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-gcp
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.NoteVelero creates a secret named
velero-repo-credentials
in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update isData[repository-password]
.After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is
velero-repo-credentials
, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password invelero-repo-credentials
, and therefore, Velero will not be able to connect with the older backups.
-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - gcp - openshift 1 resourceTimeout: 10m 2 restic: enable: true 3 podConfig: nodeSelector: <node_selector> 4 backupLocations: - velero: provider: gcp default: true credential: key: cloud 5 name: cloud-credentials-gcp 6 objectStorage: bucket: <bucket_name> 7 prefix: <prefix> 8 snapshotLocations: 9 - velero: provider: gcp default: true config: project: <project> snapshotLocation: us-west1 10 credential: key: cloud name: cloud-credentials-gcp 11
- 1
- The
openshift
plugin is mandatory. - 2
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 3
- Set this value to
false
if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. In OADP version 1.1, addspec.defaultVolumesToRestic: true
to theBackup
CR. - 4
- Specify on which nodes Restic is available. By default, Restic runs on all nodes.
- 5
- Secret key that contains credentials. For Google workload identity federation cloud authentication use
service_account.json
. - 6
- Secret name that contains credentials. If you do not specify this value, the default name,
cloud-credentials-gcp
, is used. - 7
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 8
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 9
- Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.
- 10
- The snapshot location must be in the same region as the PVs.
- 11
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials-gcp
, is used. If you specify a custom name, the custom name is used for the snapshot location.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.5.5. Google workload identity federation cloud authentication
Applications running outside Google Cloud use service account keys, such as usernames and passwords, to gain access to Google Cloud resources. These service account keys might become a security risk if they are not properly managed.
With Google’s workload identity federation, you can use Identity and Access Management (IAM) to offer IAM roles, including the ability to impersonate service accounts, to external identities. This eliminates the maintenance and security risks associated with service account keys.
Workload identity federation handles encrypting and decrypting certificates, extracting user attributes, and validation. Identity federation externalizes authentication, passing it over to Security Token Services (STS), and reduces the demands on individual developers. Authorization and controlling access to resources remain the responsibility of the application.
Google workload identity federation is available for OADP 1.3.x and later.
When backing up volumes, OADP on GCP with Google workload identity federation authentication only supports CSI snapshots.
OADP on GCP with Google workload identity federation authentication does not support Volume Snapshot Locations (VSL) backups. For more details, see Google workload identity federation known issues.
If you do not use Google workload identity federation cloud authentication, continue to Installing the Data Protection Application.
Prerequisites
- You have installed a cluster in manual mode with GCP Workload Identity configured.
-
You have access to the Cloud Credential Operator utility (
ccoctl
) and to the associated workload identity pool.
Procedure
Create an
oadp-credrequest
directory by running the following command:$ mkdir -p oadp-credrequest
Create a
CredentialsRequest.yaml
file as following:echo 'apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: oadp-operator-credentials namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: GCPProviderSpec permissions: - compute.disks.get - compute.disks.create - compute.disks.createSnapshot - compute.snapshots.get - compute.snapshots.create - compute.snapshots.useReadOnly - compute.snapshots.delete - compute.zones.get - storage.objects.create - storage.objects.delete - storage.objects.get - storage.objects.list - iam.serviceAccounts.signBlob skipServiceCheck: true secretRef: name: cloud-credentials-gcp namespace: <OPERATOR_INSTALL_NS> serviceAccountNames: - velero ' > oadp-credrequest/credrequest.yaml
Use the
ccoctl
utility to process theCredentialsRequest
objects in theoadp-credrequest
directory by running the following command:$ ccoctl gcp create-service-accounts \ --name=<name> \ --project=<gcp_project_id> \ --credentials-requests-dir=oadp-credrequest \ --workload-identity-pool=<pool_id> \ --workload-identity-provider=<provider_id>
The
manifests/openshift-adp-cloud-credentials-gcp-credentials.yaml
file is now available to use in the following steps.Create a namespace by running the following command:
$ oc create namespace <OPERATOR_INSTALL_NS>
Apply the credentials to the namespace by running the following command:
$ oc apply -f manifests/openshift-adp-cloud-credentials-gcp-credentials.yaml
4.4.5.5.1. Google workload identity federation known issues
-
Volume Snapshot Location (VSL) backups finish with a
PartiallyFailed
phase when GCP workload identity federation is configured. Google workload identity federation authentication does not support VSL backups.
4.4.5.6. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-gcp
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. -
Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.
NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: <OPERATOR_INSTALL_NS> 1 spec: configuration: velero: defaultPlugins: - gcp - openshift 2 resourceTimeout: 10m 3 nodeAgent: 4 enable: true 5 uploaderType: kopia 6 podConfig: nodeSelector: <node_selector> 7 backupLocations: - velero: provider: gcp default: true credential: key: cloud 8 name: cloud-credentials-gcp 9 objectStorage: bucket: <bucket_name> 10 prefix: <prefix> 11 snapshotLocations: 12 - velero: provider: gcp default: true config: project: <project> snapshotLocation: us-west1 13 credential: key: cloud name: cloud-credentials-gcp 14 backupImages: true 15
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- The
openshift
plugin is mandatory. - 3
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 4
- The administrative agent that routes the administrative requests to servers.
- 5
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 6
- Enter
kopia
orrestic
as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 7
- Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
- 8
- Secret key that contains credentials. For Google workload identity federation cloud authentication use
service_account.json
. - 9
- Secret name that contains credentials. If you do not specify this value, the default name,
cloud-credentials-gcp
, is used. - 10
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 11
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes. - 12
- Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.
- 13
- The snapshot location must be in the same region as the PVs.
- 14
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials-gcp
, is used. If you specify a custom name, the custom name is used for the backup location. - 15
- Google workload identity federation supports internal image backup. Set this field to
false
if you do not want to use image backup.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.5.6.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.5.6.2. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1
- 1
- Add the
csi
default plugin.
4.4.5.6.3. Disabling the node agent in DataProtectionApplication
If you are not using Restic
, Kopia
, or DataMover
for your backups, you can disable the nodeAgent
field in the DataProtectionApplication
custom resource (CR). Before you disable nodeAgent
, ensure the OADP Operator is idle and not running any backups.
Procedure
To disable the
nodeAgent
, set theenable
flag tofalse
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: false 1 uploaderType: kopia # ...
- 1
- Disables the node agent.
To enable the
nodeAgent
, set theenable
flag totrue
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: true 1 uploaderType: kopia # ...
- 1
- Enables the node agent.
You can set up a job to enable and disable the nodeAgent
field in the DataProtectionApplication
CR. For more information, see "Running tasks in pods using jobs".
4.4.6. Configuring the OpenShift API for Data Protection with Multicloud Object Gateway
You install the OpenShift API for Data Protection (OADP) with Multicloud Object Gateway (MCG) by installing the OADP Operator. The Operator installs Velero 1.14.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You configure Multicloud Object Gateway as a backup location. MCG is a component of OpenShift Data Foundation. You configure MCG as a backup location in the DataProtectionApplication
custom resource (CR).
The CloudStorage
API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You create a Secret
for the backup location and then you install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.
4.4.6.1. Retrieving Multicloud Object Gateway credentials
You must retrieve the Multicloud Object Gateway (MCG) credentials, which you need to create a Secret
custom resource (CR) for the OpenShift API for Data Protection (OADP).
Although the MCG Operator is deprecated, the MCG plugin is still available for OpenShift Data Foundation. To download the plugin, browse to Download Red Hat OpenShift Data Foundation and download the appropriate MCG plugin for your operating system.
Prerequisites
- You must deploy OpenShift Data Foundation by using the appropriate Red Hat OpenShift Data Foundation deployment guide.
Procedure
-
Obtain the S3 endpoint,
AWS_ACCESS_KEY_ID
, andAWS_SECRET_ACCESS_KEY
by running thedescribe
command on theNooBaa
custom resource. Create a
credentials-velero
file:$ cat << EOF > ./credentials-velero [default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> EOF
You use the
credentials-velero
file to create aSecret
object when you install the Data Protection Application.
4.4.6.2. About backup and snapshot locations and their secrets
You specify backup and snapshot locations and their secrets in the DataProtectionApplication
custom resource (CR).
Backup locations
You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.
Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.
Snapshot locations
If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.
If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass
CR to register the CSI driver.
If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.
Secrets
If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret
.
If the backup and snapshot locations use different credentials, you create two secret objects:
-
Custom
Secret
for the backup location, which you specify in theDataProtectionApplication
CR. -
Default
Secret
for the snapshot location, which is not referenced in theDataProtectionApplication
CR.
The Data Protection Application requires a default Secret
. Otherwise, the installation will fail.
If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret
with an empty credentials-velero
file.
4.4.6.2.1. Creating a default Secret
You create a default Secret
if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
The default name of the Secret
is cloud-credentials
.
The DataProtectionApplication
custom resource (CR) requires a default Secret
. Otherwise, the installation will fail. If the name of the backup location Secret
is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret
with the default name by using an empty credentials-velero
file.
Prerequisites
- Your object storage and cloud storage, if any, must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.4.6.2.2. Creating secrets for different credentials
If your backup and snapshot locations use different credentials, you must create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: config: profile: "default" region: <region_name> 1 s3Url: <url> insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" provider: aws default: true credential: key: cloud name: <custom_secret> 2 objectStorage: bucket: <bucket_name> prefix: <prefix>
4.4.6.3. Configuring the Data Protection Application
You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.
4.4.6.3.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Use the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.
For more details, see Configuring node agents and node labels.
4.4.6.3.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 # ...
4.4.6.3.2.1. Using CA certificates with the velero command aliased for Velero deployment
You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. You must have the OpenShift CLI (
oc
) installed.To use an aliased Velero command, run the following command:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
Check that the alias is working by running the following command:
Example
$ velero version Client: Version: v1.12.1-OADP Git commit: - Server: Version: v1.12.1-OADP
To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:
$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}') $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
To fetch the backup logs, run the following command:
$ velero backup logs <backup_name> --cacert /tmp/<your_cacert.txt>
You can use these logs to view failures and warnings for the resources that you cannot back up.
-
If the Velero pod restarts, the
/tmp/your-cacert.txt
file disappears, and you must re-create the/tmp/your-cacert.txt
file by re-running the commands from the previous step. You can check if the
/tmp/your-cacert.txt
file still exists, in the file location where you stored it, by running the following command:$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt" /tmp/your-cacert.txt
In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.
4.4.6.4. Installing the Data Protection Application 1.2 and earlier
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.NoteVelero creates a secret named
velero-repo-credentials
in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update isData[repository-password]
.After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is
velero-repo-credentials
, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password invelero-repo-credentials
, and therefore, Velero will not be able to connect with the older backups.
-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - aws 1 - openshift 2 resourceTimeout: 10m 3 restic: enable: true 4 podConfig: nodeSelector: <node_selector> 5 backupLocations: - velero: config: profile: "default" region: <region_name> 6 s3Url: <url> 7 insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" provider: aws default: true credential: key: cloud name: cloud-credentials 8 objectStorage: bucket: <bucket_name> 9 prefix: <prefix> 10
- 1
- An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is
aws
. For Azure and GCP object stores, theazure
orgcp
plugin is required. - 2
- The
openshift
plugin is mandatory. - 3
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 4
- Set this value to
false
if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. In OADP version 1.1, addspec.defaultVolumesToRestic: true
to theBackup
CR. - 5
- Specify on which nodes Restic is available. By default, Restic runs on all nodes.
- 6
- Specify the region, following the naming convention of the documentation of your object storage server.
- 7
- Specify the URL of the S3 endpoint.
- 8
- If you do not specify this value, the default name,
cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. - 9
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 10
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.6.5. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. -
Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.
NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp 1 spec: configuration: velero: defaultPlugins: - aws 2 - openshift 3 resourceTimeout: 10m 4 nodeAgent: 5 enable: true 6 uploaderType: kopia 7 podConfig: nodeSelector: <node_selector> 8 backupLocations: - velero: config: profile: "default" region: <region_name> 9 s3Url: <url> 10 insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" provider: aws default: true credential: key: cloud name: cloud-credentials 11 objectStorage: bucket: <bucket_name> 12 prefix: <prefix> 13
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is
aws
. For Azure and GCP object stores, theazure
orgcp
plugin is required. - 3
- The
openshift
plugin is mandatory. - 4
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 5
- The administrative agent that routes the administrative requests to servers.
- 6
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 7
- Enter
kopia
orrestic
as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 8
- Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
- 9
- Specify the region, following the naming convention of the documentation of your object storage server.
- 10
- Specify the URL of the S3 endpoint.
- 11
- Specify the name of the
Secret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. - 12
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 13
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.6.5.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.6.5.2. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1
- 1
- Add the
csi
default plugin.
4.4.6.5.3. Disabling the node agent in DataProtectionApplication
If you are not using Restic
, Kopia
, or DataMover
for your backups, you can disable the nodeAgent
field in the DataProtectionApplication
custom resource (CR). Before you disable nodeAgent
, ensure the OADP Operator is idle and not running any backups.
Procedure
To disable the
nodeAgent
, set theenable
flag tofalse
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: false 1 uploaderType: kopia # ...
- 1
- Disables the node agent.
To enable the
nodeAgent
, set theenable
flag totrue
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: true 1 uploaderType: kopia # ...
- 1
- Enables the node agent.
You can set up a job to enable and disable the nodeAgent
field in the DataProtectionApplication
CR. For more information, see "Running tasks in pods using jobs".
4.4.7. Configuring the OpenShift API for Data Protection with OpenShift Data Foundation
You install the OpenShift API for Data Protection (OADP) with OpenShift Data Foundation by installing the OADP Operator and configuring a backup location and a snapshot location. Then, you install the Data Protection Application.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You can configure Multicloud Object Gateway or any AWS S3-compatible object storage as a backup location.
The CloudStorage
API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
You create a Secret
for the backup location and then you install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.
4.4.7.1. About backup and snapshot locations and their secrets
You specify backup and snapshot locations and their secrets in the DataProtectionApplication
custom resource (CR).
Backup locations
You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.
Velero backs up OpenShift Container Platform resources, Kubernetes objects, and internal images as an archive file on object storage.
Snapshot locations
If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.
If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass
CR to register the CSI driver.
If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.
Secrets
If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret
.
If the backup and snapshot locations use different credentials, you create two secret objects:
-
Custom
Secret
for the backup location, which you specify in theDataProtectionApplication
CR. -
Default
Secret
for the snapshot location, which is not referenced in theDataProtectionApplication
CR.
The Data Protection Application requires a default Secret
. Otherwise, the installation will fail.
If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret
with an empty credentials-velero
file.
Additional resources
4.4.7.1.1. Creating a default Secret
You create a default Secret
if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
The default name of the Secret
is cloud-credentials
, unless your backup storage provider has a default plugin, such as aws
, azure
, or gcp
. In that case, the default name is specified in the provider-specific OADP installation procedure.
The DataProtectionApplication
custom resource (CR) requires a default Secret
. Otherwise, the installation will fail. If the name of the backup location Secret
is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret
with the default name by using an empty credentials-velero
file.
Prerequisites
- Your object storage and cloud storage, if any, must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.4.7.1.2. Creating secrets for different credentials
If your backup and snapshot locations use different credentials, you must create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: provider: <provider> default: true credential: key: cloud name: <custom_secret> 1 objectStorage: bucket: <bucket_name> prefix: <prefix>
- 1
- Backup location
Secret
with custom name.
4.4.7.2. Configuring the Data Protection Application
You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.
4.4.7.2.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Use the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.
For more details, see Configuring node agents and node labels.
4.4.7.2.1.1. Adjusting Ceph CPU and memory requirements based on collected data
The following recommendations are based on observations of performance made in the scale and performance lab. The changes are specifically related to Red Hat OpenShift Data Foundation (ODF). If working with ODF, consult the appropriate tuning guides for official recommendations.
4.4.7.2.1.1.1. CPU and memory requirement for configurations
Backup and restore operations require large amounts of CephFS PersistentVolumes
(PVs). To avoid Ceph MDS pods restarting with an out-of-memory
(OOM) error, the following configuration is suggested:
Configuration types | Request | Max limit |
---|---|---|
CPU | Request changed to 3 | Max limit to 3 |
Memory | Request changed to 8 Gi | Max limit to 128 Gi |
4.4.7.2.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 # ...
4.4.7.2.2.1. Using CA certificates with the velero command aliased for Velero deployment
You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. You must have the OpenShift CLI (
oc
) installed.To use an aliased Velero command, run the following command:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
Check that the alias is working by running the following command:
Example
$ velero version Client: Version: v1.12.1-OADP Git commit: - Server: Version: v1.12.1-OADP
To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:
$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}') $ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"
$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt
To fetch the backup logs, run the following command:
$ velero backup logs <backup_name> --cacert /tmp/<your_cacert.txt>
You can use these logs to view failures and warnings for the resources that you cannot back up.
-
If the Velero pod restarts, the
/tmp/your-cacert.txt
file disappears, and you must re-create the/tmp/your-cacert.txt
file by re-running the commands from the previous step. You can check if the
/tmp/your-cacert.txt
file still exists, in the file location where you stored it, by running the following command:$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt" /tmp/your-cacert.txt
In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.
4.4.7.3. Installing the Data Protection Application 1.2 and earlier
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.NoteVelero creates a secret named
velero-repo-credentials
in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update isData[repository-password]
.After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is
velero-repo-credentials
, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password invelero-repo-credentials
, and therefore, Velero will not be able to connect with the older backups.
-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - aws 1 - kubevirt 2 - csi 3 - openshift 4 resourceTimeout: 10m 5 restic: enable: true 6 podConfig: nodeSelector: <node_selector> 7 backupLocations: - velero: provider: gcp 8 default: true credential: key: cloud name: <default_secret> 9 objectStorage: bucket: <bucket_name> 10 prefix: <prefix> 11
- 1
- An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is
aws
. For Azure and GCP object stores, theazure
orgcp
plugin is required. - 2
- Optional: The
kubevirt
plugin is used with OpenShift Virtualization. - 3
- Specify the
csi
default plugin if you use CSI snapshots to back up PVs. Thecsi
plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location. - 4
- The
openshift
plugin is mandatory. - 5
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 6
- Set this value to
false
if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. In OADP version 1.1, addspec.defaultVolumesToRestic: true
to theBackup
CR. - 7
- Specify on which nodes Restic is available. By default, Restic runs on all nodes.
- 8
- Specify the backup provider.
- 9
- Specify the correct default name for the
Secret
, for example,cloud-credentials-gcp
, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify aSecret
name, the default name is used. - 10
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 11
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.7.4. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. -
Secret
with another custom name for the snapshot location. You add thisSecret
to theDataProtectionApplication
CR.
NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.-
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp 1 spec: configuration: velero: defaultPlugins: - aws 2 - kubevirt 3 - csi 4 - openshift 5 resourceTimeout: 10m 6 nodeAgent: 7 enable: true 8 uploaderType: kopia 9 podConfig: nodeSelector: <node_selector> 10 backupLocations: - velero: provider: gcp 11 default: true credential: key: cloud name: <default_secret> 12 objectStorage: bucket: <bucket_name> 13 prefix: <prefix> 14
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- An object store plugin corresponding to your storage locations is required. For all S3 providers, the required plugin is
aws
. For Azure and GCP object stores, theazure
orgcp
plugin is required. - 3
- Optional: The
kubevirt
plugin is used with OpenShift Virtualization. - 4
- Specify the
csi
default plugin if you use CSI snapshots to back up PVs. Thecsi
plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location. - 5
- The
openshift
plugin is mandatory. - 6
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 7
- The administrative agent that routes the administrative requests to servers.
- 8
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 9
- Enter
kopia
orrestic
as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 10
- Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
- 11
- Specify the backup provider.
- 12
- Specify the correct default name for the
Secret
, for example,cloud-credentials-gcp
, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify aSecret
name, the default name is used. - 13
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 14
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.7.4.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.7.4.2. Creating an Object Bucket Claim for disaster recovery on OpenShift Data Foundation
If you use cluster storage for your Multicloud Object Gateway (MCG) bucket backupStorageLocation
on OpenShift Data Foundation, create an Object Bucket Claim (OBC) using the OpenShift Web Console.
Failure to configure an Object Bucket Claim (OBC) might lead to backups not being available.
Unless specified otherwise, "NooBaa" refers to the open source project that provides lightweight object storage, while "Multicloud Object Gateway (MCG)" refers to the Red Hat distribution of NooBaa.
For more information on the MCG, see Accessing the Multicloud Object Gateway with your applications.
Procedure
- Create an Object Bucket Claim (OBC) using the OpenShift web console as described in Creating an Object Bucket Claim using the OpenShift Web Console.
4.4.7.4.3. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1
- 1
- Add the
csi
default plugin.
4.4.7.4.4. Disabling the node agent in DataProtectionApplication
If you are not using Restic
, Kopia
, or DataMover
for your backups, you can disable the nodeAgent
field in the DataProtectionApplication
custom resource (CR). Before you disable nodeAgent
, ensure the OADP Operator is idle and not running any backups.
Procedure
To disable the
nodeAgent
, set theenable
flag tofalse
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: false 1 uploaderType: kopia # ...
- 1
- Disables the node agent.
To enable the
nodeAgent
, set theenable
flag totrue
. See the following example:Example
DataProtectionApplication
CR# ... configuration: nodeAgent: enable: true 1 uploaderType: kopia # ...
- 1
- Enables the node agent.
You can set up a job to enable and disable the nodeAgent
field in the DataProtectionApplication
CR. For more information, see "Running tasks in pods using jobs".
4.4.8. Configuring the OpenShift API for Data Protection with OpenShift Virtualization
You can install the OpenShift API for Data Protection (OADP) with OpenShift Virtualization by installing the OADP Operator and configuring a backup location. Then, you can install the Data Protection Application.
Back up and restore virtual machines by using the OpenShift API for Data Protection.
OpenShift API for Data Protection with OpenShift Virtualization supports the following backup and restore storage options:
- Container Storage Interface (CSI) backups
- Container Storage Interface (CSI) backups with DataMover
The following storage options are excluded:
- File system backup and restore
- Volume snapshot backups and restores
For more information, see Backing up applications with File System Backup: Kopia or Restic.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.4.8.1. Installing and configuring OADP with OpenShift Virtualization
As a cluster administrator, you install OADP by installing the OADP Operator.
The latest version of the OADP Operator installs Velero 1.14.
Prerequisites
-
Access to the cluster as a user with the
cluster-admin
role.
Procedure
- Install the OADP Operator according to the instructions for your storage provider.
-
Install the Data Protection Application (DPA) with the
kubevirt
andopenshift
OADP plugins. Back up virtual machines by creating a
Backup
custom resource (CR).WarningRed Hat support is limited to only the following options:
- CSI backups
- CSI backups with DataMover.
You restore the Backup
CR by creating a Restore
CR.
4.4.8.2. Installing the Data Protection Application 1.3
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
Procedure
- Click Operators → Installed Operators and select the OADP Operator.
- Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp 1 spec: configuration: velero: defaultPlugins: - kubevirt 2 - gcp 3 - csi 4 - openshift 5 resourceTimeout: 10m 6 nodeAgent: 7 enable: true 8 uploaderType: kopia 9 podConfig: nodeSelector: <node_selector> 10 backupLocations: - velero: provider: gcp 11 default: true credential: key: cloud name: <default_secret> 12 objectStorage: bucket: <bucket_name> 13 prefix: <prefix> 14
- 1
- The default namespace for OADP is
openshift-adp
. The namespace is a variable and is configurable. - 2
- The
kubevirt
plugin is mandatory for OpenShift Virtualization. - 3
- Specify the plugin for the backup provider, for example,
gcp
, if it exists. - 4
- The
csi
plugin is mandatory for backing up PVs with CSI snapshots. Thecsi
plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location. - 5
- The
openshift
plugin is mandatory. - 6
- Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
- 7
- The administrative agent that routes the administrative requests to servers.
- 8
- Set this value to
true
if you want to enablenodeAgent
and perform File System Backup. - 9
- Enter
kopia
as your uploader to use the Built-in DataMover. ThenodeAgent
deploys a daemon set, which means that thenodeAgent
pods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: true
to theBackup
CR. - 10
- Specify the nodes on which Kopia are available. By default, Kopia runs on all nodes.
- 11
- Specify the backup provider.
- 12
- Specify the correct default name for the
Secret
, for example,cloud-credentials-gcp
, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify aSecret
name, the default name is used. - 13
- Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
- 14
- Specify a prefix for Velero backups, for example,
velero
, if the bucket is used for multiple purposes.
- Click Create.
Verification
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
Verify that the
DataProtectionApplication
(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
Example output
{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
-
Verify the
type
is set toReconciled
. Verify the backup storage location and confirm that the
PHASE
isAvailable
by running the following command:$ oc get backupStorageLocation -n openshift-adp
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
4.4.8.2.1. Configuring node agents and node labels
The DPA of OADP uses the nodeSelector
field to select which nodes can run the node agent. The nodeSelector
field is the simplest recommended form of node selection constraint.
Any label specified must match the labels on each node.
The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""
Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector
, which you used for labeling nodes. For example:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""
The following example is an anti-pattern of nodeSelector
and does not work unless both labels, 'node-role.kubernetes.io/infra: ""'
and 'node-role.kubernetes.io/worker: ""'
, are on the node:
configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
4.4.8.3. About incremental back up support
OADP supports incremental backups of block
and Filesystem
persistent volumes for both containerized, and OpenShift Virtualization workloads. The following table summarizes the support for File System Backup (FSB), Container Storage Interface (CSI), and CSI Data Mover:
Volume mode | FSB - Restic | FSB - Kopia | CSI | CSI Data Mover |
---|---|---|---|---|
Filesystem | S [1], I [2] | S [1], I [2] | S [1] | S [1], I [2] |
Block | N [3] | N [3] | S [1] | S [1], I [2] |
Volume mode | FSB - Restic | FSB - Kopia | CSI | CSI Data Mover |
---|---|---|---|---|
Filesystem | N [3] | N [3] | S [1] | S [1], I [2] |
Block | N [3] | N [3] | S [1] | S [1], I [2] |
- Backup supported
- Incremental backup supported
- Not supported
The CSI Data Mover backups use Kopia regardless of uploaderType
.
Red Hat only supports the combination of OADP versions 1.3.0 and later, and OpenShift Virtualization versions 4.14 and later.
OADP versions before 1.3.0 are not supported for back up and restore of OpenShift Virtualization.
4.4.9. Configuring the OpenShift API for Data Protection (OADP) with more than one Backup Storage Location
You can configure one or more backup storage locations (BSLs) in the Data Protection Application (DPA). You can also select the location to store the backup in when you create the backup. With this configuration, you can store your backups in the following ways:
- To different regions
- To a different storage provider
OADP supports multiple credentials for configuring more than one BSL, so that you can specify the credentials to use with any BSL.
4.4.9.1. Configuring the DPA with more than one BSL
You can configure the DPA with more than one BSL and specify the credentials provided by the cloud provider.
Prerequisites
- You must install the OADP Operator.
- You must create the secrets by using the credentials provided by the cloud provider.
Procedure
Configure the DPA with more than one BSL. See the following example.
Example DPA
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication #... backupLocations: - name: aws 1 velero: provider: aws default: true 2 objectStorage: bucket: <bucket_name> 3 prefix: <prefix> 4 config: region: <region_name> 5 profile: "default" credential: key: cloud name: cloud-credentials 6 - name: odf 7 velero: provider: aws default: false objectStorage: bucket: <bucket_name> prefix: <prefix> config: profile: "default" region: <region_name> s3Url: <url> 8 insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" credential: key: cloud name: <custom_secret_name_odf> 9 #...
- 1
- Specify a name for the first BSL.
- 2
- This parameter indicates that this BSL is the default BSL. If a BSL is not set in the
Backup CR
, the default BSL is used. You can set only one BSL as the default. - 3
- Specify the bucket name.
- 4
- Specify a prefix for Velero backups; for example,
velero
. - 5
- Specify the AWS region for the bucket.
- 6
- Specify the name of the default
Secret
object that you created. - 7
- Specify a name for the second BSL.
- 8
- Specify the URL of the S3 endpoint.
- 9
- Specify the correct name for the
Secret
; for example,custom_secret_name_odf
. If you do not specify aSecret
name, the default name is used.
Specify the BSL to be used in the backup CR. See the following example.
Example backup CR
apiVersion: velero.io/v1 kind: Backup # ... spec: includedNamespaces: - <namespace> 1 storageLocation: <backup_storage_location> 2 defaultVolumesToFsBackup: true
4.4.9.2. OADP use case for two BSLs
In this use case, you configure the DPA with two storage locations by using two cloud credentials. You back up an application with a database by using the default BSL. OADP stores the backup resources in the default BSL. You then backup the application again by using the second BSL.
Prerequisites
- You must install the OADP Operator.
- You must configure two backup storage locations: AWS S3 and Multicloud Object Gateway (MCG).
- You must have an application with a database deployed on a Red Hat OpenShift cluster.
Procedure
Create the first
Secret
for the AWS S3 storage provider with the default name by running the following command:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=<aws_credentials_file_name> 1
- 1
- Specify the name of the cloud credentials file for AWS S3.
Create the second
Secret
for MCG with a custom name by running the following command:$ oc create secret generic mcg-secret -n openshift-adp --from-file cloud=<MCG_credentials_file_name> 1
- 1
- Specify the name of the cloud credentials file for MCG. Note the name of the
mcg-secret
custom secret.
Configure the DPA with the two BSLs as shown in the following example.
Example DPA
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: two-bsl-dpa namespace: openshift-adp spec: backupLocations: - name: aws velero: config: profile: default region: <region_name> 1 credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> 2 prefix: velero provider: aws - name: mcg velero: config: insecureSkipTLSVerify: "true" profile: noobaa region: <region_name> 3 s3ForcePathStyle: "true" s3Url: <s3_url> 4 credential: key: cloud name: mcg-secret 5 objectStorage: bucket: <bucket_name_mcg> 6 prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - openshift - aws
Create the DPA by running the following command:
$ oc create -f <dpa_file_name> 1
- 1
- Specify the file name of the DPA you configured.
Verify that the DPA has reconciled by running the following command:
$ oc get dpa -o yaml
Verify that the BSLs are available by running the following command:
$ oc get bsl
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT aws Available 5s 3m28s true mcg Available 5s 3m28s
Create a backup CR with the default BSL.
NoteIn the following example, the
storageLocation
field is not specified in the backup CR.Example backup CR
apiVersion: velero.io/v1 kind: Backup metadata: name: test-backup1 namespace: openshift-adp spec: includedNamespaces: - <mysql_namespace> 1 defaultVolumesToFsBackup: true
- 1
- Specify the namespace for the application installed in the cluster.
Create a backup by running the following command:
$ oc apply -f <backup_file_name> 1
- 1
- Specify the name of the backup CR file.
Verify that the backup completed with the default BSL by running the following command:
$ oc get backup <backup_name> -o yaml 1
- 1
- Specify the name of the backup.
Create a backup CR by using MCG as the BSL. In the following example, note that the second
storageLocation
value is specified at the time of backup CR creation.Example backup
CR
apiVersion: velero.io/v1 kind: Backup metadata: name: test-backup1 namespace: openshift-adp spec: includedNamespaces: - <mysql_namespace> 1 storageLocation: mcg 2 defaultVolumesToFsBackup: true
Create a second backup by running the following command:
$ oc apply -f <backup_file_name> 1
- 1
- Specify the name of the backup CR file.
Verify that the backup completed with the storage location as MCG by running the following command:
$ oc get backup <backup_name> -o yaml 1
- 1
- Specify the name of the backup.
Additional resources
4.4.10. Configuring the OpenShift API for Data Protection (OADP) with more than one Volume Snapshot Location
You can configure one or more Volume Snapshot Locations (VSLs) to store the snapshots in different cloud provider regions.
4.4.10.1. Configuring the DPA with more than one VSL
You configure the DPA with more than one VSL and specify the credentials provided by the cloud provider. Make sure that you configure the snapshot location in the same region as the persistent volumes. See the following example.
Example DPA
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication #... snapshotLocations: - velero: config: profile: default region: <region> 1 credential: key: cloud name: cloud-credentials provider: aws - velero: config: profile: default region: <region> credential: key: cloud name: <custom_credential> 2 provider: aws #...
4.5. Uninstalling OADP
4.5.1. Uninstalling the OpenShift API for Data Protection
You uninstall the OpenShift API for Data Protection (OADP) by deleting the OADP Operator. See Deleting Operators from a cluster for details.
4.6. OADP backing up
4.6.1. Backing up applications
Frequent backups might consume storage on the backup storage location. Check the frequency of backups, retention time, and the amount of data of the persistent volumes (PVs) if using non-local backups, for example, S3 buckets. Because all taken backup remains until expired, also check the time to live (TTL) setting of the schedule.
You can back up applications by creating a Backup
custom resource (CR). For more information, see Creating a Backup CR.
-
The
Backup
CR creates backup files for Kubernetes resources and internal images on S3 object storage. -
If your cloud provider has a native snapshot API or supports CSI snapshots, the
Backup
CR backs up persistent volumes (PVs) by creating snapshots. For more information about working with CSI snapshots, see Backing up persistent volumes with CSI snapshots.
For more information about CSI volume snapshots, see CSI volume snapshots.
The CloudStorage
API, which automates the creation of a bucket for object storage, is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The CloudStorage
API is a Technology Preview feature when you use a CloudStorage
object and want OADP to use the CloudStorage
API to automatically create an S3 bucket for use as a BackupStorageLocation
.
The CloudStorage
API supports manually creating a BackupStorageLocation
object by specifying an existing S3 bucket. The CloudStorage
API that creates an S3 bucket automatically is currently only enabled for AWS S3 storage.
- If your cloud provider does not support snapshots or if your applications are on NFS data volumes, you can create backups by using Kopia or Restic. See Backing up applications with File System Backup: Kopia or Restic.
…/.snapshot: read-only file system
error
The …/.snapshot
directory is a snapshot copy directory, which is used by several NFS servers. This directory has read-only access by default, so Velero cannot restore to this directory.
Do not give Velero write access to the .snapshot
directory, and disable client access to this directory.
The OpenShift API for Data Protection (OADP) does not support backing up volume snapshots that were created by other software.
4.6.1.1. Previewing resources before running backup and restore
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete. To preview the resources in advance, you can do a dry run of the backup and restore operations.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
$ velero backup create <backup-name> --snapshot-volumes false 1
- 1
- Specify the value of
--snapshot-volumes
parameter asfalse
.
To know more details about the backup resources, run the following command:
$ velero describe backup <backup_name> --details 1
- 1
- Specify the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
$ velero restore create --from-backup <backup-name> 1
- 1
- Specify the name of the backup created to review the backup resources.
ImportantThe
velero restore create
command creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
$ velero describe restore <restore_name> --details 1
- 1
- Specify the name of the restore.
You can create backup hooks to run commands before or after the backup operation. See Creating backup hooks.
You can schedule backups by creating a Schedule
CR instead of a Backup
CR. See Scheduling backups using Schedule CR.
4.6.1.2. Known issues
OpenShift Container Platform 4.15 enforces a pod security admission (PSA) policy that can hinder the readiness of pods during a Restic restore process.
This issue has been resolved in the OADP 1.1.6 and OADP 1.2.2 releases, therefore it is recommended that users upgrade to these releases.
For more information, see Restic restore partially failing on OCP 4.15 due to changed PSA policy.
4.6.2. Creating a Backup CR
You back up Kubernetes resources, internal images, and persistent volumes (PVs) by creating a Backup
custom resource (CR).
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state. Backup location prerequisites:
- You must have S3 object storage configured for Velero.
-
You must have a backup location configured in the
DataProtectionApplication
CR.
Snapshot location prerequisites:
- Your cloud provider must have a native snapshot API or support Container Storage Interface (CSI) snapshots.
-
For CSI snapshots, you must create a
VolumeSnapshotClass
CR to register the CSI driver. -
You must have a volume location configured in the
DataProtectionApplication
CR.
Procedure
Retrieve the
backupStorageLocations
CRs by entering the following command:$ oc get backupStorageLocations -n openshift-adp
Example output
NAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31m
Create a
Backup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> labels: velero.io/storage-location: default namespace: openshift-adp spec: hooks: {} includedNamespaces: - <namespace> 1 includedResources: [] 2 excludedResources: [] 3 storageLocation: <velero-sample-1> 4 ttl: 720h0m0s labelSelector: 5 matchLabels: app: <label_1> app: <label_2> app: <label_3> orLabelSelectors: 6 - matchLabels: app: <label_1> app: <label_2> app: <label_3>
- 1
- Specify an array of namespaces to back up.
- 2
- Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.
- 3
- Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.
- 4
- Specify the name of the
backupStorageLocations
CR. - 5
- Map of {key,value} pairs of backup resources that have all the specified labels.
- 6
- Map of {key,value} pairs of backup resources that have one or more of the specified labels.
Verify that the status of the
Backup
CR isCompleted
:$ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'
4.6.3. Backing up persistent volumes with CSI snapshots
You back up persistent volumes with Container Storage Interface (CSI) snapshots by editing the VolumeSnapshotClass
custom resource (CR) of the cloud storage before you create the Backup
CR, see CSI volume snapshots.
For more information, see Creating a Backup CR.
Prerequisites
- The cloud provider must support CSI snapshots.
-
You must enable CSI in the
DataProtectionApplication
CR.
Procedure
Add the
metadata.labels.velero.io/csi-volumesnapshot-class: "true"
key-value pair to theVolumeSnapshotClass
CR:Example configuration file
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: <volume_snapshot_class_name> labels: velero.io/csi-volumesnapshot-class: "true" 1 annotations: snapshot.storage.kubernetes.io/is-default-class: true 2 driver: <csi_driver> deletionPolicy: <deletion_policy_type> 3
Next steps
-
You can now create a
Backup
CR.
4.6.4. Backing up applications with File System Backup: Kopia or Restic
You can use OADP to back up and restore Kubernetes volumes attached to pods from the file system of the volumes. This process is called File System Backup (FSB) or Pod Volume Backup (PVB). It is accomplished by using modules from the open source backup tools Restic or Kopia.
If your cloud provider does not support snapshots or if your applications are on NFS data volumes, you can create backups by using FSB.
FSB integration with OADP provides a solution for backing up and restoring almost any type of Kubernetes volumes. This integration is an additional capability of OADP and is not a replacement for existing functionality.
You back up Kubernetes resources, internal images, and persistent volumes with Kopia or Restic by editing the Backup
custom resource (CR).
You do not need to specify a snapshot location in the DataProtectionApplication
CR.
In OADP version 1.3 and later, you can use either Kopia or Restic for backing up applications.
For the Built-in DataMover, you must use Kopia.
In OADP version 1.2 and earlier, you can only use Restic for backing up applications.
FSB does not support backing up hostPath
volumes. For more information, see FSB limitations.
…/.snapshot: read-only file system
error
The …/.snapshot
directory is a snapshot copy directory, which is used by several NFS servers. This directory has read-only access by default, so Velero cannot restore to this directory.
Do not give Velero write access to the .snapshot
directory, and disable client access to this directory.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
You must not disable the default
nodeAgent
installation by settingspec.configuration.nodeAgent.enable
tofalse
in theDataProtectionApplication
CR. -
You must select Kopia or Restic as the uploader by setting
spec.configuration.nodeAgent.uploaderType
tokopia
orrestic
in theDataProtectionApplication
CR. -
The
DataProtectionApplication
CR must be in aReady
state.
Procedure
Create the
Backup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> labels: velero.io/storage-location: default namespace: openshift-adp spec: defaultVolumesToFsBackup: true 1 ...
- 1
- In OADP version 1.2 and later, add the
defaultVolumesToFsBackup: true
setting within thespec
block. In OADP version 1.1, adddefaultVolumesToRestic: true
.
4.6.5. Creating backup hooks
When performing a backup, it is possible to specify one or more commands to execute in a container within a pod, based on the pod being backed up.
The commands can be configured to performed before any custom action processing (Pre hooks), or after all custom actions have been completed and any additional items specified by the custom action have been backed up (Post hooks).
You create backup hooks to run commands in a container in a pod by editing the Backup
custom resource (CR).
Procedure
Add a hook to the
spec.hooks
block of theBackup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> 1 excludedNamespaces: 2 - <namespace> includedResources: [] - pods 3 excludedResources: [] 4 labelSelector: 5 matchLabels: app: velero component: server pre: 6 - exec: container: <container> 7 command: - /bin/uname 8 - -a onError: Fail 9 timeout: 30s 10 post: 11 ...
- 1
- Optional: You can specify namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Optional: You can specify namespaces to which the hook does not apply.
- 3
- Currently, pods are the only supported resource that hooks can apply to.
- 4
- Optional: You can specify resources to which the hook does not apply.
- 5
- Optional: This hook only applies to objects matching the label. If this value is not specified, the hook applies to all objects.
- 6
- Array of hooks to run before the backup.
- 7
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 8
- This is the entry point for the
init
container being added. - 9
- Allowed values for error handling are
Fail
andContinue
. The default isFail
. - 10
- Optional: How long to wait for the commands to run. The default is
30s
. - 11
- This block defines an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.
4.6.6. Scheduling backups using Schedule CR
The schedule operation allows you to create a backup of your data at a particular time, specified by a Cron expression.
You schedule backups by creating a Schedule
custom resource (CR) instead of a Backup
CR.
Leave enough time in your backup schedule for a backup to finish before another backup is created.
For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state.
Procedure
Retrieve the
backupStorageLocations
CRs:$ oc get backupStorageLocations -n openshift-adp
Example output
NAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31m
Create a
Schedule
CR, as in the following example:$ cat << EOF | oc apply -f - apiVersion: velero.io/v1 kind: Schedule metadata: name: <schedule> namespace: openshift-adp spec: schedule: 0 7 * * * 1 template: hooks: {} includedNamespaces: - <namespace> 2 storageLocation: <velero-sample-1> 3 defaultVolumesToFsBackup: true 4 ttl: 720h0m0s EOF
- 1
cron
expression to schedule the backup, for example,0 7 * * *
to perform a backup every day at 7:00.NoteTo schedule a backup at specific intervals, enter the
<duration_in_minutes>
in the following format:schedule: "*/10 * * * *"
Enter the minutes value between quotation marks (
" "
).- 2
- Array of namespaces to back up.
- 3
- Name of the
backupStorageLocations
CR. - 4
- Optional: In OADP version 1.2 and later, add the
defaultVolumesToFsBackup: true
key-value pair to your configuration when performing backups of volumes with Restic. In OADP version 1.1, add thedefaultVolumesToRestic: true
key-value pair when you back up volumes with Restic.Verify that the status of the
Schedule
CR isCompleted
after the scheduled backup runs:$ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'
4.6.7. Deleting backups
You can remove backup files by deleting the Backup
custom resource (CR).
After you delete the Backup
CR and the associated object storage data, you cannot recover the deleted data.
Prerequisites
-
You created a
Backup
CR. -
You know the name of the
Backup
CR and the namespace that contains it. - You downloaded the Velero CLI tool.
- You can access the Velero binary in your cluster.
Procedure
Choose one of the following actions to delete the
Backup
CR:To delete the
Backup
CR and keep the associated object storage data, run the following command:$ oc delete backup <backup_CR_name> -n <velero_namespace>
To delete the
Backup
CR and delete the associated object storage data, run the following command:$ velero backup delete <backup_CR_name> -n <velero_namespace>
Where:
- <backup_CR_name>
-
The name of the
Backup
custom resource. - <velero_namespace>
-
The namespace that contains the
Backup
custom resource.
4.6.8. About Kopia
Kopia is a fast and secure open-source backup and restore tool that allows you to create encrypted snapshots of your data and save the snapshots to remote or cloud storage of your choice.
Kopia supports network and local storage locations, and many cloud or remote storage locations, including:
- Amazon S3 and any cloud storage that is compatible with S3
- Azure Blob Storage
- Google Cloud Storage platform
Kopia uses content-addressable storage for snapshots:
- Snapshots are always incremental; data that is already included in previous snapshots is not re-uploaded to the repository. A file is only uploaded to the repository again if it is modified.
- Stored data is deduplicated; if multiple copies of the same file exist, only one of them is stored.
- If files are moved or renamed, Kopia can recognize that they have the same content and does not upload them again.
4.6.8.1. OADP integration with Kopia
OADP 1.3 supports Kopia as the backup mechanism for pod volume backup in addition to Restic. You must choose one or the other at installation by setting the uploaderType
field in the DataProtectionApplication
custom resource (CR). The possible values are restic
or kopia
. If you do not specify an uploaderType
, OADP 1.3 defaults to using Kopia as the backup mechanism. The data is written to and read from a unified repository.
The following example shows a DataProtectionApplication
CR configured for using Kopia:
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample spec: configuration: nodeAgent: enable: true uploaderType: kopia # ...
4.7. OADP restoring
4.7.1. Restoring applications
You restore application backups by creating a Restore
custom resource (CR). See Creating a Restore CR.
You can create restore hooks to run commands in a container in a pod by editing the Restore
CR. See Creating restore hooks.
4.7.1.1. Previewing resources before running backup and restore
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete. To preview the resources in advance, you can do a dry run of the backup and restore operations.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
$ velero backup create <backup-name> --snapshot-volumes false 1
- 1
- Specify the value of
--snapshot-volumes
parameter asfalse
.
To know more details about the backup resources, run the following command:
$ velero describe backup <backup_name> --details 1
- 1
- Specify the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
$ velero restore create --from-backup <backup-name> 1
- 1
- Specify the name of the backup created to review the backup resources.
ImportantThe
velero restore create
command creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
$ velero describe restore <restore_name> --details 1
- 1
- Specify the name of the restore.
4.7.1.2. Creating a Restore CR
You restore a Backup
custom resource (CR) by creating a Restore
CR.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state. -
You must have a Velero
Backup
CR. - The persistent volume (PV) capacity must match the requested size at backup time. Adjust the requested size if needed.
Procedure
Create a
Restore
CR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: backupName: <backup> 1 includedResources: [] 2 excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io restorePVs: true 3
- 1
- Name of the
Backup
CR. - 2
- Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example,
po
forpods
) or fully-qualified. If unspecified, all resources are included. - 3
- Optional: The
restorePVs
parameter can be set tofalse
to turn off restore ofPersistentVolumes
fromVolumeSnapshot
of Container Storage Interface (CSI) snapshots or from native snapshots whenVolumeSnapshotLocation
is configured.
Verify that the status of the
Restore
CR isCompleted
by entering the following command:$ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
Verify that the backup resources have been restored by entering the following command:
$ oc get all -n <namespace> 1
- 1
- Namespace that you backed up.
If you restore
DeploymentConfig
with volumes or if you use post-restore hooks, run thedc-post-restore.sh
cleanup script by entering the following command:$ bash dc-restic-post-restore.sh -> dc-post-restore.sh
NoteDuring the restore process, the OADP Velero plug-ins scale down the
DeploymentConfig
objects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restoredDeploymentConfig
pods immediately on restore and to allow the restore and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales anyDeploymentConfig
objects back up to the appropriate number of replicas.Example 4.1.
dc-restic-post-restore.sh → dc-post-restore.sh
cleanup script#!/bin/bash set -e # if sha256sum exists, use it to check the integrity of the file if command -v sha256sum >/dev/null 2>&1; then CHECKSUM_CMD="sha256sum" else CHECKSUM_CMD="shasum -a 256" fi label_name () { if [ "${#1}" -le "63" ]; then echo $1 return fi sha=$(echo -n $1|$CHECKSUM_CMD) echo "${1:0:57}${sha:0:6}" } if [[ $# -ne 1 ]]; then echo "usage: ${BASH_SOURCE} restore-name" exit 1 fi echo "restore: $1" label=$(label_name $1) echo "label: $label" echo Deleting disconnected restore pods oc delete pods --all-namespaces -l oadp.openshift.io/disconnected-from-dc=$label for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}') do IFS=',' read -ra dc_arr <<< "$dc" if [ ${#dc_arr[0]} -gt 0 ]; then echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]} cat <<EOF | oc patch dc -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin spec: replicas: ${dc_arr[2]} paused: ${dc_arr[3]} EOF fi done
4.7.1.3. Creating restore hooks
You create restore hooks to run commands in a container in a pod by editing the Restore
custom resource (CR).
You can create two types of restore hooks:
An
init
hook adds an init container to a pod to perform setup tasks before the application container starts.If you restore a Restic backup, the
restic-wait
init container is added before the restore hook init container.-
An
exec
hook runs commands or scripts in a container of a restored pod.
Procedure
Add a hook to the
spec.hooks
block of theRestore
CR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> 1 excludedNamespaces: - <namespace> includedResources: - pods 2 excludedResources: [] labelSelector: 3 matchLabels: app: velero component: server postHooks: - init: initContainers: - name: restore-hook-init image: alpine:latest volumeMounts: - mountPath: /restores/pvc1-vm name: pvc1-vm command: - /bin/ash - -c timeout: 4 - exec: container: <container> 5 command: - /bin/bash 6 - -c - "psql < /backup/backup.sql" waitTimeout: 5m 7 execTimeout: 1m 8 onError: Continue 9
- 1
- Optional: Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Currently, pods are the only supported resource that hooks can apply to.
- 3
- Optional: This hook only applies to objects matching the label selector.
- 4
- Optional: Timeout specifies the maximum length of time Velero waits for
initContainers
to complete. - 5
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 6
- This is the entrypoint for the init container being added.
- 7
- Optional: How long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.
- 8
- Optional: How long to wait for the commands to run. The default is
30s
. - 9
- Allowed values for error handling are
Fail
andContinue
:-
Continue
: Only command failures are logged. -
Fail
: No more restore hooks run in any container in any pod. The status of theRestore
CR will bePartiallyFailed
.
-
During a File System Backup (FSB) restore operation, a Deployment
resource referencing an ImageStream
is not restored properly. The restored pod that runs the FSB, and the postHook
is terminated prematurely.
This happens because, during the restore operation, OpenShift controller updates the spec.template.spec.containers[0].image
field in the Deployment
resource with an updated ImageStreamTag
hash. The update triggers the rollout of a new pod, terminating the pod on which velero
runs the FSB and the post restore hook. For more information about image stream trigger, see "Triggering updates on image stream changes".
The workaround for this behavior is a two-step restore process:
First, perform a restore excluding the
Deployment
resources, for example:$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --exclude-resources=deployment.apps
After the first restore is successful, perform a second restore by including these resources, for example:
$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --include-resources=deployment.apps
Additional resources
4.8. OADP and ROSA
4.8.1. Backing up applications on ROSA clusters using OADP
You can use OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS (ROSA) clusters to back up and restore application data.
ROSA is a fully-managed, turnkey application platform that allows you to deliver value to your customers by building and deploying applications.
ROSA provides seamless integration with a wide range of Amazon Web Services (AWS) compute, database, analytics, machine learning, networking, mobile, and other services to speed up the building and delivery of differentiating experiences to your customers.
You can subscribe to the service directly from your AWS account.
After you create your clusters, you can operate your clusters with the OpenShift Container Platform web console or through Red Hat OpenShift Cluster Manager. You can also use ROSA with OpenShift APIs and command-line interface (CLI) tools.
For additional information about ROSA installation, see Installing Red Hat OpenShift Service on AWS (ROSA) interactive walkthrough.
Before installing OpenShift API for Data Protection (OADP), you must set up role and policy credentials for OADP so that it can use the Amazon Web Services API.
This process is performed in the following two stages:
- Prepare AWS credentials
- Install the OADP Operator and give it an IAM role
4.8.1.1. Preparing AWS credentials for OADP
An Amazon Web Services account must be prepared and configured to accept an OpenShift API for Data Protection (OADP) installation.
Procedure
Create the following environment variables by running the following commands:
ImportantChange the cluster name to match your ROSA cluster, and ensure you are logged into the cluster as an administrator. Ensure that all fields are outputted correctly before continuing.
$ export CLUSTER_NAME=my-cluster 1 export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id) export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id) export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.') export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials" export SCRATCH="/tmp/${CLUSTER_NAME}/oadp" mkdir -p ${SCRATCH} echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"
- 1
- Replace
my-cluster
with your ROSA cluster name.
On the AWS account, create an IAM policy to allow access to AWS S3:
Check to see if the policy exists by running the following command:
$ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text) 1
- 1
- Replace
RosaOadp
with your policy name.
Enter the following command to create the policy JSON file and then create the policy in ROSA:
NoteIf the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the
if
statement intentionally skips the policy creation.$ if [[ -z "${POLICY_ARN}" ]]; then cat << EOF > ${SCRATCH}/policy.json 1 { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:PutBucketTagging", "s3:GetBucketTagging", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:PutLifecycleConfiguration", "s3:GetLifecycleConfiguration", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucketMultipartUploads", "s3:AbortMultipartUploads", "s3:ListMultipartUploadParts", "s3:DescribeSnapshots", "ec2:DescribeVolumes", "ec2:DescribeVolumeAttribute", "ec2:DescribeVolumesModifications", "ec2:DescribeVolumeStatus", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" } ]} EOF POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \ --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \ --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \ --output text) fi
- 1
SCRATCH
is a name for a temporary directory created for the environment variables.
View the policy ARN by running the following command:
$ echo ${POLICY_ARN}
Create an IAM role trust policy for the cluster:
Create the trust policy file by running the following command:
$ cat <<EOF > ${SCRATCH}/trust-policy.json { "Version":2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_ENDPOINT}:sub": [ "system:serviceaccount:openshift-adp:openshift-adp-controller-manager", "system:serviceaccount:openshift-adp:velero"] } } }] } EOF
Create the role by running the following command:
$ ROLE_ARN=$(aws iam create-role --role-name \ "${ROLE_NAME}" \ --assume-role-policy-document file://${SCRATCH}/trust-policy.json \ --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID} Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=openshift-oadp \ --query Role.Arn --output text)
View the role ARN by running the following command:
$ echo ${ROLE_ARN}
Attach the IAM policy to the IAM role by running the following command:
$ aws iam attach-role-policy --role-name "${ROLE_NAME}" \ --policy-arn ${POLICY_ARN}
4.8.1.2. Installing the OADP Operator and providing the IAM role
AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. OpenShift Container Platform (ROSA) with STS is the recommended credential mode for ROSA clusters. This document describes how to install OpenShift API for Data Protection (OADP) on ROSA with AWS STS.
Restic and Kopia are not supported in the OADP on ROSA with AWS STS environment. Verify that the Restic and Kopia node agent is disabled. For backing up volumes, OADP on ROSA with AWS STS supports only native snapshots and Container Storage Interface (CSI) snapshots.
In an Amazon ROSA cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.
The Data Mover feature is not currently supported in ROSA clusters. You can use native AWS S3 tools for moving data.
Prerequisites
-
An OpenShift Container Platform ROSA cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including
ROLE_ARN
, for each cluster.
Procedure
Create an OpenShift Container Platform secret from your AWS token file by entering the following commands:
Create the credentials file:
$ cat <<EOF > ${SCRATCH}/credentials [default] role_arn = ${ROLE_ARN} web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token EOF
Create a namespace for OADP:
$ oc create namespace openshift-adp
Create the OpenShift Container Platform secret:
$ oc -n openshift-adp create secret generic cloud-credentials \ --from-file=${SCRATCH}/credentials
NoteIn OpenShift Container Platform versions 4.14 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the OpenShift Container Platform web console, for more information see Installing from OperatorHub using the web console.
The preceding secret is created automatically by CCO.
Install the OADP Operator:
- In the OpenShift Container Platform web console, browse to Operators → OperatorHub.
- Search for the OADP Operator.
- In the role_ARN field, paste the role_arn that you created previously and click Install.
Create AWS cloud storage using your AWS credentials by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: CloudStorage metadata: name: ${CLUSTER_NAME}-oadp namespace: openshift-adp spec: creationSecret: key: credentials name: cloud-credentials enableSharedConfig: true name: ${CLUSTER_NAME}-oadp provider: aws region: $REGION EOF
Check your application’s storage default storage class by entering the following command:
$ oc get pvc -n <namespace>
Example output
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE applog Bound pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8 1Gi RWO gp3-csi 4d19h mysql Bound pvc-16b8e009-a20a-4379-accc-bc81fedd0621 1Gi RWO gp3-csi 4d19h
Get the storage class by running the following command:
$ oc get storageclass
Example output
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 4d21h gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h
NoteThe following storage classes will work:
- gp3-csi
- gp2-csi
- gp3
- gp2
If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.
Create the
DataProtectionApplication
resource to configure the connection to the storage where the backups and volume snapshots are stored:If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ${CLUSTER_NAME}-dpa namespace: openshift-adp spec: backupImages: true 1 features: dataMover: enable: false backupLocations: - bucket: cloudStorageRef: name: ${CLUSTER_NAME}-oadp credential: key: credentials name: cloud-credentials prefix: velero default: true config: region: ${REGION} configuration: velero: defaultPlugins: - openshift - aws - csi restic: enable: false EOF
- 1
- ROSA supports internal image backup. Set this field to
false
if you do not want to use image backup.
If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ${CLUSTER_NAME}-dpa namespace: openshift-adp spec: backupImages: true 1 features: dataMover: enable: false backupLocations: - bucket: cloudStorageRef: name: ${CLUSTER_NAME}-oadp credential: key: credentials name: cloud-credentials prefix: velero default: true config: region: ${REGION} configuration: velero: defaultPlugins: - openshift - aws nodeAgent: 2 enable: false uploaderType: restic snapshotLocations: - velero: config: credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials 3 enableSharedConfig: "true" 4 profile: default 5 region: ${REGION} 6 provider: aws EOF
- 1
- ROSA supports internal image backup. Set this field to false if you do not want to use image backup.
- 2
- See the important note regarding the
nodeAgent
attribute. - 3
- The
credentialsFile
field is the mounted location of the bucket credential on the pod. - 4
- The
enableSharedConfig
field allows thesnapshotLocations
to share or reuse the credential defined for the bucket. - 5
- Use the profile name set in the AWS credentials file.
- 6
- Specify
region
as your AWS region. This must be the same as the cluster region.
You are now ready to back up and restore OpenShift Container Platform applications, as described in Backing up applications.
The enable
parameter of restic
is set to false
in this configuration, because OADP does not support Restic in ROSA environments.
If you use OADP 1.2, replace this configuration:
nodeAgent: enable: false uploaderType: restic
with the following configuration:
restic: enable: false
If you want to use two different clusters for backing up and restoring, the two clusters must have the same AWS S3 storage names in both the cloud storage CR and the OADP DataProtectionApplication
configuration.
Additional resources
4.8.1.3. Example: Backing up workload on OADP ROSA STS, with an optional cleanup
4.8.1.3.1. Performing a backup with OADP and ROSA STS
The following example hello-world
application has no persistent volumes (PVs) attached. Perform a backup with OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS (ROSA) STS.
Either Data Protection Application (DPA) configuration will work.
Create a workload to back up by running the following commands:
$ oc create namespace hello-world
$ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift
Expose the route by running the following command:
$ oc expose service/hello-openshift -n hello-world
Check that the application is working by running the following command:
$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`
Example output
Hello OpenShift!
Back up the workload by running the following command:
$ cat << EOF | oc create -f - apiVersion: velero.io/v1 kind: Backup metadata: name: hello-world namespace: openshift-adp spec: includedNamespaces: - hello-world storageLocation: ${CLUSTER_NAME}-dpa-1 ttl: 720h0m0s EOF
Wait until the backup is completed and then run the following command:
$ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"
Example output
{ "completionTimestamp": "2022-09-07T22:20:44Z", "expiration": "2022-10-07T22:20:22Z", "formatVersion": "1.1.0", "phase": "Completed", "progress": { "itemsBackedUp": 58, "totalItems": 58 }, "startTimestamp": "2022-09-07T22:20:22Z", "version": 1 }
Delete the demo workload by running the following command:
$ oc delete ns hello-world
Restore the workload from the backup by running the following command:
$ cat << EOF | oc create -f - apiVersion: velero.io/v1 kind: Restore metadata: name: hello-world namespace: openshift-adp spec: backupName: hello-world EOF
Wait for the Restore to finish by running the following command:
$ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"
Example output
{ "completionTimestamp": "2022-09-07T22:25:47Z", "phase": "Completed", "progress": { "itemsRestored": 38, "totalItems": 38 }, "startTimestamp": "2022-09-07T22:25:28Z", "warnings": 9 }
Check that the workload is restored by running the following command:
$ oc -n hello-world get pods
Example output
NAME READY STATUS RESTARTS AGE hello-openshift-9f885f7c6-kdjpj 1/1 Running 0 90s
Check the JSONPath by running the following command:
$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`
Example output
Hello OpenShift!
For troubleshooting tips, see the OADP team’s troubleshooting documentation.
4.8.1.3.2. Cleaning up a cluster after a backup with OADP and ROSA STS
If you need to uninstall the OpenShift API for Data Protection (OADP) Operator together with the backups and the S3 bucket from this example, follow these instructions.
Procedure
Delete the workload by running the following command:
$ oc delete ns hello-world
Delete the Data Protection Application (DPA) by running the following command:
$ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa
Delete the cloud storage by running the following command:
$ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp
WarningIf this command hangs, you might need to delete the finalizer by running the following command:
$ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge
If the Operator is no longer required, remove it by running the following command:
$ oc -n openshift-adp delete subscription oadp-operator
Remove the namespace from the Operator:
$ oc delete ns openshift-adp
If the backup and restore resources are no longer required, remove them from the cluster by running the following command:
$ oc delete backup hello-world
To delete backup, restore and remote objects in AWS S3 run the following command:
$ velero backup delete hello-world
If you no longer need the Custom Resource Definitions (CRD), remove them from the cluster by running the following command:
$ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done
Delete the AWS S3 bucket by running the following commands:
$ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive
$ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp
Detach the policy from the role by running the following command:
$ aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"
Delete the role by running the following command:
$ aws iam delete-role --role-name "${ROLE_NAME}"
4.9. OADP and AWS STS
4.9.1. Backing up applications on AWS STS using OADP
You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) by installing the OADP Operator. The Operator installs Velero 1.14.
Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.
You configure AWS for Velero, create a default Secret
, and then install the Data Protection Application. For more details, see Installing the OADP Operator.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
You can install OADP on an AWS Security Token Service (STS) (AWS STS) cluster manually. Amazon AWS provides AWS STS as a web service that enables you to request temporary, limited-privilege credentials for users. You use STS to provide trusted users with temporary access to resources via API calls, your AWS console, or the AWS command line interface (CLI).
Before installing OpenShift API for Data Protection (OADP), you must set up role and policy credentials for OADP so that it can use the Amazon Web Services API.
This process is performed in the following two stages:
- Prepare AWS credentials.
- Install the OADP Operator and give it an IAM role.
4.9.1.1. Preparing AWS STS credentials for OADP
An Amazon Web Services account must be prepared and configured to accept an OpenShift API for Data Protection (OADP) installation. Prepare the AWS credentials by using the following procedure.
Procedure
Define the
cluster_name
environment variable by running the following command:$ export CLUSTER_NAME= <AWS_cluster_name> 1
- 1
- The variable can be set to any value.
Retrieve all of the details of the
cluster
such as theAWS_ACCOUNT_ID, OIDC_ENDPOINT
by running the following command:$ export CLUSTER_VERSION=$(oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}') export AWS_CLUSTER_ID=$(oc get clusterversion version -o jsonpath='{.spec.clusterID}{"\n"}') export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') export REGION=$(oc get infrastructures cluster -o jsonpath='{.status.platformStatus.aws.region}' --allow-missing-template-keys=false || echo us-east-2) export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"
Create a temporary directory to store all of the files by running the following command:
$ export SCRATCH="/tmp/${CLUSTER_NAME}/oadp" mkdir -p ${SCRATCH}
Display all of the gathered details by running the following command:
$ echo "Cluster ID: ${AWS_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"
On the AWS account, create an IAM policy to allow access to AWS S3:
Check to see if the policy exists by running the following commands:
$ export POLICY_NAME="OadpVer1" 1
- 1
- The variable can be set to any value.
$ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='$POLICY_NAME'].{ARN:Arn}" --output text)
Enter the following command to create the policy JSON file and then create the policy:
NoteIf the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the
if
statement intentionally skips the policy creation.$ if [[ -z "${POLICY_ARN}" ]]; then cat << EOF > ${SCRATCH}/policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:PutBucketTagging", "s3:GetBucketTagging", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:PutLifecycleConfiguration", "s3:GetLifecycleConfiguration", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucketMultipartUploads", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "ec2:DescribeSnapshots", "ec2:DescribeVolumes", "ec2:DescribeVolumeAttribute", "ec2:DescribeVolumesModifications", "ec2:DescribeVolumeStatus", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" } ]} EOF POLICY_ARN=$(aws iam create-policy --policy-name $POLICY_NAME \ --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \ --tags Key=openshift_version,Value=${CLUSTER_VERSION} Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=oadp \ --output text) 1 fi
- 1
SCRATCH
is a name for a temporary directory created for storing the files.
View the policy ARN by running the following command:
$ echo ${POLICY_ARN}
Create an IAM role trust policy for the cluster:
Create the trust policy file by running the following command:
$ cat <<EOF > ${SCRATCH}/trust-policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_ENDPOINT}:sub": [ "system:serviceaccount:openshift-adp:openshift-adp-controller-manager", "system:serviceaccount:openshift-adp:velero"] } } }] } EOF
Create an IAM role trust policy for the cluster by running the following command:
$ ROLE_ARN=$(aws iam create-role --role-name \ "${ROLE_NAME}" \ --assume-role-policy-document file://${SCRATCH}/trust-policy.json \ --tags Key=cluster_id,Value=${AWS_CLUSTER_ID} Key=openshift_version,Value=${CLUSTER_VERSION} Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=oadp --query Role.Arn --output text)
View the role ARN by running the following command:
$ echo ${ROLE_ARN}
Attach the IAM policy to the IAM role by running the following command:
$ aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn ${POLICY_ARN}
4.9.1.1.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> 1 resourceAllocations: 2 limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Mi
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
4.9.1.2. Installing the OADP Operator and providing the IAM role
AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. This document describes how to install OpenShift API for Data Protection (OADP) on an AWS STS cluster manually.
Restic and Kopia are not supported in the OADP AWS STS environment. Verify that the Restic and Kopia node agent is disabled. For backing up volumes, OADP on AWS STS supports only native snapshots and Container Storage Interface (CSI) snapshots.
In an AWS cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.
The Data Mover feature is not currently supported in AWS STS clusters. You can use native AWS S3 tools for moving data.
Prerequisites
-
An OpenShift Container Platform AWS STS cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including
ROLE_ARN
, for each cluster.
Procedure
Create an OpenShift Container Platform secret from your AWS token file by entering the following commands:
Create the credentials file:
$ cat <<EOF > ${SCRATCH}/credentials [default] role_arn = ${ROLE_ARN} web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token EOF
Create a namespace for OADP:
$ oc create namespace openshift-adp
Create the OpenShift Container Platform secret:
$ oc -n openshift-adp create secret generic cloud-credentials \ --from-file=${SCRATCH}/credentials
NoteIn OpenShift Container Platform versions 4.14 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the OpenShift Container Platform web console, for more information see Installing from OperatorHub using the web console.
The preceding secret is created automatically by CCO.
Install the OADP Operator:
- In the OpenShift Container Platform web console, browse to Operators → OperatorHub.
- Search for the OADP Operator.
- In the role_ARN field, paste the role_arn that you created previously and click Install.
Create AWS cloud storage using your AWS credentials by en