Chapter 1. OADP Application backup and restore
1.1. Introduction to OpenShift API for Data Protection Copy linkLink copied to clipboard!
Use OpenShift API for Data Protection (OADP) to safeguard applications, application-related cluster resources, persistent volumes, and internal images on Red Hat OpenShift Service on AWS classic architecture. OADP backs up containerized applications and virtual machines (VMs). This helps you ensure disaster recovery.
OADP support is applicable to customer workload namespaces and cluster scope resources.
Full cluster backup and restore are not supported.
1.1.1. OpenShift API for Data Protection APIs Copy linkLink copied to clipboard!
OADP provides APIs that enable multiple approaches to customizing backups and preventing the inclusion of unnecessary or inappropriate resources.
OADP provides the following APIs. See the Additional resources section for more details.
-
Backup -
Restore -
Schedule -
BackupStorageLocation -
VolumeSnapshotLocation
1.1.1.1. Support for OpenShift API for Data Protection Copy linkLink copied to clipboard!
Review the OADP support matrix for version compatibility with Red Hat OpenShift Service on AWS classic architecture releases and lifecycle policy information, including Extended Update Support (EUS) options.
| Version | Red Hat OpenShift Service on AWS classic architecture version | General availability | Full support ends | Maintenance ends | Extended Update Support (EUS) | Extended Update Support Term 2 (EUS Term 2) |
| 1.6 |
| 09 Jun 2026 | Release of 1.7 | Release of 1.8 | 30 Jun 2028 | 30 Jun 2029 |
| 1.5 |
| 17 June 2025 | Release of 1.6 | Release of 1.7 | 30 Jun 2028 | 30 Jun 2029 EUS must be on Red Hat OpenShift Service on AWS classic architecture 4.21 |
| EUS Term 2 must be on Red Hat OpenShift Service on AWS classic architecture 4.21 | 1.4 |
| 10 Jul 2024 | Release of 1.5 | Release of 1.6 | 27 Jun 2026 EUS must be on Red Hat OpenShift Service on AWS classic architecture 4.16 |
| 27 Jun 2027 EUS Term 2 must be on Red Hat OpenShift Service on AWS classic architecture 4.16 | 1.3 |
| 29 Nov 2023 | 10 Jul 2024 | Release of 1.5 | 31 Oct 2025 EUS must be on Red Hat OpenShift Service on AWS classic architecture 4.14 |
1.1.1.1.1. Unsupported versions of the OADP Operator Copy linkLink copied to clipboard!
| Version | General availability | Full support ended | Maintenance ended |
| 1.2 | 14 Jun 2023 | 29 Nov 2023 | 10 Jul 2024 |
| 1.1 | 01 Sep 2022 | 14 Jun 2023 | 29 Nov 2023 |
| 1.0 | 09 Feb 2022 | 01 Sep 2022 | 14 Jun 2023 |
For more details about EUS, see Extended Update Support.
For more details about EUS Term 2, see Extended Update Support Term 2.
1.2. OADP release notes Copy linkLink copied to clipboard!
1.2.1. OADP 1.6 release notes Copy linkLink copied to clipboard!
Review new features, resolved issues, known issues, Technology Preview features, removed features, and other additions in OpenShift API for Data Protection (OADP) 1.6.
For additional information about OADP, see OpenShift API for Data Protection (OADP) FAQ.
1.2.1.1. OADP 1.6.1 release notes Copy linkLink copied to clipboard!
OpenShift API for Data Protection (OADP) 1.6.1 release notes list resolved and known issues.
The following Red Hat Security Advisory (RHSA) is available for OADP 1.6.1:
1.2.1.1.1. Resolved issues Copy linkLink copied to clipboard!
- OADP CLI no longer exposes the namespace flag for non-admin operations
Before this update, the OADP command-line interface (CLI) exposed the global
--namespaceor-nflag in the help output of alloc oadp nonadmincommands, even though non-admin operations do not support namespace selection because of multi-tenancy boundaries. As a consequence, if you specified the namespace flag during a non-admin backup or restore operation, the CLI ignored the provided value and used the activeoc projectcontext instead, potentially creating resources in an unintended namespace.With this update, the
--namespaceflag is hidden from alloc oadp nonadminsubcommands so that it is no longer displayed in the help output and does not accept user input for non-admin operations. As a result, non-admin CLI commands operate correctly only within your current project context, which eliminates the risk of namespace mismatch.- Wildcard handling for
excludedClusterScopedResourcesno longer causes validation failures Before this update, the OADP Operator incorrectly appended default resources to
excludedClusterScopedResourcesin theNonAdminBackupcustom resource (CR) when a wildcard (*) was present. As a consequence,FailedValidationerrors occurred when you excluded all cluster-scoped resources. With this update, the Operator correctly handles the wildcard without appending default resources. As a result, validation failures no longer occur when you use a wildcard to exclude cluster-scoped resources.VolumePolicysupport for phase conditions to allow skipping PVCsBefore this update, OADP volume policies lacked the capability to evaluate and filter persistent volume claims (PVCs) based on their lifecycle status phase. As a consequence, backups could trigger unwanted warnings or errors for unbound or pending PVCs that you excluded, such as in scaled-down serverless environments.
With this update, volume policy actions have been improved to natively support the phase condition of PVCs. As a result, you can define custom volume policies in your
ConfigMapobject to automatically skip pending or unbound PVCs during backup operations. For example:apiVersion: v1 kind: ConfigMap metadata: name: volume-policy namespace: velero data: policy.yaml: | version: v1 volumePolicies: - conditions: pvcPhase: ["Pending", "Lost"] action: type: skip
1.2.1.1.2. Known issues Copy linkLink copied to clipboard!
CloudStoragebucket creation fails on Google Cloud with WIF authenticationWhen you configure
CloudStorageon OpenShift clusters on Google Cloud by using Workload Identity Federation (WIF) authentication, theCloudStorageresource fails to detect that a Google Cloud bucket does not exist. As a consequence, the resource returns aBucketCheckErrorcondition and prevents automatic bucket creation.To work around this problem, use an existing Google Cloud bucket when configuring
CloudStorageon OpenShift clusters on Google Cloud using Workload Identity Federation (WIF) authentication. As a result, the controller skips bucket creation and proceeds with the existing bucket for backup storage.- Non-admin backups of ODF encrypted volumes fail with OADP
When the OADP Data Mover backs up an encrypted Red Hat OpenShift Data Foundation (ODF) volume, it creates a temporary persistent volume claim (PVC) in the
openshift-adpnamespace. The ceph-csi driver then attempts to fetch theceph-csi-kms-tokensecret from theopenshift-adpnamespace instead of the original application namespace that contains the secret. As a consequence, theNonAdminBackupcustom resource (CR) fails with an error similar to the following output:openshift-adp 2h11m Warning ProvisioningFailed persistentvolumeclaim/NAMESPACE-rh-cf00000c-0000-0000-0000-22abf391t6dr failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-encrypted": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed fetching token from openshift-adp/ceph-csi-kms-token: secrets "ceph-csi-kms-token" not foundTo work around this problem, copy
ceph-csi-kms-tokenand the associated Key Management Service (KMS) config map from the application namespace toopenshift-adpbefore running the backup. As a result, the ceph-csi driver can locate the encryption credentials and the backup completes successfully.
1.2.1.2. OADP 1.6.0 release notes Copy linkLink copied to clipboard!
Review new features and enhancements, resolved issues, known issues, Technology Preview features, and removed features in OpenShift API for Data Protection (OADP) 1.6.0.
1.2.1.2.1. New features Copy linkLink copied to clipboard!
- Parallel backup processing
OADP supports parallel backup processing. Before this update, OADP processed backups sequentially where each backup had to complete before the next one could begin. This created bottlenecks for environments with many backup schedules or large-scale data protection needs, increasing the overall time required to complete all backup operations. With this release, multiple backups can run in parallel if they do not share namespaces. As a result, you can now define more granular or frequent backup schedules without worrying about queuing delays.
- File-level restore with VMFR from KubeVirt virtual machine backups
OADP introduces the virtual machine file restore (VMFR) feature. Before this release, restoring even a single file from a KubeVirt VM backup required restoring the entire VM, which was time-consuming and resource-intensive. With this release, you can selectively recover individual files from KubeVirt VM backups directly within Red Hat OpenShift Service on AWS classic architecture. VMFR also includes
RestoreItemAction, a plugin for collision prevention in multi-backup scenarios. This plugin renames PVCs during restore, ensuring unique names and avoiding conflicts, thereby improving the reliability and efficiency of VMFR backups and restores.- File-level backups with VMDP in OpenShift Virtualization environments
OADP introduces the virtual machine data protection (VMDP) feature, a command-line tool that runs inside virtual machines (VMs). Before this update, OADP supported only snapshot-based backup of entire VMs, with no option for granular, user-driven file-level data protection from within a VM. Using VMDP, you can selectively back up and restore individual files and directories by using a Kopia-based client/server architecture without cluster administrator intervention.
- OADP CLI plugin
OADP includes a
kubectlcommand-line interface (CLI) plugin that provides akubectlnative interface for managing backups and restores. Before this update, you aliased to the Velero CLI to perform backup operations. Non-cluster administrators had no way to independently manage their own backups, creating a dependency on cluster admins for routine data protection tasks. As a result, admins can runkubectl oadpcommands to create, delete, and retrieve logs for backups and restores. Non-cluster admins can independently create, delete, and inspect non-admin backups within their permitted namespaces.- OADP must-gather collects VM file restore CRDs
With this enhancement, the oadp-must-gather tool collects the
VirtualMachineBackupsDiscoveryandVirtualMachineFileRestorecustom resource definitions (CRDs) introduced in OADP 1.6 for the virtual machine file-level restore feature. As a result, when you run theoc adm must-gathercommand, both new resource types are automatically collected cluster-wide, including CRDs, per-namespace CR YAMLs, and dedicated sections in the markdown summary. This gives support teams and administrators full visibility into VM file restore operations.- Backup and restore of custom SCC through
ClusterRoleandClusterRoleBinding OADP supports backup and restore of custom security context constraints (SCC) with
ServiceAccountsthroughClusterRoleandClusterRoleBindingresources, not only those listed in the SCC user fields. This enhancement simplifies the backup and restore process of custom SCCs, as it no longer relies on the user field of the SCC. Instead, it uses the location of the SCC used by a pod and the service account associated with the pod. This results in a more reliable and secure backup and restore process for Red Hat OpenShift Service on AWS classic architecture clusters, essential for IBM Cloud Pak for Data.VolumePolicysupport for bypassing pending PVCs during backupsWith this update, you can bypass unbound persistent volume claims (PVCs) during backups. This enhancement improves backup success rates, especially when PVCs are in a
Pendingstate. Additionally, it expands the capabilities of volume policy actions, offering greater control and flexibility in managing backups.- Optimized Data Mover performance in containerized environments with new Go version
With the migration to Go 1.25, the Data Mover defaults the
MaxParallelFileReadsvalue to thecgroupCPU bandwidth limit corresponding to the container CPU limit rather than the total number of CPU cores on the node. Before this update,MaxParallelFileReadsused the full CPU count of the node unlessdatamoverConfig.ParallelFilesUploadwas explicitly set in the node-agent-config map, which could lead to overuse of resources in constrained container environments. With this update, Data Mover operations automatically respect container CPU limits, improving resource efficiency and stability without requiring manual configuration changes.- Custom labels and annotations for repository maintenance job pods
Before this update, maintenance job pods did not support custom labels, which made it difficult to apply namespace-level network policies or firewall rules that rely on pod labels. With this release, you can configure custom
podLabelsandpodAnnotationsfor repository maintenance jobs through therepositoryMaintenancesection of theDataProtectionApplicationcustom resource.- Prometheus-based monitoring for maintenance jobs
With this update, you can monitor maintenance job success and failures more efficiently using Prometheus metrics. This enhancement replaces the current method of tracking job success by using the
--keep-latest-maintenance-jobsargument. The new solution provides a unified approach for monitoring various aspects of the backup process, using existing tools for alerts and notifications. This improvement reduces the risk of performance degradation due to job failures and helps you set up alerts for maintenance job issues, improving overall backup management and system health.- Improved error messaging for
ConfigMapvalidation in the Velero CLI Before this update,
ConfigMapvalidation failures did not return meaningful errors, making it difficult to diagnose and resolve configuration issues. With this release, the Velero command-line interface (CLI) displays detailed error messages whenConfigMapvalidation fails for backup repository, node agent, and backup repository maintenance job configurations. As a result, you can quickly identify and fix invalidConfigMapconfigurations with actionable error messages.- Improved diagnostics for Data Mover expose failures
The Data Mover expose diagnostic includes events from backup pods, restore pods, backup persistent volume claims (PVCs), and restore PVCs for both CSI snapshot data movement and file system backup. Before this update, informative messages from these resources were only available in their individual events and were not captured in the diagnostic output. As a result, you can get a more complete picture of expose failures directly from the diagnostic, without manually inspecting events on each related resource.
VolumeSnapshotClassno longer required for CSI backup and restoreThe Container Storage Interface (CSI) backup and restore process no longer includes or requires the
VolumeSnapshotClassresource. Before this update, theVolumeSnapshotClasswas included in CSI backups and needed to be present during restore, adding an unnecessary dependency. With this update, CSI snapshot-based backups and restores complete successfully withoutVolumeSnapshotClassin the cluster, simplifying the backup resource footprint and improving restore reliability in environments where theVolumeSnapshotClassmight not be pre-configured.- Configurable cache volumes for node agent pods
This release introduces configurable repository cache storage locations for the node agent Data Mover and pod volume restore operations. Before this update, backup engines such as Kopia stored metadata and snapshot caches implicitly within the ephemeral root file system of the pod. This often caused host node disk exhaustion,
DiskPressuretaints, and unexpected node evictions during heavy or concurrent operations.With this release, administrators can isolate and route repository cache files to dedicated local volumes or dynamically provisioned persistent volume claims. As a result, cache volumes protect the system disk of the host node from running out of space and this increases the reliability of large-scale backups and restores.
- Repository maintenance jobs inherit tolerations from the Velero deployment
Repository maintenance jobs inherit tolerations defined in the
DataProtectionApplicationconfiguration. Before this update, maintenance pods did not inherit tolerations from the Velero deployment, which caused scheduling failures in clusters where nodes have taints configured. As a result, repository maintenance jobs can be scheduled correctly on tainted nodes, ensuring reliable backup repository maintenance in environments that use node taints and tolerations.- Opt-in automatic space reclamation for restored PVCs on ODF clusters
You can enable automatic space reclamation on restored persistent volume claims (PVCs) by setting the
oadp.openshift.io/reclaim-space-scheduleannotation on theRestorecustom resource. Before this update, after a Data Mover restore on a Red Hat OpenShift Data Foundation (ODF) cluster, you had to manually runrbd sparsifyto reclaim unused block-device space. As a result, the PVC restore plugin propagates the schedule annotation to each restored PVC, and the Container Storage Interface controller automatically creates aReclaimSpaceCronJobthat handles sparsification without manual intervention. This annotation is opt-in and has no effect on non-ODF clusters.- Scheduled backup queue no longer accumulates during extended blocking scenarios
Velero scheduled backups no longer accumulate in a
Newstate queue when backup processing is blocked for extended periods, for example, during long-running backups, Velero server downtime, or infrastructure issues. Before this update, the schedule controller prevented new backup execution if any backup from that schedule was in aNeworInProgressstate, which led to indefinite queuing and backup accumulation. As a result, unreconciled backups, including those with an empty phase, are properly handled by the scheduler, preventing it from creating excessive new backup runs and ensuring reliable scheduled backup behavior.- Added annotations to control cluster-autoscaler behavior
With this update, you can add annotations to Velero and node agent pods in OADP. This enhancement provides control over cluster-autoscaler behavior, preventing data operations from failing due to the interaction with the autoscaler. By adding annotations, users can mitigate an issue with OADP 1.5 and earlier, where data uploads and downloads can be canceled on node agent startup. This improvement ensures smoother operation and reduced downtime, and provides flexibility to customize the application according to specific requirements.
- Wildcard support for namespace selection during backup operations
You can use wildcard patterns when specifying namespaces in backup operations. Before this update, you had to explicitly list every namespace in the backup object and update it each time a new namespace was created. With this release, you can use patterns such as
*-helmin the--include-namespacesflag to dynamically match multiple namespaces, simplifying backup management and reducing manual configuration overhead.- Encryption of backup data with SSE-C for AWS S3 buckets
With this update, you can secure your backup data by using server side encryption with customer-provided keys (SSE-C) in OADP. This enhancement improves overall data protection and privacy. The feature is compatible with AWS S3 buckets. Additionally, it provides a workaround for using S3-compatible storage.
- Configurable priority class for node agent and Velero pods
With this update, you can configure the
priorityClassNamefield for node agent and Velero pods in aDataProtectionApplicationobject, prioritizing these pods during resource contention. This is particularly useful after worker node outages, ensuring critical pods are scheduled first.
1.2.1.2.2. Resolved issues Copy linkLink copied to clipboard!
- Velero backup with
labelSelectorno longer has degraded performance during resource discovery Before this update, a performance regression in the resource discovery phase caused Velero to issue an excessive number of API list calls when a backup combined
includedNamespaces: '*',includeClusterResources: true,labelSelector, and a largeexcludedResourceslist. As a consequence, backups that completed before in seconds took about 18 minutes to back up as few as 8 resources. With this release, the resource discovery logic is optimized so that unnecessary API list calls are eliminated during the item collection phase. As a result, backup performance is restored to expected levels, completing in seconds.- CSI snapshot early frequent polling can be configured in the DPA CR
Before this update, a Velero change introduced 1-second Container Storage Interface (CSI) snapshot readiness polling for the first 10 seconds of every backup, designed specifically for Microsoft Volume Shadow Copy Service (VSS) workloads on Windows. OADP did not expose this behavior as a configurable option in the
DataProtectionApplication(DPA) custom resource (CR). As a consequence, all CSI-enabled backups used the earlier polling behavior regardless of whether VSS compatibility was needed, and you had no way to disable or control this behavior through OADP. This release adds a newenableCSISnapshotEarlyFrequentPollingfield underconfiguration.veleroin the DPA CR. By default, this feature is disabled. If your workloads require this frequent polling behavior, you must enable it.- Controller-manager no longer enters a
CrashLoopBackOffloop on single-node OpenShift clusters without Console CRD Before this update, the controller-manager entered a
CrashLoopBackOffloop due to unconditional interaction with a missingConsoleCLIDownloadcustom resource definition (CRD) on clusters without Console capability. As a consequence, you experienced pod crashes on single-node OpenShift clusters due to the missingConsoleCLIDownloadCRD. With this release, the OADP Operator skips CLI download setup when the Console CRD is not available. As a result, the controller-manager no longer enters aCrashLoopBackOffloop on single-node OpenShift clusters, improving its stability.- DPA no longer reports an error when the
podConfigvalue is empty andnodeAgentaffinity is specified Before this update,
DataProtectionApplication(DPA) custom resource reported an error due to an emptypodConfigwith specifiednodeAgentaffinity. This caused DPA to fail to reconcile. With this release, DPA reports error only whennodeAgent.podConfig.nodeSelectordoes not matchspec.configuration.nodeAgent.LoadAffinityConfigwhenpodConfigis explicitly set.- Backup phase
QueuedandReadyToStartno longer cause reconcile issues innonAdminController Before this update, the
QueuedandReadyToStartbackup phases innonAdminControllercaused reconcile issues due to their unsupported status. With this release, theQueuedandReadyToStartno longer cause reconcile issues innonAdminController.- CSI
VolumeGroupSnapshotrestore no longer fails with Ceph RBD storage class Before this update, restoring a
VolumeGroupSnapshot(VGS) backed up from Ceph RBD storage failed with acannot find the group snapshot for VolumeSnapshoterror message. This error occurred because Velero deleted theVolumeGroupSnapshotContentafter backup, leaving no matching resource during restore. Also theVolumeSnapshotClassNamewas deleted from restoredVolumeSnapshotContent, preventing the Container Storage Interface (CSI) controller from authenticating to verify snapshots. With this update, Velero preserves the group snapshot handle as an annotation during backup, recreates a stubVolumeGroupSnapshotContentduring restores, and restores the correctVolumeSnapshotClassreference. As a result, VGS restores with Ceph RBD complete successfully. Note that VGS is a Technology Preview feature.- SCC map is synchronized in
ServiceAccount Before this update, the
ServiceAccountbackup plugin failed with aPartiallyFailederror due to unsynchronized access to the security context constraints (SCC) map. As a consequence, backup failures occurred, affecting data integrity in multiple namespaces. With this release, the SCC map is synchronized in theServiceAccountplugin, preventing its crash during backup execution.PrepareQueueLengthis present in DPAThe
PrepareQueueLengthfield is a configuration setting for the node agent that limits the number of intermediate resources created during backup and restore processes. Before this update,PrepareQueueLengthwas missing in theLoadConcurrencyfield in theDataProtectionApplication(DPA) configuration. As a consequence, you encountered an error while configuringPrepareQueueLength, causing the configuration to be removed. With this release, thePrepareQueueLengthfield is added toLoadConcurrencyin the DPA configuration.BackupRepositorymaintenance jobs no longer fail when the repository name exceeds 63 charactersBefore this update, the
BackupRepositoryname was used directly as a Kubernetes label value for repository maintenance jobs. As a consequence, when theBackupRepositoryname exceeded the 63-character Kubernetes label limit, the maintenance job failed to build, preventing scheduled repository maintenance from running. With this release, theBackupRepositoryname is hashed before being used as a label value, ensuring it always fits within the 63-character limit. As a result, repository maintenance jobs are created and completed successfully regardless of theBackupRepositoryname length.- Restore transitions to
Completedonly afterPodVolumeRestorescompletes Before this update, restores were marked as completed prematurely before all
PodVolumeRestores. With this release, restores complete only afterPodVolumeRestores.- Node agent pod restart no longer cancels
DataUploadsacross the entire cluster Before this update, when any node agent pod restarted, Data Mover canceled all
DataUploadsinAcceptedstatus across all node agent pods, regardless of which pod was handling them. As a consequence, backup operations failed and the backup queue could be blocked for extended periods even when the remaining node agent pods were healthy. With this release, the controller only cancelsDataUploadsthat were accepted by the specific node agent pod that restarted, leavingDataUploadson other nodes unaffected. As a result, backups handled by healthy node agent pods complete successfully and the backup queue is no longer blocked by an unrelated pod restart.- Improved volume policy performance for clusters with many pods and PVCs
Before this update, the volume policy check required scanning every pod in the namespace for every individual persistent volume claim (PVC). As a consequence, backup performance degraded significantly in clusters with large numbers of pods and PVCs. With this release, a PVC-to-Pod cache is added to eliminate redundant pod listings and reduce iteration overhead. As a result, all PV snapshot checks complete within the same second, improving performance for backup operations that were taking excessively long.
- Custom labels and annotations are supported on DPA resources
Before this update, the
DataProtectionApplication(DPA) custom resource did not support custom labels or annotations, and it was not possible to propagate them to child resources managed by the OADP Operator. As a consequence, it was not possible to exclude these child resources from external reconciliation tools at the DPA level, causing unnecessary reconciliation loops and high CPU usage. With this release, the DPA includesresourceLabelsandresourceAnnotationsfields that propagate user-defined labels and annotations to all Operator-managed resources. As a result, you can control reconciliation behavior for all managed resources from a single configuration point, reducing unnecessary processing and CPU usage.- File system backup works on clusters requiring privileged pod security context
Before this update, Velero did not provide an option to run file system backup in a privileged security context. As a consequence, file system backups failed with permission errors on OpenShift clusters that require privileged security context to access mount paths. With this release, a new
--privileged-fs-backupinstall and node agent server option was added, allowing file system backup pods to run in privileged mode independently of the existing node agent privileged flag. As a result, file system backups complete successfully on clusters that require privileged pod security context.- OADP removes OVN-K and Multus CNI annotations during pod restore
Before this update, pod network annotations injected by OVN-K and Multus container network interface (CNI) were restored during backup. As a consequence, this caused conflicts with the CNI and required manual intervention. With this release, OADP removes OVN-K and Multus CNI annotations during pod restore.
- Kopia
BackupRepositoryno longer displays wrongresticIdentifierpath Before this update, creating a backup with Kopia generated a
BackupRepositoryresource that contained aresticIdentifierfield referencing a Restic path instead of the actual Kopia repository path. As a consequence, this caused user confusion during troubleshooting. With this release, theresticIdentifierfield is removed from Kopia-typeBackupRepositoryresources, as it was only relevant for Restic repositories.- AWS S3 bucket region is auto-detected in
BackupStorageLocationconfiguration Before this update, the AWS bucket region was missing in the
DataProtectionApplicationconfiguration when using S3BackupStorageLocation(BSL) withsnapshotLocations. As a consequence, backups failed due to missing region in AWS S3 bucket. With this release, AWS S3 bucket regions are automatically detected in BSL.- BSL status message returns an error when an Azure storage bucket does not exist
Before this update, when a
DataProtectionApplicationwas configured with a non-existing Azure storage bucket, theBackupStorageLocation(BSL) validation returned a raw HTTP response in thestatus.messagefield. As a consequence, the BSL status was unclear. With this release, the BSLstatus.messagefield returns a concise, human-readable error such asContainerNotFound: The specified container does not exist.- OADP Data Mover restore operations honor
loadAffinitynode selection Before this update, the node agent
loadAffinitysetting was implemented only for backup operations, not for restore operations. As a consequence, whennodeSelectorwas configured in theDataProtectionApplicationspecification, the Data Mover pod could be scheduled to a node without a running node agent during a restore. This caused the restore to hang indefinitely. With this release, theloadAffinityconfiguration is applied to both backup and restore operations. As a result, Data Mover restore pods are correctly scheduled on nodes with running node agents, preventing restore hangs andPartiallyFailederrors.- Kopia cache no longer persists on worker nodes after backup and restore operations
Before this update, during file system restore, the
/varpartition on worker nodes increased because the cache was not deleted after backup completion. Only a node agent restart cleared the cache. As a consequence, unintended disk space consumption occurred on worker nodes after file system restore. With this release, the cache for Kopia operation stored inephemeral-storageis cleared after restore operation.- Data Mover restore no longer fails when
nodeSelectoris used Before this update, when a
DataProtectionApplication(DPA) custom resource was created with thenodeSelectorfield set in thenodeAgentparameter,nodeSelectorwas not properly handled for the Data Mover restore action. As a consequence, Data Mover restore actions partially failed because they were scheduled on a wrong node. With this release,nodeSelectoris correctly handled for Data Mover restore actions. As a result, restore operations complete successfully when the field is configured in the DPA.- Velero CLI automatically uses the CA certificate from
BackupStorageLocation Before this update, running Velero containers did not store the certificate authority (CA) certificate as a file. As a consequence, you had to manually pass the
--cacertflag with a path to a CA certificate file on every Velero command-line interface (CLI) command, or download the Velero binary and CA certificate bundle to your workstation. This made the experience inefficient in environments with custom or self-signed certificates.With this release, the Velero CLI automatically reads the CA certificate from the
BackupStorageLocation(BSL) configuration, and gracefully falls back to the caCert of the BSL if a--cacertflag is provided but points to an invalid path. As a result, you can run Velero CLI commands through theoc execalias without manually specifying--cacert, significantly simplifying day-to-day backup and restore operations in environments that use custom CA certificates.- Velero no longer restores unnecessary restore-wait init containers
Before this update, Velero did not check whether the restore-wait init container was necessary during a restore operation. As a consequence, when a namespace was backed up with CSI or native Data Mover and then was restored, Velero incorrectly injected the restore-wait init container into the application pod spec. With this release, Velero skips restoring the restore-wait init container when it is not required by the backup method used.
- The
defaultVolumesToFSBackupflag matchesdefaultVolumesToFsBackup Before this update, the
dpa.spec.configuration.velero.defaultVolumesToFSBackupflag used different casing than thebackup.spec.defaultVolumesToFsBackupflag. As a consequence, the inconsistency between the two flags could cause confusion when configuring pod volume file system backup settings. With this release, the two flags are identical.- OADP CLI automatically inherits the CA certificate from
BackupStorageLocation Before this update, the OADP command-line interface (CLI) did not inherit the certificate authority (CA) certificate from the
BackupStorageLocation(BSL) when retrieving backup logs. As a consequence, you were unable to retrieve backup logs by usingoc oadp backup logs, even though the backup itself completed successfully and theDataProtectionApplication(DPA) custom resource was correctly configured with a valid CA certificate. With this release, the OADP CLI automatically uses the CA certificate from the BSL when retrieving backup logs. As a result, backup log retrieval works correctly when a valid CA certificate is present in the BSL.- PVC no longer retains stale CSI snapshot labels after a file system restore
Before this update, during a Container Storage Interface (CSI) backup, Velero added a
velero.io/volume-snapshot-namelabel to the persistent volume claim (PVC), and this label was not removed when next file system backup and restore was performed. With this release, PVCs no longer contain leftovervelero.io/volume-snapshot-namelabels after a successful file system restore.PodVolumeBackupandPodVolumeRestoreCRs are marked as failed when the parent backup or restore failsBefore this update, when the backup or restore CR failed, the associated
PodVolumeBackupandPodVolumeRestorecustom resources (CRs) continued their data transfer. As a consequence, this led to an inconsistent state where individual volume CRs reportedCompletedwhile the parent operation wasFailed. With this release,PodVolumeBackupandPodVolumeRestoreCRs are properly canceled and marked asFailedwhen the parent backup or restore CR fails. As a result, volume-level CR statuses are consistent with the parent operation status.
1.2.1.2.3. Known issues Copy linkLink copied to clipboard!
CloudStoragebucket creation fails on Google Cloud with WIF authenticationThe
CloudStorageresource fails to detect that a Google Cloud bucket does not exist and returns aBucketCheckErrorcondition, preventing automatic bucket creation.To work around this problem, use an existing Google Cloud bucket when configuring
CloudStorageon OpenShift clusters on Google Cloud using Workload Identity Federation (WIF) authentication. As a result, the controller skips bucket creation and proceeds with the existing bucket for backup storage.- Non-admin backups of ODF encrypted volumes fail with OADP
When OADP Data Mover backs up an encrypted Red Hat OpenShift Data Foundation (ODF) volume, it creates a temporary persistent volume claim (PVC) in the
openshift-adpnamespace. The ceph-csi driver then attempts to fetch theceph-csi-kms-tokensecret fromopenshift-adpinstead of the original application namespace that has the secret. As a consequence, theNonAdminBackupfails with an error similar to the following:openshift-adp 2h11m Warning ProvisioningFailed persistentvolumeclaim/NAMESPACE-rh-cf00000c-0000-0000-0000-22abf391t6dr failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-encrypted": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed fetching token from openshift-adp/ceph-csi-kms-token: secrets "ceph-csi-kms-token" not foundTo work around this problem, copy
ceph-csi-kms-tokenand the associated Key Management Service (KMS) config map from the application namespace toopenshift-adpbefore running the backup. As a result, the ceph-csi driver can locate the encryption credentials and the backup completes successfully.- The Velero server might be terminated with an
OOMKilledevent when accessing large Kopia repositories When the Velero server accesses large Kopia repositories, for example, during backup deletion, the Velero server might be terminated by the Out Of Memory (OOM) process. This is typically due to high memory usage loading many Kopia index blobs and related repository metadata.
To work around this problem, perform the following actions:
-
Check Velero logs around the
OOMKilledevent for repository-open or index-loading activity, and note the approximate maximum memory usage observed for the Velero pod (from oc or kubectl top pods or similar). -
Increase the Velero container memory request or limit in
spec.velero.podConfigin theDataProtectionApplicationcustom resource to exceed that observed peak. -
Redeploy the Velero pod until
OOMKilledevents no longer occur. - Record the final stable memory values for your environment for future sizing guidance.
As a result, the Velero server has enough memory to load the repository metadata without being terminated.
-
Check Velero logs around the
1.2.1.2.4. Technology Preview features Copy linkLink copied to clipboard!
- Incremental QCOW2 backups for KubeVirt VMs (Technology Preview)
OADP introduces storage agnostic QCOW2 incremental backup for KubeVirt virtual machines (VMs) as a Technology Preview feature. This feature provides a lightweight alternative to traditional VolumeSnapshot-based backups by using
libvirttools to perform incremental QCOW2 backups. After an initial full backup, later backups capture only changed blocks, improving backup performance and reducing resource usage compared to Container Storage Interface (CSI) snapshot methods. This enables more frequent backup schedules with minimal impact on running VM performance.To use this feature, enable QEMU backup on VM disks (
CBT = true) and set up the volume policy tovirtfor target volumes.VolumeGroupSnapshot(Technology Preview)With
VolumeGroupSnapshot(VGS) you can group and snapshot multiple persistent volume claims (PVCs) together using label-based selection. This ensures point-in-time data consistency across related volumes, improving backup reliability for stateful applications that use multiple persistent volumes. VGS is a Technology Preview feature.
1.2.1.2.5. Removed features Copy linkLink copied to clipboard!
- Restic backup is removed
Creating file system backups with Restic as an uploader type is removed and is no longer supported. Setting
uploaderType: resticinspec.configuration.nodeAgent.uploaderTypein theDataProtectionApplication(DPA) custom resource (CR) results in a reconcile error. You can create file system backups by using Kopia instead.Note that restoring existing Restic backups remains supported. To continue restoring Restic backups, configure Kopia in the DPA CR. OADP then uses Restic to restore Restic backups.
velero-restore-helperconfig map is removedThe
fs-restore-action-configconfig map, which pointed to a separatevelero-restore-helpercontainer image for file system restore operations, is removed and is no longer supported. You can restore workloads by using the built-in restore helper binary included in the Velero image instead.- Changing PVC selected-node is removed
The Changing PVC selected-node feature, which allowed overriding the
volume.kubernetes.io/selected-nodeannotation to change the node selected for a persistent volume claim (PVC) during restore, is removed and is no longer supported. You can ensure PVCs are scheduled to the correct node by using the native KubernetesWaitForFirstConsumervolume binding mode instead.
1.2.2. Upgrading OADP 1.5 to 1.6 Copy linkLink copied to clipboard!
Learn how to upgrade your existing OpenShift API for Data Protection 1.5 installation to OADP 1.6.
Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OADP 1.1 to 1.3, upgrade first to 1.2, and then to 1.3.
1.2.2.1. Changes from OADP 1.5 to 1.6 Copy linkLink copied to clipboard!
The Velero server has been updated from version 1.16 to 1.18.
This brings the following changes:
- Parallel backup processing
- OADP supports parallel backup processing. Before this update, OADP processed backups sequentially where each backup had to complete before the next one could begin. This created bottlenecks for environments with many backup schedules or large-scale data protection needs, increasing the overall time required to complete all backup operations. With this release, multiple backups can run in parallel if they do not share namespaces. As a result, you can now define more granular or frequent backup schedules without worrying about queuing delays.
- File-level restore with VMFR from KubeVirt virtual machine backups
-
OADP introduces the virtual machine file restore (VMFR) feature. Before this release, restoring even a single file from a KubeVirt VM backup required restoring the entire VM, which was time-consuming and resource-intensive. With this release, you can selectively recover individual files from KubeVirt VM backups directly within Red Hat OpenShift Service on AWS classic architecture. VMFR also includes
RestoreItemAction, a plugin for collision prevention in multi-backup scenarios. This plugin renames PVCs during restore, ensuring unique names and avoiding conflicts, thereby improving the reliability and efficiency of VMFR backups and restores. - File-level backups with VMDP in OpenShift Virtualization environments
- OADP introduces the virtual machine data protection (VMDP) feature, a command-line tool that runs inside virtual machines (VMs). Before this update, OADP supported only snapshot-based backup of entire VMs, with no option for granular, user-driven file-level data protection from within a VM. Using VMDP, you can selectively back up and restore individual files and directories by using a Kopia-based client/server architecture without cluster administrator intervention.
- OADP CLI plugin
-
OADP includes a
kubectlcommand-line interface (CLI) plugin that provides akubectlnative interface for managing backups and restores. Before this update, you aliased to the Velero CLI to perform backup operations. Non-cluster administrators had no way to independently manage their own backups, creating a dependency on cluster admins for routine data protection tasks. As a result, admins can runkubectl oadpcommands to create, delete, and retrieve logs for backups and restores. Non-cluster admins can independently create, delete, and inspect non-admin backups within their permitted namespaces. - Configurable priority class for node agent and Velero pods
-
With this update, you can configure the
priorityClassNamefield for node agent and Velero pods in aDataProtectionApplicationobject, prioritizing these pods during resource contention. This is particularly useful after worker node outages, ensuring critical pods are scheduled first. - Wildcard support for namespace selection during backup operations
-
You can use wildcard patterns when specifying namespaces in backup operations. Before this update, you had to explicitly list every namespace in the backup object and update it each time a new namespace was created. With this release, you can use patterns such as
*-helmin the--include-namespacesflag to dynamically match multiple namespaces, simplifying backup management and reducing manual configuration overhead. VolumeSnapshotClassno longer required for CSI backup and restore-
The Container Storage Interface (CSI) backup and restore process no longer includes or requires the
VolumeSnapshotClassresource. Before this update, theVolumeSnapshotClasswas included in CSI backups and needed to be present during restore, adding an unnecessary dependency. With this update, CSI snapshot-based backups and restores complete successfully withoutVolumeSnapshotClassin the cluster, simplifying the backup resource footprint and improving restore reliability in environments where theVolumeSnapshotClassmight not be pre-configured. - Optimized Data Mover performance in containerized environments with new Go version
-
With the migration to Go 1.25, the Data Mover defaults the
MaxParallelFileReadsvalue to thecgroupCPU bandwidth limit corresponding to the container CPU limit rather than the total number of CPU cores on the node. Before this update,MaxParallelFileReadsused the full CPU count of the node unlessdatamoverConfig.ParallelFilesUploadwas explicitly set in the node-agent-config map, which could lead to overuse of resources in constrained container environments. With this update, Data Mover operations automatically respect container CPU limits, improving resource efficiency and stability without requiring manual configuration changes. - Changing PVC selected-node is removed
-
The Changing PVC selected-node feature, which allowed overriding the
volume.kubernetes.io/selected-nodeannotation to change the node selected for a persistent volume claim (PVC) during restore, is removed and is no longer supported. You can ensure PVCs are scheduled to the correct node by using the native KubernetesWaitForFirstConsumervolume binding mode instead.
1.2.2.2. Backing up the DPA configuration Copy linkLink copied to clipboard!
You must back up your current DataProtectionApplication (DPA) configuration.
Procedure
Save your current DPA configuration by running the following command:
Example command
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
1.2.2.3. Upgrading the OADP Operator Copy linkLink copied to clipboard!
You can upgrade the OpenShift API for Data Protection (OADP) Operator by using the following procedure.
Do not install OADP 1.6.0 on a OpenShift 4.21 cluster.
Prerequisites
- You have installed the latest OADP 1.5.5.
- You have backed up your data.
Procedure
- Upgrade OpenShift 4.21 to OpenShift 4.22.
-
Ensure your subscription channel for the OADP Operator is
stable. - Wait for the Operator and containers to update and restart.
1.2.2.4. Converting DPA to the new version for OADP 1.6.0 Copy linkLink copied to clipboard!
OpenShift API for Data Protection (OADP) 1.5 is not supported on OpenShift 4.22. You can convert OADP to the new 1.6 version by using the new spec.configuration.nodeAgent field and its sub-fields.
Procedure
To configure
nodeAgentdaemon set, use thespec.configuration.nodeAgentparameter in DPA. See the following example:Example
DataProtectionApplicationconfiguration... spec: configuration: nodeAgent: enable: true uploaderType: kopia ...To configure
nodeAgentdaemon set by using theConfigMapresource namednode-agent-config, see the following example configuration:Example config map
... spec: configuration: nodeAgent: backupPVC: ... loadConcurrency: ... podResources: ... restorePVC: ... ...
1.2.2.5. Verifying the upgrade Copy linkLink copied to clipboard!
You can verify the OpenShift API for Data Protection (OADP) upgrade by using the following procedure.
Procedure
Verify that the
DataProtectionApplication(DPA) has been reconciled successfully:$ oc get dpa dpa-sample -n openshift-adpExample output
NAME RECONCILED AGE dpa-sample True 2m51sNoteThe
RECONCILEDcolumn must beTrue.Verify that the installation finished by viewing the OADP resources by running the following command:
$ oc get all -n openshift-adpExample output
NAME READY STATUS RESTARTS AGE pod/node-agent-9pjz9 1/1 Running 0 3d17h pod/node-agent-fmn84 1/1 Running 0 3d17h pod/node-agent-xw2dg 1/1 Running 0 3d17h pod/openshift-adp-controller-manager-76b8bc8d7b-kgkcw 1/1 Running 0 3d17h pod/velero-64475b8c5b-nh2qc 1/1 Running 0 3d17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/openshift-adp-controller-manager-metrics-service ClusterIP 172.30.194.192 <none> 8443/TCP 3d17h service/openshift-adp-velero-metrics-svc ClusterIP 172.30.190.174 <none> 8085/TCP 3d17h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 3d17h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/openshift-adp-controller-manager 1/1 1 1 3d17h deployment.apps/velero 1/1 1 1 3d17h NAME DESIRED CURRENT READY AGE replicaset.apps/openshift-adp-controller-manager-76b8bc8d7b 1 1 1 3d17h replicaset.apps/openshift-adp-controller-manager-85fff975b8 0 0 0 3d17h replicaset.apps/velero-64475b8c5b 1 1 1 3d17h replicaset.apps/velero-8b5bc54fd 0 0 0 3d17h replicaset.apps/velero-f5c9ffb66 0 0 0 3d17hVerify the backup storage location and confirm that the
PHASEisAvailableby running the following command:$ oc get backupstoragelocations.velero.io -n openshift-adpExample output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
1.3. OADP performance Copy linkLink copied to clipboard!
1.3.1. OADP recommended network settings Copy linkLink copied to clipboard!
Keep a stable network across your OpenShift nodes, AWS Simple Storage Service (S3) storage, and cloud environments. Meeting these recommended network settings helps you ensure successful OpenShift API for Data Protection (OADP) backup and restore operations, even when using remote AWS S3 buckets.
1.3.1.1. OADP network requirements Copy linkLink copied to clipboard!
For a supported experience with OpenShift API for Data Protection (OADP), you should have a stable and resilient network across OpenShift nodes, AWS Simple Storage Service (S3)-compatible object storage, and in supported cloud environments that meet OpenShift network requirements.
For deployments that use remote S3 buckets located off-cluster with suboptimal data paths, such as high-latency or geographically distant locations, successful backup and restore operations require specific configurations. Ensure your network settings meet the following minimum requirements:
- Bandwidth (network upload speed to object storage): Greater than 2 Mbps for small backups and 10-100 Mbps depending on the data volume for larger backups.
- Packet loss: 1%
- Packet corruption: 1%
- Latency: 100 ms
Ensure that your Red Hat OpenShift Service on AWS classic architecture network performs optimally and meets Red Hat OpenShift Service on AWS classic architecture network requirements.
Although Red Hat provides support for standard backup and restore failures, it does not provide support for failures caused by network settings that do not meet the recommended thresholds.
1.4. OADP features and plugins Copy linkLink copied to clipboard!
Review OpenShift API for Data Protection (OADP) features and default plugins that integrate Velero with cloud providers to back up and restore Red Hat OpenShift Service on AWS classic architecture resources. This helps you to select the right plugins and features for your backup and restore environment.
1.4.1. OADP features Copy linkLink copied to clipboard!
Review the backup, restore, and scheduling features of OpenShift API for Data Protection (OADP) for protecting applications on Red Hat OpenShift Service on AWS classic architecture. This helps you to understand the available capabilities for your data protection strategy.
- Backup
You can use OADP to back up all applications on the OpenShift Platform, or you can filter the resources by type, namespace, or label.
OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Restore
You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the objects by namespace, PV, or label.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Schedule
- You can schedule backups at specified intervals.
- Hooks
-
You can use hooks to run commands in a container on a pod, for example,
fsfreezeto freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.
1.4.2. OADP plugins Copy linkLink copied to clipboard!
Review the default Velero plugins provided by OpenShift API for Data Protection (OADP) that integrate with storage providers to support backup and snapshot operations. This helps you to select and configure the right plugins for your cloud environment.
OADP also provides plugins for Red Hat OpenShift Service on AWS classic architecture resource backups, OpenShift Virtualization resource backups, and Container Storage Interface (CSI) snapshots.
| OADP plugin | Function | Storage location |
|---|---|---|
|
| Backs up and restores Kubernetes objects. | AWS S3 |
| Backs up and restores volumes with snapshots. | AWS EBS | |
|
| Backs up and restores Red Hat OpenShift Service on AWS classic architecture resources. [1] | Object store |
|
| Backs up and restores OpenShift Virtualization resources. [2] | Object store |
|
| Backs up and restores volumes with CSI snapshots. [3] | Cloud storage that supports CSI snapshots |
|
| Backs up and restores HyperShift hosted cluster resources. [4] | Object store |
- Mandatory.
- Virtual machine disks are backed up with CSI snapshots or Restic.
The
csiplugin uses the Kubernetes CSI snapshot API.-
OADP 1.1 or later uses
snapshot.storage.k8s.io/v1 -
OADP 1.0 uses
snapshot.storage.k8s.io/v1beta1
-
OADP 1.1 or later uses
-
Do not add the
hypershiftplugin in theDataProtectionApplicationcustom resource if the cluster is not a HyperShift hosted cluster.
1.4.3. About OADP Velero plugins Copy linkLink copied to clipboard!
Review how to configure default cloud provider plugins or install custom plugins during the OADP deployment to connect your specific storage solutions. This helps you to successfully back up and restore resources across your environments.
1.4.3.1. Default Velero cloud provider plugins Copy linkLink copied to clipboard!
You can install any of the following default Velero cloud provider plugins when you configure the oadp_v1alpha1_dpa.yaml file during deployment:
-
aws(Amazon Web Services) -
openshift(OpenShift Velero plugin) -
csi(Container Storage Interface) -
kubevirt(KubeVirt)
You specify the desired default plugins in the oadp_v1alpha1_dpa.yaml file during deployment.
The following .yaml file installs the openshift, aws, azure, and gcp plugins:
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-sample
spec:
configuration:
velero:
defaultPlugins:
- openshift
- aws
- azure
- gcp
1.4.3.2. Custom Velero plugins Copy linkLink copied to clipboard!
You can install a custom Velero plugin by specifying the plugin image and name when you configure the oadp_v1alpha1_dpa.yaml file during deployment.
You specify the desired custom plugins in the oadp_v1alpha1_dpa.yaml file during deployment.
The following .yaml file installs the default openshift, azure, and gcp plugins and a custom plugin that has the name custom-plugin-example and the image quay.io/example-repo/custom-velero-plugin:
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa-sample
spec:
configuration:
velero:
defaultPlugins:
- openshift
- azure
- gcp
customPlugins:
- name: custom-plugin-example
image: quay.io/example-repo/custom-velero-plugin
1.4.4. OADP and FIPS Copy linkLink copied to clipboard!
Federal Information Processing Standards (FIPS) are a set of computer security standards developed by the United States federal government inline with the Federal Information Security Management Act (FISMA).
OpenShift API for Data Protection (OADP) has been tested and works on FIPS-enabled Red Hat OpenShift Service on AWS classic architecture clusters.
1.4.5. Avoiding the Velero plugin panic error Copy linkLink copied to clipboard!
Label a custom Backup Storage Location (BSL) to resolve Velero plugin panic errors during imagestream backups. This helps you to ensure the OADP controller creates the required registry secret when you manage the BSL outside the DataProtectionApplication (DPA) CR.
A missing secret can cause a panic error for the Velero plugin during image stream backups. When the backup and the BSL are managed outside the scope of the DPA, the OADP controller does not create the relevant oadp-<bsl_name>-<bsl_provider>-registry-secret parameter.
During the backup operation, the OpenShift Velero plugin panics on the imagestream backup, with the following panic error:
024-02-27T10:46:50.028951744Z time="2024-02-27T10:46:50Z" level=error msg="Error backing up item"
backup=openshift-adp/<backup name> error="error executing custom action (groupResource=imagestreams.image.openshift.io,
namespace=<BSL Name>, name=postgres): rpc error: code = Aborted desc = plugin panicked:
runtime error: index out of range with length 1, stack trace: goroutine 94…
Procedure
Label the custom BSL with the relevant label by using the following command:
$ oc label backupstoragelocations.velero.io <bsl_name> app.kubernetes.io/component=bslAfter the BSL is labeled, wait until the DPA reconciles.
NoteYou can force the reconciliation by making any minor change to the DPA itself.
Verification
After the DPA is reconciled, confirm that the parameter has been created and that the correct registry data has been populated into it by entering the following command:
$ oc -n openshift-adp get secret/oadp-<bsl_name>-<bsl_provider>-registry-secret -o json | jq -r '.data'
1.4.6. Workaround for OpenShift ADP Controller segmentation fault Copy linkLink copied to clipboard!
Define either velero or cloudstorage in your Data Protection Application (DPA) configuration to prevent indefinite pod crashes. This configuration resolves a segmentation fault in the openshift-adp-controller-manager pod that occurs when both components are enabled.
The openshift-adp-controller-manager pod fails with a crash loop segmentation fault due to the following settings:
-
If you define both
veleroandcloudstorage, theopenshift-adp-controller-managerfails. -
If you do not define both
veleroandcloudstorage, theopenshift-adp-controller-managerfails.
See OADP-1054 for more information.
1.5. OADP use cases Copy linkLink copied to clipboard!
1.5.1. Backing up workloads on OADP with Red Hat OpenShift Service on AWS classic architecture Copy linkLink copied to clipboard!
To back up and restore workloads on Red Hat OpenShift Service on AWS classic architecture, you can use OADP. You can create a backup of a workload, restore it from the backup, and verify the restoration. You can also clean up the OADP Operator, backup storage, and AWS resources when they are no longer needed.
1.5.1.1. Example: Performing a backup with OADP and Red Hat OpenShift Service on AWS classic architecture Copy linkLink copied to clipboard!
Perform a backup by using OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS classic architecture. The following example hello-world application has no persistent volumes (PVs) attached.
Either Data Protection Application (DPA) configuration will work.
Procedure
Create a workload to back up by running the following commands:
$ oc create namespace hello-world$ oc new-app -n hello-world --image=docker.io/openshift/hello-openshiftExpose the route by running the following command:
$ oc expose service/hello-openshift -n hello-worldCheck that the application is working by running the following command:
$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`You should see an output similar to the following example:
Hello OpenShift!Back up the workload by running the following command:
$ cat << EOF | oc create -f - apiVersion: velero.io/v1 kind: Backup metadata: name: hello-world namespace: openshift-adp spec: includedNamespaces: - hello-world storageLocation: ${CLUSTER_NAME}-dpa-1 ttl: 720h0m0s EOFWait until the backup is complete, and then run the following command:
$ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"You should see an output similar to the following example:
{ "completionTimestamp": "2022-09-07T22:20:44Z", "expiration": "2022-10-07T22:20:22Z", "formatVersion": "1.1.0", "phase": "Completed", "progress": { "itemsBackedUp": 58, "totalItems": 58 }, "startTimestamp": "2022-09-07T22:20:22Z", "version": 1 }Delete the demo workload by running the following command:
$ oc delete ns hello-worldRestore the workload from the backup by running the following command:
$ cat << EOF | oc create -f - apiVersion: velero.io/v1 kind: Restore metadata: name: hello-world namespace: openshift-adp spec: backupName: hello-world EOFWait for the Restore to finish by running the following command:
$ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"You should see an output similar to the following example:
{ "completionTimestamp": "2022-09-07T22:25:47Z", "phase": "Completed", "progress": { "itemsRestored": 38, "totalItems": 38 }, "startTimestamp": "2022-09-07T22:25:28Z", "warnings": 9 }Check that the workload is restored by running the following command:
$ oc -n hello-world get podsYou should see an output similar to the following example:
NAME READY STATUS RESTARTS AGE hello-openshift-9f885f7c6-kdjpj 1/1 Running 0 90sCheck the JSONPath by running the following command:
$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`You should see an output similar to the following example:
Hello OpenShift!NoteFor troubleshooting tips, see the troubleshooting documentation.
1.5.1.2. Cleaning up a cluster after a backup with OADP Copy linkLink copied to clipboard!
Uninstall the OpenShift API for Data Protection (OADP) Operator together with the backups and the S3 bucket from the hello-world example.
Procedure
Delete the workload by running the following command:
$ oc delete ns hello-worldDelete the Data Protection Application (DPA) by running the following command:
$ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpaDelete the cloud storage by running the following command:
$ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadpWarningIf this command hangs, you might need to delete the finalizer by running the following command:
$ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=mergeIf the Operator is no longer required, remove it by running the following command:
$ oc -n openshift-adp delete subscription oadp-operatorRemove the namespace from the Operator:
$ oc delete ns openshift-adpIf the backup and restore resources are no longer required, remove them from the cluster by running the following command:
$ oc delete backups.velero.io hello-worldTo delete backup, restore and remote objects in AWS S3 run the following command:
$ velero backup delete hello-worldIf you no longer need the Custom Resource Definitions (CRD), remove them from the cluster by running the following command:
$ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; doneDelete the AWS S3 bucket by running the following commands:
$ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive$ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadpDetach the policy from the role by running the following command:
$ aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"Delete the role by running the following command:
$ aws iam delete-role --role-name "${ROLE_NAME}"
1.5.2. OpenShift API for Data Protection (OADP) restore use case Copy linkLink copied to clipboard!
Following is a use case for using OADP to restore a backup to a different namespace.
1.5.2.1. Restoring an application to a different namespace using OADP Copy linkLink copied to clipboard!
Restore a backup of an application by using OADP to a new target namespace, test-restore-application. To restore a backup, you create a restore custom resource (CR) as shown in the following example. In the restore CR, the source namespace refers to the application namespace that you included in the backup. You then verify the restore by changing your project to the new restored namespace and verifying the resources.
Prerequisites
- You installed the OADP Operator.
- You have the backup of an application to be restored.
Procedure
Create a restore CR as shown in the following example:
apiVersion: velero.io/v1 kind: Restore metadata: name: test-restore namespace: openshift-adp spec: backupName: <backup_name> restorePVs: true namespaceMapping: <application_namespace>: test-restore-applicationwhere:
test-restore- Specifies the name of the restore CR.
<backup_name>- Specifies the name of the backup.
<application_namespace>-
Specifies the target namespace to restore to.
namespaceMappingmaps the source application namespace to the target application namespace.test-restore-applicationis the name of target namespace where you want to restore the backup.
Apply the restore CR by running the following command:
$ oc apply -f <restore_cr_filename>
Verification
Verify that the restore is in the
Completedphase by running the following command:$ oc describe restores.velero.io <restore_name> -n openshift-adpChange to the restored namespace
test-restore-applicationby running the following command:$ oc project test-restore-applicationVerify the restored resources such as persistent volume claim (pvc), service (svc), deployment, secret, and config map by running the following command:
$ oc get pvc,svc,deployment,secret,configmapExample output
NAME STATUS VOLUME persistentvolumeclaim/mysql Bound pvc-9b3583db-...-14b86 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql ClusterIP 172....157 <none> 3306/TCP 2m56s service/todolist ClusterIP 172.....15 <none> 8000/TCP 2m56s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 0/1 1 0 2m55s NAME TYPE DATA AGE secret/builder-dockercfg-6bfmd kubernetes.io/dockercfg 1 2m57s secret/default-dockercfg-hz9kz kubernetes.io/dockercfg 1 2m57s secret/deployer-dockercfg-86cvd kubernetes.io/dockercfg 1 2m57s secret/mysql-persistent-sa-dockercfg-rgp9b kubernetes.io/dockercfg 1 2m57s NAME DATA AGE configmap/kube-root-ca.crt 1 2m57s configmap/openshift-service-ca.crt 1 2m57s
1.6. Installing and configuring OADP Copy linkLink copied to clipboard!
1.6.1. Installing OADP Copy linkLink copied to clipboard!
Use OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS classic architecture clusters to back up and restore application data.
Before installing OpenShift API for Data Protection (OADP), you must set up role and policy credentials for OADP so that it can use the Amazon Web Services API.
This process is performed in the following two stages:
- Prepare AWS credentials
- Install the OADP Operator and give it an IAM role
1.6.1.1. Preparing AWS credentials for OADP Copy linkLink copied to clipboard!
Prepare and configure an Amazon Web Services account to install OpenShift API for Data Protection (OADP).
Procedure
Create the following environment variables by running the following commands:
ImportantChange the cluster name to match your cluster, and ensure you are logged into the cluster as an administrator. Ensure that all fields are outputted correctly before continuing.
$ export CLUSTER_NAME=<my_cluster>Replace
<my_cluster>with your cluster name.$ export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id)$ export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)$ export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.')$ export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"$ export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"$ mkdir -p ${SCRATCH}$ echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"On the AWS account, create an IAM policy to allow access to AWS S3:
Check to see if the policy exists by running the following command:
$ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text)-
RosaOadp: ReplaceRosaOadpwith your policy name.
-
Enter the following command to create the policy JSON file and then create the policy:
NoteIf the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the
ifstatement intentionally skips the policy creation.$ if [[ -z "${POLICY_ARN}" ]]; then cat << EOF > ${SCRATCH}/policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:PutBucketTagging", "s3:GetBucketTagging", "s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration", "s3:PutLifecycleConfiguration", "s3:GetLifecycleConfiguration", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucketMultipartUploads", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "ec2:DescribeSnapshots", "ec2:DescribeVolumes", "ec2:DescribeVolumeAttribute", "ec2:DescribeVolumesModifications", "ec2:DescribeVolumeStatus", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" } ]} EOF POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \ --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \ --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \ --output text) fi-
SCRATCH:SCRATCHis a name for a temporary directory created for the environment variables.
-
View the policy ARN by running the following command:
$ echo ${POLICY_ARN}
Create an IAM role trust policy for the cluster:
Create the trust policy file by running the following command:
$ cat <<EOF > ${SCRATCH}/trust-policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_ENDPOINT}:sub": [ "system:serviceaccount:openshift-adp:openshift-adp-controller-manager", "system:serviceaccount:openshift-adp:velero"] } } }] } EOFCreate the role by running the following command:
$ ROLE_ARN=$(aws iam create-role --role-name \ "${ROLE_NAME}" \ --assume-role-policy-document file://${SCRATCH}/trust-policy.json \ --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID} \ Key=rosa_openshift_version,Value=${CLUSTER_VERSION} \ Key=rosa_role_prefix,Value=ManagedOpenShift \ Key=operator_namespace,Value=openshift-adp \ Key=operator_name,Value=openshift-oadp \ --query Role.Arn --output text)View the role ARN by running the following command:
$ echo ${ROLE_ARN}
Attach the IAM policy to the IAM role by running the following command:
$ aws iam attach-role-policy --role-name "${ROLE_NAME}" \ --policy-arn ${POLICY_ARN}
1.6.1.2. Installing the OADP Operator and providing the IAM role Copy linkLink copied to clipboard!
Install OpenShift API for Data Protection (OADP) on clusters with AWS STS. AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. Red Hat OpenShift Service on AWS classic architecture with STS is the recommended credential mode.
Restic is unsupported.
Kopia file system backup (FSB) is supported when backing up file systems that do not support Container Storage Interface (CSI) snapshots.
Example file systems include the following:
- Amazon Elastic File System (EFS)
- Network File System (NFS)
-
emptyDirvolumes - Local volumes
For backing up volumes on clusters with AWS STS, OADP recommends native snapshots and Container Storage Interface (CSI) snapshots. Data Mover backups are supported, but can be slower than native snapshots.
In a Red Hat OpenShift Service on AWS classic architecture cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.
Prerequisites
-
A Red Hat OpenShift Service on AWS classic architecture cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including
ROLE_ARN, for each cluster.
Procedure
Create a Red Hat OpenShift Service on AWS classic architecture secret from your AWS token file by entering the following commands:
Create the credentials file:
$ cat <<EOF > ${SCRATCH}/credentials [default] role_arn = ${ROLE_ARN} web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token region = <aws_region> EOFReplace
<aws_region>with the AWS region to use for the STS endpoint.Create a namespace for OADP:
$ oc create namespace openshift-adpCreate the Red Hat OpenShift Service on AWS classic architecture secret:
$ oc -n openshift-adp create secret generic cloud-credentials \ --from-file=${SCRATCH}/credentialsNoteIn Red Hat OpenShift Service on AWS classic architecture versions 4.15 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the Red Hat OpenShift Service on AWS classic architecture web console, for more information see Installing from software catalog using the web console.
The preceding secret is created automatically by CCO.
Install the OADP Operator:
-
In the Red Hat OpenShift Service on AWS classic architecture web console, browse to Ecosystem
Software Catalog. - Search for the OADP Operator.
- In the role_ARN field, paste the role_arn that you created previously and click Install.
-
In the Red Hat OpenShift Service on AWS classic architecture web console, browse to Ecosystem
Create AWS cloud storage using your AWS credentials by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: CloudStorage metadata: name: ${CLUSTER_NAME}-oadp namespace: openshift-adp spec: creationSecret: key: credentials name: cloud-credentials enableSharedConfig: true name: ${CLUSTER_NAME}-oadp provider: aws region: $REGION EOFCheck your application’s storage default storage class by entering the following command:
$ oc get pvc -n <namespace>NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE applog Bound pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8 1Gi RWO gp3-csi 4d19h mysql Bound pvc-16b8e009-a20a-4379-accc-bc81fedd0621 1Gi RWO gp3-csi 4d19hGet the storage class by running the following command:
$ oc get storageclassNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 4d21h gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21hNoteThe following storage classes will work:
- gp3-csi
- gp2-csi
- gp3
- gp2
If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.
Create the
DataProtectionApplicationresource to configure the connection to the storage where the backups and volume snapshots are stored:If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ${CLUSTER_NAME}-dpa namespace: openshift-adp spec: backupImages: true features: dataMover: enable: false backupLocations: - bucket: cloudStorageRef: name: ${CLUSTER_NAME}-oadp credential: key: credentials name: cloud-credentials prefix: velero default: true config: region: ${REGION} configuration: velero: defaultPlugins: - openshift - aws - csi nodeAgent: enable: false uploaderType: kopia EOFwhere:
backupImages-
Red Hat OpenShift Service on AWS classic architecture supports internal image backup. Set this field to
falseif you do not want to use image backup. nodeAgent-
See the important note regarding the
nodeAgentattribute at the end of this procedure. uploaderType-
Specifies the type of uploader. The built-in Data Mover uses Kopia as the default uploader mechanism regardless of the value of the
uploaderTypefield.
If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:
$ cat << EOF | oc create -f - apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ${CLUSTER_NAME}-dpa namespace: openshift-adp spec: backupImages: true backupLocations: - bucket: cloudStorageRef: name: ${CLUSTER_NAME}-oadp credential: key: credentials name: cloud-credentials prefix: velero default: true config: region: ${REGION} configuration: velero: defaultPlugins: - openshift - aws nodeAgent: enable: false uploaderType: restic snapshotLocations: - velero: config: credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials enableSharedConfig: "true" profile: default region: ${REGION} provider: aws EOFwhere:
backupImages-
Red Hat OpenShift Service on AWS classic architecture supports internal image backup. Set this field to
falseif you do not want to use image backup. nodeAgent-
See the important note regarding the
nodeAgentattribute at the end of this procedure. credentialsFile- Specifies the mounted location of the bucket credential on the pod.
enableSharedConfig-
Specifies whether the
snapshotLocationscan share or reuse the credential defined for the bucket. profile- Specifies the profile name set in the AWS credentials file.
regionSpecifies your AWS region. This must be the same as the cluster region.
You are now ready to back up and restore Red Hat OpenShift Service on AWS classic architecture applications, as described in Backing up applications.
ImportantThe
enableparameter ofresticis set tofalsein this configuration, because OADP does not support Restic in Red Hat OpenShift Service on AWS classic architecture environments.
If you want to use two different clusters for backing up and restoring, the two clusters must have the same AWS S3 storage names in both the cloud storage CR and the OADP
DataProtectionApplicationconfiguration.
1.6.1.3. Updating the IAM role ARN in the OADP Operator subscription Copy linkLink copied to clipboard!
Update the OADP Operator subscription to fix an installation error due to incorrect IAM role Amazon Resource Name (ARN).
While installing the OADP Operator on a Red Hat OpenShift Service on AWS classic architecture cluster, if you provide an incorrect IAM role Amazon Resource Name (ARN), the openshift-adp-controller pod gives an error. The credential requests that are generated contain the wrong IAM role ARN. To update the credential requests object with the correct IAM role ARN, you can edit the OADP Operator subscription and patch the IAM role ARN with the correct value. By editing the OADP Operator subscription, you do not have to uninstall and reinstall OADP to update the IAM role ARN.
Prerequisites
- You have a Red Hat OpenShift Service on AWS classic architecture cluster with the required access and tokens.
- You have installed OADP on the Red Hat OpenShift Service on AWS classic architecture cluster.
Procedure
To verify that the OADP subscription has the wrong IAM role ARN environment variable set, run the following command:
$ oc get sub -o yaml redhat-oadp-operatorExample subscription
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: annotations: creationTimestamp: "2025-01-15T07:18:31Z" generation: 1 labels: operators.coreos.com/redhat-oadp-operator.openshift-adp: "" name: redhat-oadp-operator namespace: openshift-adp resourceVersion: "77363" uid: 5ba00906-5ad2-4476-ae7b-ffa90986283d spec: channel: stable-1.4 config: env: - name: ROLEARN value: arn:aws:iam::11111111:role/wrong-role-arn installPlanApproval: Manual name: redhat-oadp-operator source: prestage-operators sourceNamespace: openshift-marketplace startingCSV: oadp-operator.v1.4.2where:
ROLEARN-
Verify the value of
ROLEARNyou want to update.
Update the
ROLEARNfield of the subscription with the correct role ARN by running the following command:$ oc patch subscription redhat-oadp-operator -p '{"spec": {"config": {"env": [{"name": "ROLEARN", "value": "<role_arn>"}]}}}' --type='merge'where:
<role_arn>-
Specifies the IAM role ARN to be updated. For example,
arn:aws:iam::160…..6956:role/oadprosa…..8wlf.
Verify that the
secretobject is updated with correct role ARN value by running the following command:$ oc get secret cloud-credentials -o jsonpath='{.data.credentials}' | base64 -dExample output
[default] sts_regional_endpoints = regional role_arn = arn:aws:iam::160.....6956:role/oadprosa.....8wlf web_identity_token_file = /var/run/secrets/openshift/serviceaccount/tokenConfigure the
DataProtectionApplicationcustom resource (CR) manifest file as shown in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-rosa-dpa namespace: openshift-adp spec: backupLocations: - bucket: config: region: us-east-1 cloudStorageRef: name: <cloud_storage> credential: name: cloud-credentials key: credentials prefix: velero default: true configuration: velero: defaultPlugins: - aws - openshiftwhere:
<cloud_storage>-
Specifies the
CloudStorageCR.
Create the
DataProtectionApplicationCR by running the following command:$ oc create -f <dpa_manifest_file>Verify that the
DataProtectionApplicationCR is reconciled and thestatusis set to"True"by running the following command:$ oc get dpa -n openshift-adp -o yamlExample
DataProtectionApplicationapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... status: conditions: - lastTransitionTime: "2023-07-31T04:48:12Z" message: Reconcile complete reason: Complete status: "True" type: ReconciledVerify that the
BackupStorageLocationCR is in an available state by running the following command:$ oc get backupstoragelocations.velero.io -n openshift-adpExample
BackupStorageLocationNAME PHASE LAST VALIDATED AGE DEFAULT ts-dpa-1 Available 3s 6s true
1.7. Uninstalling OADP Copy linkLink copied to clipboard!
1.7.1. Uninstalling the OpenShift API for Data Protection Copy linkLink copied to clipboard!
You uninstall the OpenShift API for Data Protection (OADP) by deleting the OADP Operator.
1.8. OADP backing up Copy linkLink copied to clipboard!
1.8.1. Backing up applications Copy linkLink copied to clipboard!
Back up applications by creating a Backup custom resource (CR) using snapshots, CSI, or File System Backup with Kopia or Restic.
Frequent backups might consume storage on the backup storage location. Check the frequency of backups, retention time, and the amount of data of the persistent volumes (PVs) if using non-local backups, for example, S3 buckets. Because all backups are retained until they expire, check the time to live (TTL) setting of the schedule.
Review the following information regarding backing up applications by using OADP:
The Backup CR creates backup files for Kubernetes resources and internal images on S3 object storage.
The …/.snapshot directory is a snapshot copy directory, which is used by several NFS servers. This directory has read-only access by default, so Velero cannot restore to this directory.
Do not give Velero write access to the .snapshot directory, and disable client access to this directory.
1.8.1.1. Previewing resources before running backup and restore Copy linkLink copied to clipboard!
Preview the backup and restore resources in advance by doing a dry run of the backup and restore operations. This helps you to verify which resources will be included before committing to a full backup or restore.
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
$ velero backup create <backup-name> --snapshot-volumes falseSpecify the value of
--snapshot-volumesparameter asfalse.To know more details about the backup resources, run the following command:
$ velero describe backup <backup_name> --detailsReplace
<backup_name>with the name of the backup.To preview the resources included in the restore before running the actual restore, run the following command:
$ velero restore create --from-backup <backup_name>Replace
<backup_name>with the name of the backup.ImportantThe
velero restore createcommand creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
$ velero describe restore <restore_name> --detailsReplace
<restore_name>with the name of the restore.
1.8.2. Creating a Backup CR Copy linkLink copied to clipboard!
Back up Kubernetes resources, internal images, and persistent volumes (PVs) by creating a Backup custom resource (CR). This helps you to protect your application data and configuration for disaster recovery.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplicationCR must be in aReadystate. Backup location prerequisites:
- You must have S3 object storage configured for Velero.
-
You must have a backup location configured in the
DataProtectionApplicationCR.
Snapshot location prerequisites:
- Your cloud provider must have a native snapshot API or support Container Storage Interface (CSI) snapshots.
-
For CSI snapshots, you must create a
VolumeSnapshotClassCR to register the CSI driver. -
You must have a volume location configured in the
DataProtectionApplicationCR.
Procedure
Retrieve the
backupStorageLocationsCRs by entering the following command:$ oc get backupstoragelocations.velero.io -n openshift-adpNAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31mCreate a
BackupCR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> labels: velero.io/storage-location: default namespace: openshift-adp spec: hooks: {} includedNamespaces: - <namespace> includedResources: [] excludedResources: [] storageLocation: <velero-sample-1> ttl: 720h0m0s labelSelector: matchLabels: app: <label_1> app: <label_2> app: <label_3> orLabelSelectors: - matchLabels: app: <label_1> app: <label_2> app: <label_3>where:
<namespace>- Specifies an array of namespaces to back up.
includedResources-
Optional: Specifies an array of resources to include in the backup. Resources might be shortcuts (for example,
poforpods) or fully-qualified. If unspecified, all resources are included. excludedResources-
Optional: Specifies an array of resources to exclude from the backup. Resources might be shortcuts (for example,
poforpods) or fully-qualified. <velero-sample-1>-
Specifies the name of the
backupStorageLocationsCR. labelSelector- Specifies a map of {key,value} pairs of backup resources that have all the specified labels.
orLabelSelectors- Specifies a map of {key,value} pairs of backup resources that have one or more of the specified labels.
Verify that the status of the
BackupCR isCompleted:$ oc get backups.velero.io -n openshift-adp <backup> -o jsonpath='{.status.phase}'
1.8.3. Parallel backup processing in OADP Copy linkLink copied to clipboard!
With parallel backup processing in OpenShift API for Data Protection, you can back up several applications from different namespaces simultaneously. This concurrent processing ensures that a single large backup task does not delay the backups for other applications in your cluster.
By default, Velero processes backup resources sequentially - one resource in the InProgress phase at a time. Parallel backup changes this behavior to handle backups concurrently.
The following considerations apply for enabling parallel backups in OADP:
- Resource allocation: Increasing the concurrency settings requires additional CPU and memory for the Velero container. Monitor resource consumption to ensure cluster stability.
- Backup scope: Parallel processing provides the most benefit for backups that contain a large number of Kubernetes resources or many smaller volumes. Backups dominated by a few large volumes see less benefit because most time is spent waiting for asynchronous data movement to complete.
Namespace partitioning: To keep data integrity, OADP uses namespace-level isolation when processing parallel backups. Two backups cannot run simultaneously if they share any included namespaces.
Expand Table 1.4. Parallel backup execution examples Scenario Included namespaces Processing behavior Backup A and Backup B
namespace1andnamespace2Parallel: The namespaces are isolated.
Backup A and Backup B
namespace1andnamespace1Serialized: The shared namespace creates a lock.
Full cluster and Backup A
All namespaces and
namespace1Serialized: A cluster-wide backup includes all namespaces and locks the entire cluster.
1.8.4. Enabling parallel backup processing Copy linkLink copied to clipboard!
By default, OpenShift API for Data Protection processes only one backup in the InProgress phase at a time. Configure the DataProtectionApplication (DPA) custom resource (CR) to run several backups simultaneously to prevent smaller backups from being queued behind larger operations.
Prerequisites
-
You must be logged in as a user with
cluster-adminprivileges. - You must have a DPA CR configured and deployed in your cluster.
Procedure
Edit your DPA CR:
$ oc edit dpa <dpa_name> -n openshift-adpAdd the
concurrentBackupsfield in thespec.configuration.velerosection of your DPA CR:configuration: velero: concurrentBackups: <integer_limit> defaultPlugins: - openshift - aws - csiReplace
<integer_limit>with the maximum number of backups you want OADP to process simultaneously. The default value is1. Backups that target overlapping namespaces are automatically serialized.
Verification
Verify that all backups are in the
InProgressphase simultaneously:$ oc get backups.velero.io -n openshift-adpExample output
NAME STATUS CREATED EXPIRES backup-namespace1 InProgress 2026-04-28 10:00:00 +0000 UTC 29d backup-namespace2 InProgress 2026-04-28 10:00:05 +0000 UTC 29d
1.8.5. Creating backup hooks Copy linkLink copied to clipboard!
Create backup hooks to run commands in a container in a pod by editing the Backup custom resource (CR). This helps you to run pre-backup and post-backup actions such as quiescing a database or flushing data to disk.
The commands can be configured to performed before any custom action processing (Pre hooks), or after all custom actions have been completed and any additional items specified by the custom action have been backed up (Post hooks).
Procedure
Add a hook to the
spec.hooksblock of theBackupCR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> excludedNamespaces: - <namespace> includedResources: [] - pods excludedResources: [] labelSelector: matchLabels: app: velero component: server pre: - exec: container: <container> command: - /bin/uname - -a onError: Fail timeout: 30s post: ...where:
<namespace>- Optional: Specifies the namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
excludedNamespaces- Optional: Specifies the namespaces to which the hook does not apply.
pods- Currently, pods are the only supported resource that hooks can apply to.
excludedResources- Optional: Specifies the resources to which the hook does not apply.
labelSelector- Optional: This hook only applies to objects matching the label. If this value is not specified, the hook applies to all objects.
pre- Specifies an array of hooks to run before the backup.
<container>- Optional: Specifies the container in which the command runs. If the container is not specified, the command runs in the first container in the pod.
/bin/uname-
Specifies the entry point for the
initcontainer being added. onError: Fail-
Specifies the error handling behavior. Allowed values are
FailandContinue. The default isFail. timeout: 30s-
Optional: Specifies how long to wait for the commands to run. The default is
30s. post- Specifies an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.
1.8.6. Scheduling backups using Schedule CR Copy linkLink copied to clipboard!
Schedule backup operations by creating a Schedule custom resource (CR) with a Cron expression. This helps you to automate recurring backups of your application data at regular intervals.
You schedule backups by creating a Schedule custom resource (CR) instead of a Backup CR.
Leave enough time in your backup schedule for a backup to finish before another backup is created.
For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplicationCR must be in aReadystate.
Procedure
Retrieve the
backupStorageLocationsCRs:$ oc get backupStorageLocations -n openshift-adpNAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31mCreate a
ScheduleCR, as in the following example:$ cat << EOF | oc apply -f - apiVersion: velero.io/v1 kind: Schedule metadata: name: <schedule> namespace: openshift-adp spec: schedule: 0 7 * * * template: hooks: {} includedNamespaces: - <namespace> storageLocation: <velero-sample-1> defaultVolumesToFsBackup: true ttl: 720h0m0s EOFwhere:
schedule: 0 7 * * *Specifies the
cronexpression to schedule the backup, for example,0 7 * * *to perform a backup every day at 7:00.NoteTo schedule a backup at specific intervals, enter the
<duration_in_minutes>in the following format:schedule: "*/10 * * * *"Enter the minutes value between quotation marks (
" ").<namespace>- Specifies an array of namespaces to back up.
<velero-sample-1>-
Specifies the name of the
backupStorageLocationsCR. defaultVolumesToFsBackup: true-
Optional: In OADP version 1.2 and later, add the
defaultVolumesToFsBackup: truekey-value pair to your configuration when performing backups of volumes with Restic. In OADP version 1.1, add thedefaultVolumesToRestic: truekey-value pair when you back up volumes with Restic.
Verification
Verify that the status of the
ScheduleCR isCompletedafter the scheduled backup runs:$ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'
1.8.7. Deleting backups Copy linkLink copied to clipboard!
Delete a backup by creating the DeleteBackupRequest custom resource (CR) or by running the velero backup delete command. This helps you to free up storage space and remove outdated backup artifacts.
The volume backup artifacts are deleted at different times depending on the backup method:
- Restic: The artifacts are deleted in the next full maintenance cycle, after the backup is deleted.
- Container Storage Interface (CSI): The artifacts are deleted immediately when the backup is deleted.
- Kopia: The artifacts are deleted after three full maintenance cycles of the Kopia repository, after the backup is deleted.
1.8.7.1. Deleting a backup by creating a DeleteBackupRequest CR Copy linkLink copied to clipboard!
Delete a backup by creating a DeleteBackupRequest custom resource (CR). This helps you to remove specific backups and their associated volume artifacts from storage.
Prerequisites
- You have run a backup of your application.
Procedure
Create a
DeleteBackupRequestCR manifest file:apiVersion: velero.io/v1 kind: DeleteBackupRequest metadata: name: deletebackuprequest namespace: openshift-adp spec: backupName: <backup_name>Replace
<backup_name>with the name of the backup.Apply the
DeleteBackupRequestCR to delete the backup:$ oc apply -f <deletebackuprequest_cr_filename>
1.8.7.2. Deleting a backup by using the Velero CLI Copy linkLink copied to clipboard!
Delete a backup by using the Velero CLI to run the velero backup delete command. This helps you to quickly remove backups and their associated volume artifacts from storage.
Prerequisites
- You have run a backup of your application.
- You downloaded the Velero CLI and can access the Velero binary in your cluster.
Procedure
To delete the backup, run the following Velero command:
$ velero backup delete <backup_name> -n openshift-adpReplace
<backup_name>with the name of the backup.
1.8.7.3. About Kopia repository maintenance Copy linkLink copied to clipboard!
Kopia repository maintenance has two types, quick and full, that run automatically to optimize index performance and perform garbage collection. Knowing the two types helps you understand the maintenance cycle and how long it takes for backup artifacts to be deleted.
- Quick maintenance
- Runs every hour to keep the number of index blobs (n) low. A high number of indexes negatively affects the performance of Kopia operations.
- Does not delete any metadata from the repository without ensuring that another copy of the same metadata exists.
- Full maintenance
- Runs every 24 hours to perform garbage collection of repository contents that are no longer needed.
-
snapshot-gc, a full maintenance task, finds all files and directory listings that are no longer accessible from snapshot manifests and marks them as deleted. - A full maintenance is a resource-costly operation, as it requires scanning all directories in all snapshots that are active in the cluster.
1.8.7.3.1. Kopia maintenance in OADP Copy linkLink copied to clipboard!
The repo-maintain-job jobs are executed in the namespace where OADP is installed, as shown in the following example:
pod/repo-maintain-job-173...2527-2nbls 0/1 Completed 0 168m
pod/repo-maintain-job-173....536-fl9tm 0/1 Completed 0 108m
pod/repo-maintain-job-173...2545-55ggx 0/1 Completed 0 48m
You can check the logs of the repo-maintain-job for more details about the cleanup and the removal of artifacts in the backup object storage. You can find a note, as shown in the following example, in the repo-maintain-job when the next full cycle maintenance is due:
not due for full maintenance cycle until 2024-00-00 18:29:4
Three successful executions of a full maintenance cycle are required for the objects to be deleted from the backup object storage. This means you can expect up to 72 hours for all the artifacts in the backup object storage to be deleted.
1.8.7.4. Deleting a backup repository Copy linkLink copied to clipboard!
Delete the backuprepository custom resource (CR) to complete the backup deletion process. This helps you to ensure that all backup metadata and artifacts are fully removed from storage.
After you delete the backup, and after the Kopia repository maintenance cycles to delete the related artifacts are complete, the backup is no longer referenced by any metadata or manifest objects.
Prerequisites
- You have deleted the backup of your application.
- You have waited up to 72 hours after the backup is deleted. This time frame allows Kopia to run the repository maintenance cycles.
Procedure
To get the name of the backup repository CR for a backup, run the following command:
$ oc get backuprepositories.velero.io -n openshift-adpTo delete the backup repository CR, run the following command:
$ oc delete backuprepository <backup_repository_name> -n openshift-adpReplace
<backup_repository_name>with the name of the backup repository.
1.9. OADP restoring Copy linkLink copied to clipboard!
1.9.1. Restoring applications Copy linkLink copied to clipboard!
Restore application backups by previewing resources before running the restore, creating a Restore custom resource (CR), and configuring restore hooks to run commands in restored pods. This helps you to recover your application data and configuration while controlling the restore process.
1.9.1.1. Previewing resources before running backup and restore Copy linkLink copied to clipboard!
Preview the backup and restore resources in advance by doing a dry run of the backup and restore operations. This helps you to verify which resources will be included before committing to a full backup or restore.
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
$ velero backup create <backup-name> --snapshot-volumes falseSpecify the value of
--snapshot-volumesparameter asfalse.To know more details about the backup resources, run the following command:
$ velero describe backup <backup_name> --detailsReplace
<backup_name>with the name of the backup.To preview the resources included in the restore before running the actual restore, run the following command:
$ velero restore create --from-backup <backup_name>Replace
<backup_name>with the name of the backup.ImportantThe
velero restore createcommand creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
$ velero describe restore <restore_name> --detailsReplace
<restore_name>with the name of the restore.
1.9.1.2. Creating a Restore CR Copy linkLink copied to clipboard!
Restore a Backup custom resource (CR) by creating a Restore CR.
When you restore a stateful application that uses the azurefile-csi storage class, the restore operation remains in the Finalizing phase.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplicationCR must be in aReadystate. -
You must have a Velero
BackupCR. - The persistent volume (PV) capacity must match the requested size at backup time. Adjust the requested size if needed.
Procedure
Create a
RestoreCR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: backupName: <backup> includedResources: [] excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io restorePVs: truewhere:
<backup>-
Specifies the name of the
BackupCR. includedResources-
Optional: Specifies an array of resources to include in the restore process. Resources might be shortcuts (for example,
poforpods) or fully-qualified. If unspecified, all resources are included. restorePVs: true-
Optional: The
restorePVsparameter can be set tofalseto turn off restore ofPersistentVolumesfromVolumeSnapshotof Container Storage Interface (CSI) snapshots or from native snapshots whenVolumeSnapshotLocationis configured.
Verify that the status of the
RestoreCR isCompletedby entering the following command:$ oc get restores.velero.io -n openshift-adp <restore> -o jsonpath='{.status.phase}'Verify that the backup resources have been restored by entering the following command:
$ oc get all -n <namespace>where:
<namespace>- Specifies the namespace that you backed up.
If you restore
DeploymentConfigwith volumes or if you use post-restore hooks, run thedc-post-restore.shcleanup script by entering the following command:$ bash dc-restic-post-restore.sh -> dc-post-restore.shNoteDuring the restore process, the OADP Velero plug-ins scale down the
DeploymentConfigobjects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restoredDeploymentConfigpods immediately on restore and to allow the restore and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales anyDeploymentConfigobjects back up to the appropriate number of replicas.#!/bin/bash set -e # if sha256sum exists, use it to check the integrity of the file if command -v sha256sum >/dev/null 2>&1; then CHECKSUM_CMD="sha256sum" else CHECKSUM_CMD="shasum -a 256" fi label_name () { if [ "${#1}" -le "63" ]; then echo $1 return fi sha=$(echo -n $1|$CHECKSUM_CMD) echo "${1:0:57}${sha:0:6}" } if [[ $# -ne 1 ]]; then echo "usage: ${BASH_SOURCE} restore-name" exit 1 fi echo "restore: $1" label=$(label_name $1) echo "label: $label" echo Deleting disconnected restore pods oc delete pods --all-namespaces -l oadp.openshift.io/disconnected-from-dc=$label for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}') do IFS=',' read -ra dc_arr <<< "$dc" if [ ${#dc_arr[0]} -gt 0 ]; then echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]} cat <<EOF | oc patch dc -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin spec: replicas: ${dc_arr[2]} paused: ${dc_arr[3]} EOF fi done
1.9.1.3. Creating restore hooks Copy linkLink copied to clipboard!
Create restore hooks to run commands in a container in a pod by editing the Restore custom resource (CR).
You can create two types of restore hooks:
An
inithook adds an init container to a pod to perform setup tasks before the application container starts.If you restore a Restic backup, the
restic-waitinit container is added before the restore hook init container.-
An
exechook runs commands or scripts in a container of a restored pod.
Procedure
Add a hook to the
spec.hooksblock of theRestoreCR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> excludedNamespaces: - <namespace> includedResources: - pods excludedResources: [] labelSelector: matchLabels: app: velero component: server postHooks: - init: initContainers: - name: restore-hook-init image: alpine:latest volumeMounts: - mountPath: /restores/pvc1-vm name: pvc1-vm command: - /bin/ash - -c timeout: - exec: container: <container> command: - /bin/bash - -c - "psql < /backup/backup.sql" waitTimeout: 5m execTimeout: 1m onError: Continuewhere:
<namespace>- Optional: Specifies an array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
pods- Currently, pods are the only supported resource that hooks can apply to.
labelSelector- Optional: This hook only applies to objects matching the label selector.
timeout-
Optional: Specifies the maximum length of time Velero waits for
initContainersto complete. <container>- Optional: Specifies the container in which the command runs. If the container is not specified, the command runs in the first container in the pod.
/bin/bash- Specifies the entrypoint for the init container being added.
waitTimeout: 5m- Optional: Specifies how long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.
execTimeout: 1m-
Optional: Specifies how long to wait for the commands to run. The default is
30s. onError: ContinueSpecifies the error handling behavior. Allowed values are
FailandContinue:-
Continue: Only command failures are logged. -
Fail: No more restore hooks run in any container in any pod. The status of theRestoreCR will bePartiallyFailed.
-
During a File System Backup (FSB) restore operation, a Deployment resource referencing an ImageStream is not restored properly. The restored pod that runs the FSB, and the postHook is terminated prematurely.
This happens because, during the restore operation, OpenShift controller updates the spec.template.spec.containers[0].image field in the Deployment resource with an updated ImageStreamTag hash. The update triggers the rollout of a new pod, terminating the pod on which velero runs the FSB and the post restore hook. For more information about image stream trigger, see "Triggering updates on image stream changes".
The workaround for this behavior is a two-step restore process:
First, perform a restore excluding the
Deploymentresources, for example:$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --exclude-resources=deployment.appsAfter the first restore is successful, perform a second restore by including these resources, for example:
$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --include-resources=deployment.apps