Backup and restore
Backing up and restoring your Red Hat OpenShift Service on AWS cluster
Abstract
Chapter 1. OADP Application backup and restore Copy linkLink copied to clipboard!
1.1. Introduction to OpenShift API for Data Protection Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) product safeguards customer applications on Red Hat OpenShift Service on AWS. It offers comprehensive disaster recovery protection, covering Red Hat OpenShift Service on AWS applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs).
1.1.1. OpenShift API for Data Protection APIs Copy linkLink copied to clipboard!
OADP provides APIs that enable multiple approaches to customizing backups and preventing the inclusion of unnecessary or inappropriate resources.
OADP provides the following APIs:
1.1.1.1. Support for OpenShift API for Data Protection Copy linkLink copied to clipboard!
| Version | OCP version | General availability | Full support ends | Maintenance ends | Extended Update Support (EUS) | Extended Update Support Term 2 (EUS Term 2) |
| 1.5 |
| 17 June 2025 | Release of 1.6 | Release of 1.7 | EUS must be on OCP 4.20 | EUS Term 2 must be on OCP 4.20 |
| 1.4 |
| 10 Jul 2024 | Release of 1.5 | Release of 1.6 | 27 Jun 2026 EUS must be on OCP 4.16 | 27 Jun 2027 EUS Term 2 must be on OCP 4.16 |
| 1.3 |
| 29 Nov 2023 | 10 Jul 2024 | Release of 1.5 | 31 Oct 2025 EUS must be on OCP 4.14 | 31 Oct 2026 EUS Term 2 must be on OCP 4.14 |
1.1.1.1.1. Unsupported versions of the OADP Operator Copy linkLink copied to clipboard!
| Version | General availability | Full support ended | Maintenance ended |
| 1.2 | 14 Jun 2023 | 29 Nov 2023 | 10 Jul 2024 |
| 1.1 | 01 Sep 2022 | 14 Jun 2023 | 29 Nov 2023 |
| 1.0 | 09 Feb 2022 | 01 Sep 2022 | 14 Jun 2023 |
For more details about EUS, see Extended Update Support.
For more details about EUS Term 2, see Extended Update Support Term 2.
1.2. OADP release notes Copy linkLink copied to clipboard!
1.2.1. OADP 1.5 release notes Copy linkLink copied to clipboard!
The release notes for OpenShift API for Data Protection (OADP) describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.
For additional information about OADP, see OpenShift API for Data Protection (OADP) FAQs
1.2.1.1. OADP 1.5.3 release notes Copy linkLink copied to clipboard!
OpenShift API for Data Protection (OADP) 1.5.3 is a Container Grade Only (CGO) release, which is released to refresh the health grades of the containers. No code was changed in the product itself compared to that of OADP 1.5.2.
1.2.1.2. OADP 1.5.2 release notes Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) 1.5.2 release notes lists resolved issues.
1.2.1.2.1. Resolved issues Copy linkLink copied to clipboard!
Self-signed certificate for internal image backup should not break other BSLs
Before this update, OADP would only process the first custom CA certificate found among all backup storage locations (BSLs) and apply it globally. This behavior prevented multiple BSLs with different CA certificates from working correctly. Additionally, system-trusted certificates were not included, causing failures when connecting to standard services. With this update, OADP now:
- Concatenates all unique CA certificates from AWS BSLs into a single bundle.
- Includes system-trusted certificates automatically.
- Enables multiple BSLs with different custom CA certificates to operate simultaneously.
- Only processes CA certificates when image backup is enabled (default behavior).
This enhancement improves compatibility for environments using multiple storage providers with different certificate requirements, particularly when backing up internal images to AWS S3-compatible storage with self-signed certificates.
1.2.1.3. OADP 1.5.1 release notes Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) 1.5.1 release notes lists new features, resolved issues, known issues, and deprecated features.
1.2.1.3.1. New features Copy linkLink copied to clipboard!
CloudStorage API is fully supported
The CloudStorage API feature, available as a Technology Preview before this update, is fully supported from OADP 1.5.1. The CloudStorage API automates the creation of a bucket for object storage.
New DataProtectionTest custom resource is available
The DataProtectionTest (DPT) is a custom resource (CR) that provides a framework to validate your OADP configuration. The DPT CR checks and reports information for the following parameters:
- The upload performance of the backups to the object storage.
- The Container Storage Interface (CSI) snapshot readiness for persistent volume claims.
- The storage bucket configuration, such as encryption and versioning.
Using this information in the DPT CR, you can ensure that your data protection environment is properly configured and performing according to the set configuration.
Note that you must configure STORAGE_ACCOUNT_ID when using DPT with OADP on Azure.
New node agent load affinity configurations are available
-
Node agent load affinity: You can schedule the node agent pods on specific nodes by using the
spec.podConfig.nodeSelectorobject of theDataProtectionApplication(DPA) custom resource (CR). You can add more restrictions on the node agent pods scheduling by using thenodeagent.loadAffinityobject in the DPA spec. Repository maintenance job affinity configurations: You can use the repository maintenance job affinity configurations in the
DataProtectionApplication(DPA) custom resource (CR) only if you use Kopia as the backup repository.You have the option to configure the load affinity at the global level affecting all repositories, or for each repository. You can also use a combination of global and per-repository configuration.
-
Velero load affinity: You can use the
podConfig.nodeSelectorobject to assign the Velero pod to specific nodes. You can also configure thevelero.loadAffinityobject for pod-level affinity and anti-affinity.
Node agent load concurrency is available
With this update, users can control the maximum number of node agent operations that can run simultaneously on each node within their cluster. It also enables better resource management, optimizing backup and restore workflows for improved performance and a more streamlined experience.
1.2.1.3.2. Resolved issues Copy linkLink copied to clipboard!
DataProtectionApplicationSpec overflowed annotation limit, causing potential misconfiguration in deployments
Before this update, the DataProtectionApplicationSpec used deprecated PodAnnotations, which led to an annotation limit overflow. This caused potential misconfigurations in deployments. In this release, we have added PodConfig for annotations in pods deployed by the Operator, ensuring consistent annotations and improved manageability for end users. As a result, deployments should now be more reliable and easier to manage.
Root file system for OADP controller manager is now read-only
Before this update, the manager container of the openshift-adp-controller-manager-* pod was configured to run with a writable root file system. As a consequence, this could allow for tampering with the container’s file system or the writing of foreign executables. With this release, the container’s security context has been updated to set the root file system to read-only while ensuring necessary functions that require write access, such as the Kopia cache, continue to operate correctly. As a result, the container is hardened against potential threats.
nonAdmin.enable: false in multiple DPAs no longer causes reconcile issues
Before this update, when a user attempted to create a second non-admin DataProtectionApplication (DPA) on a cluster where one already existed, the new DPA failed to reconcile. With this release, the restriction on Non-Admin Controller installation to one per cluster has been removed. As a result, users can install multiple Non-Admin Controllers across the cluster without encountering errors.
OADP supports self-signed certificates
Before this update, using a self-signed certificate for backup images with a storage provider such as Minio resulted in an x509: certificate signed by unknown authority error during the backup process. With this release, certificate validation has been updated to support self-signed certificates in OADP, ensuring successful backups.
velero describe includes defaultVolumesToFsBackup
Before this update, the velero describe output command omitted the defaultVolumesToFsBackup flag. This affected the visibility of backup configuration details for users. With this release, the velero describe output includes the defaultVolumesToFsBackup flag information, improving the visibility of backup settings.
DPT CR no longer fail when s3Url is secured
Before this update, DataProtectionTest (DPT) failed to run when s3Url was secured due to an unverified certificate because the DPT CR lacked the ability to skip or add the caCert in the spec field. As a consequence, data upload failure occurred due to an unverified certificate. With this release, DPT CR has been updated to accept and skip CA cert in spec field, resolving SSL verification errors. As a result, DPT no longer fails when using secured s3Url.
Adding a backupLocation to DPA with an existing backupLocation name is not rejected
Before this update, adding a second backupLocation with the same name in DataProtectionApplication (DPA) caused OADP to enter an invalid state, leading to Backup and Restore failures due to Velero’s inability to read Secret credentials. As a consequence, Backup and Restore operations failed. With this release, the duplicate backupLocation names in DPA are no longer allowed, preventing Backup and Restore failures. As a result, duplicate backupLocation names are rejected, ensuring seamless data protection.
1.2.1.3.3. Known issues Copy linkLink copied to clipboard!
The restore fails for backups created on OpenStack using the Cinder CSI driver
When you start a restore operation for a backup that was created on an OpenStack platform using the Cinder Container Storage Interface (CSI) driver, the initial backup only succeeds after the source application is manually scaled down. The restore job fails, preventing you from successfully recovering your application’s data and state from the backup. No known workaround exists.
Datamover pods scheduled on unexpected nodes during backup if the nodeAgent.loadAffinity parameter has many elements
Due to an issue in Velero 1.14 and later, the OADP node-agent only processes the first nodeSelector element within the loadAffinity array. As a consequence, if you define multiple nodeSelector objects, all objects except the first are ignored, potentially causing datamover pods to be scheduled on unexpected nodes during a backup.
To work around this problem, consolidate all required matchExpressions from multiple nodeSelector objects into the first nodeSelector object. As a result, all node affinity rules are correctly applied, ensuring datamover pods are scheduled to the appropriate nodes.
OADP Backup fails when using CA certificates with aliased command
The CA certificate is not stored as a file on the running Velero container. As a consequence, the user experience degraded due to missing caCert in Velero container, requiring manual setup and downloads. To work around this problem, manually add cert to the Velero deployment. For instructions, see Using cacert with velero command aliased via velero deployment.
The nodeSelector spec is not supported for the Data Mover restore action
When a Data Protection Application (DPA) is created with the nodeSelector field set in the nodeAgent parameter, Data Mover restore partially fails instead of completing the restore operation. No known workaround exists.
Image streams backups are partially failing when the DPA is configured with caCert
An unverified certificate in the S3 connection during backups with caCert in DataProtectionApplication (DPA) causes the ocp-django application’s backup to partially fail and result in data loss. No known workaround exists.
Kopia does not delete cache on worker node
When the ephemeral-storage parameter is configured and running file system restore, the cache is not automatically deleted from the worker node. As a consequence, the /var partition overflows during backup restore, causing increased storage usage and potential resource exhaustion. To work around this problem, restart the node agent pod, which clears the cache. As a result, cache is deleted.
Google Cloud VSL backups fail with Workload Identity because of invalid project configuration
When performing a volumeSnapshotLocation (VSL) backup on Google Cloud Workload Identity, the Velero Google Cloud plugin creates an invalid API request if the Google Cloud project is also specified in the snapshotLocations configuration of DataProtectionApplication (DPA). As a consequence, the Google Cloud API returns a RESOURCE_PROJECT_INVALID error, and the backup job finishes with a PartiallyFailed status. No known workaround exists.
VSL backups fail for CloudStorage API on AWS with STS
The volumeSnapshotLocation (VSL) backup fails because of missing the AZURE_RESOURCE_GROUP parameter in the credentials file, even if AZURE_RESOURCE_GROUP is already mentioned in the DataProtectionApplication (DPA) config for VSL. No known workaround exists.
Backups of applications with ImageStreams fail on Azure with STS
When backing up applications that include image stream resources on an Azure cluster using STS, the OADP plugin incorrectly attempts to locate a secret-based credential for the container registry. As a consequence, the required secret is not found in the STS environment, causing the ImageStream custom backup action to fail. This results in the overall backup status marked as PartiallyFailed. No known workaround exists.
DPA reconciliation fails for CloudStorageRef configuration
When a user creates a bucket and uses the backupLocations.bucket.cloudStorageRef configuration, bucket credentials are not present in the DataProtectionApplication (DPA) custom resource (CR). As a result, the DPA reconciliation fails even if bucket credentials are present in the CloudStorage CR. To work around this problem, add the same credentials to the backupLocations section of the DPA CR.
1.2.1.3.4. Deprecated features Copy linkLink copied to clipboard!
The configuration.restic specification field has been deprecated
With OADP 1.5.0, the configuration.restic specification field has been deprecated. Use the nodeAgent section with the uploaderType field for selecting kopia or restic as a uploaderType. Note that Restic is deprecated in OADP 1.5.0.
1.2.1.4. OADP 1.5.0 release notes Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) 1.5.0 release notes lists resolved issues and known issues.
1.2.1.4.1. New features Copy linkLink copied to clipboard!
OADP 1.5.0 introduces a new Self-Service feature
OADP 1.5.0 introduces a new feature named OADP Self-Service, enabling namespace admin users to back up and restore applications on the Red Hat OpenShift Service on AWS. In the earlier versions of OADP, you needed the cluster-admin role to perform OADP operations such as backing up and restoring an application, creating a backup storage location, and so on.
From OADP 1.5.0 onward, you do not need the cluster-admin role to perform the backup and restore operations. You can use OADP with the namespace admin role. The namespace admin role has administrator access only to the namespace the user is assigned to. You can use the Self-Service feature only after the cluster administrator installs the OADP Operator and provides the necessary permissions.
Collecting logs with the must-gather tool has been improved with a Markdown summary
You can collect logs, and information about OpenShift API for Data Protection (OADP) custom resources by using the must-gather tool. The must-gather data must be attached to all customer cases. This tool generates a Markdown output file with the collected information, which is located in the must-gather logs clusters directory.
dataMoverPrepareTimeout and resourceTimeout parameters are now added to nodeAgent within the DPA
The nodeAgent field in Data Protection Application (DPA) now includes the following parameters:
-
dataMoverPrepareTimeout: Defines the duration theDataUploadorDataDownloadprocess will wait. The default value is 30 minutes. -
resourceTimeout: Sets the timeout for resource processes not addressed by other specific timeout parameters. The default value is 10 minutes.
Use the spec.configuration.nodeAgent parameter in DPA for configuring nodeAgent daemon set
Velero no longer uses the node-agent-config config map for configuring the nodeAgent daemon set. With this update, you must use the new spec.configuration.nodeAgent parameter in a Data Protection Application (DPA) for configuring the nodeAgent daemon set.
Configuring DPA with the backup repository configuration config map is now possible
With Velero 1.15 and later, you can now configure the total size of a cache per repository. This prevents pods from being removed due to running out of ephemeral storage. See the following new parameters added to the NodeAgentConfig field in DPA:
-
cacheLimitMB: Sets the local data cache size limit in megabytes. fullMaintenanceInterval: The default value is 24 hours. Controls the removal rate of deleted Velero backups from the Kopia repository using the following override options:-
normalGC: 24 hours -
fastGC: 12 hours -
eagerGC: 6 hours
-
Enhancing the node-agent security
With this update, the following changes are added:
-
A new
configurationoption is now added to thevelerofield in DPA. The default value for the
disableFsBackupparameter isfalseornon-existing. With this update, the following options are added to theSecurityContextfield:-
Privileged: true -
AllowPrivilegeEscalation: true
-
If you set the
disableFsBackupparameter totrue, it removes the following mounts from the node-agent:-
host-pods -
host-plugins
-
- Modifies that the node-agent is always run as a non-root user.
- Changes the root file system to read only.
Updates the following mount points with the write access:
-
/home/velero -
tmp/credentials
-
-
Uses the
SeccompProfileTypeRuntimeDefaultoption for theSeccompProfileparameter.
Adds DPA support for parallel item backup
By default, only one thread processes an item block. Velero 1.16 supports a parallel item backup, where multiple items within a backup can be processed in parallel.
You can use the optional Velero server parameter --item-block-worker-count to run additional worker threads to process items in parallel. To enable this in OADP, set the dpa.Spec.Configuration.Velero.ItemBlockWorkerCount parameter to an integer value greater than zero.
Running multiple full backups in parallel is not yet supported.
OADP logs are now available in the JSON format
With the of release OADP 1.5.0, the logs are now available in the JSON format. It helps to have pre-parsed data in their Elastic logs management system.
The oc get dpa command now displays RECONCILED status
With this release, the oc get dpa command now displays RECONCILED status instead of displaying only NAME and AGE to improve user experience. For example:
oc get dpa -n openshift-adp NAME RECONCILED AGE velero-sample True 2m51s
$ oc get dpa -n openshift-adp
NAME RECONCILED AGE
velero-sample True 2m51s
1.2.1.4.2. Resolved issues Copy linkLink copied to clipboard!
Containers now use FallbackToLogsOnError for terminationMessagePolicy
With this release, the terminationMessagePolicy field can now set the FallbackToLogsOnError value for the OpenShift API for Data Protection (OADP) Operator containers such as operator-manager, velero, node-agent, and non-admin-controller.
This change ensures that if a container exits with an error and the termination message file is empty, OpenShift uses the last portion of the container logs output as the termination message.
Namespace admin can now access the application after restore
Previously, the namespace admin could not execute an application after the restore operation with the following errors:
-
exec operation is not allowed because the pod’s security context exceeds your permissions -
unable to validate against any security context constraint -
not usable by user or serviceaccount, provider restricted-v2
With this update, this issue is now resolved and the namespace admin can access the application successfully after the restore.
Specifying status restoration at the individual resource instance level using the annotation is now possible
Previously, status restoration was only configured at the resource type using the restoreStatus field in the Restore custom resource (CR).
With this release, you can now specify the status restoration at the individual resource instance level using the following annotation:
metadata:
annotations:
velero.io/restore-status: "true"
metadata:
annotations:
velero.io/restore-status: "true"
Restore is now successful with excludedClusterScopedResources
Previously, on performing the backup of an application with the excludedClusterScopedResources field set to storageclasses, Namespace parameter, the backup was successful but the restore partially failed. With this update, the restore is successful.
Backup is completed even if it gets restarted during the waitingForPluginOperations phase
Previously, a backup was marked as failed with the following error message:
failureReason: found a backup with status "InProgress" during the server starting, mark it as "Failed"
failureReason: found a backup with status "InProgress" during the server starting,
mark it as "Failed"
With this update, the backup is completed if it gets restarted during the waitingForPluginOperations phase.
Error messages are now more informative when the` disableFsbackup` parameter is set to true in DPA
Previously, when the spec.configuration.velero.disableFsBackup field from a Data Protection Application (DPA) was set to true, the backup partially failed with an error, which was not informative.
This update makes error messages more useful for troubleshooting issues. For example, error messages indicating that disableFsBackup: true is the issue in a DPA or not having access to a DPA if it is for non-administrator users.
Handles AWS STS credentials in the parseAWSSecret
Previously, AWS credentials using STS authentication were not properly validated.
With this update, the parseAWSSecret function detects STS-specific fields, and updates the ensureSecretDataExists function to handle STS profiles correctly.
The repositoryMaintenance job affinity config is available to configure
Previously, the new configurations for repository maintenance job pod affinity was missing from a DPA specification.
With this update, the repositoryMaintenance job affinity config is now available to map a BackupRepository identifier to its configuration.
The ValidationErrors field fades away once the CR specification is correct
Previously, when a schedule CR was created with a wrong spec.schedule value and the same was later patched with a correct value, the ValidationErrors field still existed. Consequently, the ValidationErrors field was displaying incorrect information even though the spec was correct.
With this update, the ValidationErrors field fades away once the CR specification is correct.
The volumeSnapshotContents custom resources are restored when the includedNamesapces field is used in restoreSpec
Previously, when a restore operation was triggered with the includedNamespace field in a restore specification, restore operation was completed successfully but no volumeSnapshotContents custom resources (CR) were created and the PVCs were in a Pending status.
With this update, volumeSnapshotContents CR are restored even when the includedNamesapces field is used in restoreSpec. As a result, an application pod is in a Running state after restore.
OADP operator successfully creates bucket on top of AWS
Previously, the container was configured with the readOnlyRootFilesystem: true setting for security, but the code attempted to create temporary files in the /tmp directory using the os.CreateTemp() function. Consequently, while using the AWS STS authentication with the Cloud Credential Operator (CCO) flow, OADP failed to create temporary files that were required for AWS credential handling with the following error:
ERROR unable to determine if bucket exists. {"error": "open /tmp/aws-shared-credentials1211864681: read-only file system"}
ERROR unable to determine if bucket exists. {"error": "open /tmp/aws-shared-credentials1211864681: read-only file system"}
With this update, the following changes are added to address this issue:
-
A new
emptyDirvolume namedtmp-dirto the controller pod specification. -
A volume mount to the container, which mounts this volume to the
/tmpdirectory. -
For security best practices, the
readOnlyRootFilesystem: trueis maintained. -
Replaced the deprecated
ioutil.TempFile()function with the recommendedos.CreateTemp()function. -
Removed the unnecessary
io/ioutilimport, which is no longer needed.
For a complete list of all issues resolved in this release, see the list of OADP 1.5.0 resolved issues in Jira.
1.2.1.4.3. Known issues Copy linkLink copied to clipboard!
Kopia does not delete all the artifacts after backup expiration
Even after deleting a backup, Kopia does not delete the volume artifacts from the ${bucket_name}/kopia/$openshift-adp on the S3 location after the backup expired. Information related to the expired and removed data files remains in the metadata. To ensure that OpenShift API for Data Protection (OADP) functions properly, the data is not deleted, and it exists in the /kopia/ directory, for example:
-
kopia.repository: Main repository format information such as encryption, version, and other details. -
kopia.blobcfg: Configuration for how data blobs are named. -
kopia.maintenance: Tracks maintenance owner, schedule, and last successful build. -
log: Log blobs.
For a complete list of all known issues in this release, see the list of OADP 1.5.0 known issues in Jira.
1.2.1.4.4. Deprecated features Copy linkLink copied to clipboard!
The configuration.restic specification field has been deprecated
With OpenShift API for Data Protection (OADP) 1.5.0, the configuration.restic specification field has been deprecated. Use the nodeAgent section with the uploaderType field for selecting kopia or restic as a uploaderType. Note that, Restic is deprecated in OpenShift API for Data Protection (OADP) 1.5.0.
1.2.1.4.5. Technology Preview Copy linkLink copied to clipboard!
Support for HyperShift hosted OpenShift clusters is available as a Technology Preview
OADP can support and facilitate application migrations within HyperShift hosted OpenShift clusters as a Technology Preview. It ensures a seamless backup and restore operation for applications in hosted clusters.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
1.2.1.5. Upgrading OADP 1.4 to 1.5 Copy linkLink copied to clipboard!
Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OADP 1.1 to 1.3, upgrade first to 1.2, and then to 1.3.
1.2.1.5.1. Changes from OADP 1.4 to 1.5 Copy linkLink copied to clipboard!
The Velero server has been updated from version 1.14 to 1.16.
This changes the following:
- Version Support changes
- OpenShift API for Data Protection implements a streamlined version support policy. Red Hat supports only one version of OpenShift API for Data Protection (OADP) on one OpenShift version to ensure better stability and maintainability. OADP 1.5.0 is only supported on OpenShift 4.19 version.
- OADP Self-Service
OADP 1.5.0 introduces a new feature named OADP Self-Service, enabling namespace admin users to back up and restore applications on the Red Hat OpenShift Service on AWS. In the earlier versions of OADP, you needed the cluster-admin role to perform OADP operations such as backing up and restoring an application, creating a backup storage location, and so on.
From OADP 1.5.0 onward, you do not need the cluster-admin role to perform the backup and restore operations. You can use OADP with the namespace admin role. The namespace admin role has administrator access only to the namespace the user is assigned to. You can use the Self-Service feature only after the cluster administrator installs the OADP Operator and provides the necessary permissions.
backupPVCandrestorePVCconfigurationsA
backupPVCresource is an intermediate persistent volume claim (PVC) to access data during the data movement backup operation. You create areadonlybackup PVC by using thenodeAgent.backupPVCsection of theDataProtectionApplication(DPA) custom resource.A
restorePVCresource is an intermediate PVC that is used to write data during the Data Mover restore operation.You can configure
restorePVCin the DPA by using theignoreDelayBindingfield.
1.2.1.5.2. Backing up the DPA configuration Copy linkLink copied to clipboard!
You must back up your current DataProtectionApplication (DPA) configuration.
Procedure
Save your current DPA configuration by running the following command:
Example command
oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backupCopy to Clipboard Copied! Toggle word wrap Toggle overflow
1.2.1.5.3. Upgrading the OADP Operator Copy linkLink copied to clipboard!
You can upgrade the OpenShift API for Data Protection (OADP) Operator using the following procedure.
Do not install OADP 1.5.0 on a OpenShift 4.18 cluster.
Prerequisites
- You have installed the latest OADP 1.4.6.
- You have backed up your data.
Procedure
Upgrade OpenShift 4.18 to OpenShift 4.19.
NoteOpenShift API for Data Protection (OADP) 1.4 is not supported on OpenShift 4.19.
-
Change your subscription channel for the OADP Operator from
stable-1.4tostable. - Wait for the Operator and containers to update and restart.
1.2.1.5.4. Converting DPA to the new version for OADP 1.5.0 Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) 1.4 is not supported on OpenShift 4.19. You can convert Data Protection Application (DPA) to the new OADP 1.5 version by using the new spec.configuration.nodeAgent field and its sub-fields.
Procedure
To configure
nodeAgentdaemon set, use thespec.configuration.nodeAgentparameter in DPA. See the following example:Example
DataProtectionApplicationconfigurationCopy to Clipboard Copied! Toggle word wrap Toggle overflow To configure
nodeAgentdaemon set by using theConfigMapresource namednode-agent-config, see the following example configuration:Example config map
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.2.1.5.5. Verifying the upgrade Copy linkLink copied to clipboard!
You can verify the OpenShift API for Data Protection (OADP) upgrade by using the following procedure.
Procedure
Verify that the
DataProtectionApplication(DPA) has been reconciled successfully:oc get dpa dpa-sample -n openshift-adp
$ oc get dpa dpa-sample -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME RECONCILED AGE dpa-sample True 2m51s
NAME RECONCILED AGE dpa-sample True 2m51sCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
RECONCILEDcolumn must beTrue.Verify that the installation finished by viewing the OADP resources by running the following command:
oc get all -n openshift-adp
$ oc get all -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
node-agentpods are created only while usingresticorkopiainDataProtectionApplication(DPA). In OADP 1.4.0 and OADP 1.3.0 version, thenode-agentpods are labeled asrestic.Verify the backup storage location and confirm that the
PHASEisAvailableby running the following command:oc get backupstoragelocations.velero.io -n openshift-adp
$ oc get backupstoragelocations.velero.io -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow
1.3. OADP performance Copy linkLink copied to clipboard!
1.3.1. OADP recommended network settings Copy linkLink copied to clipboard!
For a supported experience with OpenShift API for Data Protection (OADP), you should have a stable and resilient network across OpenShift nodes, S3 storage, and in supported cloud environments that meet OpenShift network requirement recommendations.
To ensure successful backup and restore operations for deployments with remote S3 buckets located off-cluster with suboptimal data paths, it is recommended that your network settings meet the following minimum requirements in such less optimal conditions:
- Bandwidth (network upload speed to object storage): Greater than 2 Mbps for small backups and 10-100 Mbps depending on the data volume for larger backups.
- Packet loss: 1%
- Packet corruption: 1%
- Latency: 100ms
Ensure that your Red Hat OpenShift Service on AWS network performs optimally and meets Red Hat OpenShift Service on AWS network requirements.
Although Red Hat provides supports for standard backup and restore failures, it does not provide support for failures caused by network settings that do not meet the recommended thresholds.
1.4. OADP features and plugins Copy linkLink copied to clipboard!
OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications.
The default plugins enable Velero to integrate with certain cloud providers and to back up and restore Red Hat OpenShift Service on AWS resources.
1.4.1. OADP features Copy linkLink copied to clipboard!
OpenShift API for Data Protection (OADP) supports the following features:
- Backup
You can use OADP to back up all applications on the OpenShift Platform, or you can filter the resources by type, namespace, or label.
OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Restore
You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the objects by namespace, PV, or label.
NoteYou must exclude Operators from the backup of an application for backup and restore to succeed.
- Schedule
- You can schedule backups at specified intervals.
- Hooks
-
You can use hooks to run commands in a container on a pod, for example,
fsfreezeto freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.
1.4.2. OADP plugins Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) provides default Velero plugins that are integrated with storage providers to support backup and snapshot operations. You can create custom plugins based on the Velero plugins.
OADP also provides plugins for Red Hat OpenShift Service on AWS resource backups, OpenShift Virtualization resource backups, and Container Storage Interface (CSI) snapshots.
| OADP plugin | Function | Storage location |
|---|---|---|
|
| Backs up and restores Kubernetes objects. | AWS S3 |
| Backs up and restores volumes with snapshots. | AWS EBS | |
|
| Backs up and restores Red Hat OpenShift Service on AWS resources. [1] | Object store |
|
| Backs up and restores OpenShift Virtualization resources. [2] | Object store |
|
| Backs up and restores volumes with CSI snapshots. [3] | Cloud storage that supports CSI snapshots |
|
| Backs up and restores HyperShift hosted cluster resources. [4] | Object store |
- Mandatory.
- Virtual machine disks are backed up with CSI snapshots or Restic.
The
csiplugin uses the Kubernetes CSI snapshot API.-
OADP 1.1 or later uses
snapshot.storage.k8s.io/v1 -
OADP 1.0 uses
snapshot.storage.k8s.io/v1beta1
-
OADP 1.1 or later uses
-
Do not add the
hypershiftplugin in theDataProtectionApplicationcustom resource if the cluster is not a HyperShift hosted cluster.
1.4.3. About OADP Velero plugins Copy linkLink copied to clipboard!
You can configure two types of plugins when you install Velero:
- Default cloud provider plugins
- Custom plugins
Both types of plugin are optional, but most users configure at least one cloud provider plugin.
1.4.3.1. Default Velero cloud provider plugins Copy linkLink copied to clipboard!
You can install any of the following default Velero cloud provider plugins when you configure the oadp_v1alpha1_dpa.yaml file during deployment:
-
aws(Amazon Web Services) -
openshift(OpenShift Velero plugin) -
csi(Container Storage Interface) -
kubevirt(KubeVirt)
You specify the desired default plugins in the oadp_v1alpha1_dpa.yaml file during deployment.
Example file
The following .yaml file installs the openshift, aws, azure, and gcp plugins:
1.4.3.2. Custom Velero plugins Copy linkLink copied to clipboard!
You can install a custom Velero plugin by specifying the plugin image and name when you configure the oadp_v1alpha1_dpa.yaml file during deployment.
You specify the desired custom plugins in the oadp_v1alpha1_dpa.yaml file during deployment.
Example file
The following .yaml file installs the default openshift, azure, and gcp plugins and a custom plugin that has the name custom-plugin-example and the image quay.io/example-repo/custom-velero-plugin:
1.4.4. OADP and FIPS Copy linkLink copied to clipboard!
Federal Information Processing Standards (FIPS) are a set of computer security standards developed by the United States federal government in line with the Federal Information Security Management Act (FISMA).
OpenShift API for Data Protection (OADP) has been tested and works on FIPS-enabled Red Hat OpenShift Service on AWS clusters.
1.4.5. Avoiding the Velero plugin panic error Copy linkLink copied to clipboard!
A missing secret can cause a panic error for the Velero plugin during image stream backups.
When the backup and the Backup Storage Location (BSL) are managed outside the scope of the Data Protection Application (DPA), the OADP controller does not create the relevant oadp-<bsl_name>-<bsl_provider>-registry-secret parameter.
During the backup operation, the OpenShift Velero plugin panics on the imagestream backup, with the following panic error:
024-02-27T10:46:50.028951744Z time="2024-02-27T10:46:50Z" level=error msg="Error backing up item" backup=openshift-adp/<backup name> error="error executing custom action (groupResource=imagestreams.image.openshift.io, namespace=<BSL Name>, name=postgres): rpc error: code = Aborted desc = plugin panicked: runtime error: index out of range with length 1, stack trace: goroutine 94…
024-02-27T10:46:50.028951744Z time="2024-02-27T10:46:50Z" level=error msg="Error backing up item"
backup=openshift-adp/<backup name> error="error executing custom action (groupResource=imagestreams.image.openshift.io,
namespace=<BSL Name>, name=postgres): rpc error: code = Aborted desc = plugin panicked:
runtime error: index out of range with length 1, stack trace: goroutine 94…
Use the following workaround to avoid the Velero plugin panic error.
Procedure
Label the custom BSL with the relevant label by using the following command:
oc label backupstoragelocations.velero.io <bsl_name> app.kubernetes.io/component=bsl
$ oc label backupstoragelocations.velero.io <bsl_name> app.kubernetes.io/component=bslCopy to Clipboard Copied! Toggle word wrap Toggle overflow After the BSL is labeled, wait until the DPA reconciles.
NoteYou can force the reconciliation by making any minor change to the DPA itself.
Verification
After the DPA is reconciled, confirm that the parameter has been created and that the correct registry data has been populated into it by entering the following command:
oc -n openshift-adp get secret/oadp-<bsl_name>-<bsl_provider>-registry-secret -o json | jq -r '.data'
$ oc -n openshift-adp get secret/oadp-<bsl_name>-<bsl_provider>-registry-secret -o json | jq -r '.data'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.4.6. Workaround for OpenShift ADP Controller segmentation fault Copy linkLink copied to clipboard!
If you configure a Data Protection Application (DPA) with both cloudstorage and restic enabled, the openshift-adp-controller-manager pod crashes and restarts indefinitely until the pod fails with a crash loop segmentation fault.
Define either velero or cloudstorage when you configure a DPA. Otherwise, the openshift-adp-controller-manager pod fails with a crash loop segmentation fault due to the following settings:
-
If you define both
veleroandcloudstorage, theopenshift-adp-controller-managerfails. -
If you do not define both
veleroandcloudstorage, theopenshift-adp-controller-managerfails.
For more information about this issue, see OADP-1054.
1.5. OADP use cases Copy linkLink copied to clipboard!
1.5.1. Backing up workloads on OADP with Red Hat OpenShift Service on AWS Copy linkLink copied to clipboard!
To back up and restore workloads on ROSA, you can use OADP. You can create a backup of a workload, restore it from the backup, and verify the restoration. You can also clean up the OADP Operator, backup storage, and AWS resources when they are no longer needed.
1.5.1.1. Performing a backup with OADP and Red Hat OpenShift Service on AWS Copy linkLink copied to clipboard!
The following example hello-world application has no persistent volumes (PVs) attached. Perform a backup by using OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS.
Either Data Protection Application (DPA) configuration will work.
Procedure
Create a workload to back up by running the following commands:
oc create namespace hello-world
$ oc create namespace hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc new-app -n hello-world --image=docker.io/openshift/hello-openshift
$ oc new-app -n hello-world --image=docker.io/openshift/hello-openshiftCopy to Clipboard Copied! Toggle word wrap Toggle overflow Expose the route by running the following command:
oc expose service/hello-openshift -n hello-world
$ oc expose service/hello-openshift -n hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the application is working by running the following command:
curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`Copy to Clipboard Copied! Toggle word wrap Toggle overflow You should see an output similar to the following example:
Hello OpenShift!
Hello OpenShift!Copy to Clipboard Copied! Toggle word wrap Toggle overflow Back up the workload by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait until the backup is complete, and then run the following command:
watch "oc -n openshift-adp get backup hello-world -o json | jq .status"
$ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"Copy to Clipboard Copied! Toggle word wrap Toggle overflow You should see an output similar to the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the demo workload by running the following command:
oc delete ns hello-world
$ oc delete ns hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow Restore the workload from the backup by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Wait for the Restore to finish by running the following command:
watch "oc -n openshift-adp get restore hello-world -o json | jq .status"
$ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"Copy to Clipboard Copied! Toggle word wrap Toggle overflow You should see an output similar to the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check that the workload is restored by running the following command:
oc -n hello-world get pods
$ oc -n hello-world get podsCopy to Clipboard Copied! Toggle word wrap Toggle overflow You should see an output similar to the following example:
NAME READY STATUS RESTARTS AGE hello-openshift-9f885f7c6-kdjpj 1/1 Running 0 90s
NAME READY STATUS RESTARTS AGE hello-openshift-9f885f7c6-kdjpj 1/1 Running 0 90sCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the JSONPath by running the following command:
curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`$ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`Copy to Clipboard Copied! Toggle word wrap Toggle overflow You should see an output similar to the following example:
Hello OpenShift!
Hello OpenShift!Copy to Clipboard Copied! Toggle word wrap Toggle overflow
For troubleshooting tips, see the troubleshooting documentation.
1.5.1.2. Cleaning up a cluster after a backup with OADP and ROSA STS Copy linkLink copied to clipboard!
If you need to uninstall the OpenShift API for Data Protection (OADP) Operator together with the backups and the S3 bucket from this example, follow these instructions.
Procedure
Delete the workload by running the following command:
oc delete ns hello-world
$ oc delete ns hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the Data Protection Application (DPA) by running the following command:
oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa$ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpaCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the cloud storage by running the following command:
oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp$ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadpCopy to Clipboard Copied! Toggle word wrap Toggle overflow WarningIf this command hangs, you might need to delete the finalizer by running the following command:
oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge$ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=mergeCopy to Clipboard Copied! Toggle word wrap Toggle overflow If the Operator is no longer required, remove it by running the following command:
oc -n openshift-adp delete subscription oadp-operator
$ oc -n openshift-adp delete subscription oadp-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the namespace from the Operator:
oc delete ns openshift-adp
$ oc delete ns openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow If the backup and restore resources are no longer required, remove them from the cluster by running the following command:
oc delete backups.velero.io hello-world
$ oc delete backups.velero.io hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow To delete backup, restore and remote objects in AWS S3 run the following command:
velero backup delete hello-world
$ velero backup delete hello-worldCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you no longer need the Custom Resource Definitions (CRD), remove them from the cluster by running the following command:
for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done$ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the AWS S3 bucket by running the following commands:
aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive$ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursiveCopy to Clipboard Copied! Toggle word wrap Toggle overflow aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp$ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Detach the policy from the role by running the following command:
aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"$ aws iam detach-role-policy --role-name "${ROLE_NAME}" --policy-arn "${POLICY_ARN}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the role by running the following command:
aws iam delete-role --role-name "${ROLE_NAME}"$ aws iam delete-role --role-name "${ROLE_NAME}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.5.2. Backup using OpenShift API for Data Protection and Red Hat OpenShift Data Foundation (ODF) Copy linkLink copied to clipboard!
Following is a use case for using OADP and ODF to back up an application.
1.5.2.1. Backing up an application using OADP and ODF Copy linkLink copied to clipboard!
In this use case, you back up an application by using OADP and store the backup in an object storage provided by Red Hat OpenShift Data Foundation (ODF).
- You create an object bucket claim (OBC) to configure the backup storage location. You use ODF to configure an Amazon S3-compatible object storage bucket. ODF provides MultiCloud Object Gateway (NooBaa MCG) and Ceph Object Gateway, also known as RADOS Gateway (RGW), object storage service. In this use case, you use NooBaa MCG as the backup storage location.
-
You use the NooBaa MCG service with OADP by using the
awsprovider plugin. - You configure the Data Protection Application (DPA) with the backup storage location (BSL).
- You create a backup custom resource (CR) and specify the application namespace to back up.
- You create and verify the backup.
Prerequisites
- You installed the OADP Operator.
- You installed the ODF Operator.
- You have an application with a database running in a separate namespace.
Procedure
Create an OBC manifest file to request a NooBaa MCG bucket as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
test-obc- Specifies the name of the object bucket claim.
test-backup-bucket- Specifies the name of the bucket.
Create the OBC by running the following command:
oc create -f <obc_file_name>
$ oc create -f <obc_file_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<obc_file_name>- Specifies the file name of the object bucket claim manifest.
When you create an OBC, ODF creates a
secretand aconfig mapwith the same name as the object bucket claim. Thesecrethas the bucket credentials, and theconfig maphas information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:oc extract --to=- cm/test-obc
$ oc extract --to=- cm/test-obcCopy to Clipboard Copied! Toggle word wrap Toggle overflow test-obcis the name of the OBC.Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To get the bucket credentials from the generated
secret, run the following command:oc extract --to=- secret/test-obc
$ oc extract --to=- secret/test-obcCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
# AWS_ACCESS_KEY_ID ebYR....xLNMc # AWS_SECRET_ACCESS_KEY YXf...+NaCkdyC3QPym
# AWS_ACCESS_KEY_ID ebYR....xLNMc # AWS_SECRET_ACCESS_KEY YXf...+NaCkdyC3QPymCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the public URL for the S3 endpoint from the s3 route in the
openshift-storagenamespace by running the following command:oc get route s3 -n openshift-storage
$ oc get route s3 -n openshift-storageCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
cloud-credentialsfile with the object bucket credentials as shown in the following command:[default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
[default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
cloud-credentialssecret with thecloud-credentialsfile content as shown in the following command:oc create secret generic \ cloud-credentials \ -n openshift-adp \ --from-file cloud=cloud-credentials
$ oc create secret generic \ cloud-credentials \ -n openshift-adp \ --from-file cloud=cloud-credentialsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the Data Protection Application (DPA) as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
defaultSnapshotMoveData-
Set to
trueto use the OADP Data Mover to enable movement of Container Storage Interface (CSI) snapshots to a remote object storage. s3Url- Specifies the S3 URL of ODF storage.
<bucket_name>- Specifies the bucket name.
Create the DPA by running the following command:
oc apply -f <dpa_filename>
$ oc apply -f <dpa_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the DPA is created successfully by running the following command. In the example output, you can see the
statusobject hastypefield set toReconciled. This means, the DPA is successfully created.oc get dpa -o yaml
$ oc get dpa -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the backup storage location (BSL) is available by running the following command:
oc get backupstoragelocations.velero.io -n openshift-adp
$ oc get backupstoragelocations.velero.io -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 3s 15s true
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 3s 15s trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure a backup CR as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<application_namespace>- Specifies the namespace for the application to back up.
Create the backup CR by running the following command:
oc apply -f <backup_cr_filename>
$ oc apply -f <backup_cr_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify that the backup object is in the
Completedphase by running the following command. For more details, see the example output.oc describe backup test-backup -n openshift-adp
$ oc describe backup test-backup -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.5.3. OpenShift API for Data Protection (OADP) restore use case Copy linkLink copied to clipboard!
Following is a use case for using OADP to restore a backup to a different namespace.
1.5.3.1. Restoring an application to a different namespace using OADP Copy linkLink copied to clipboard!
Restore a backup of an application by using OADP to a new target namespace, test-restore-application. To restore a backup, you create a restore custom resource (CR) as shown in the following example. In the restore CR, the source namespace refers to the application namespace that you included in the backup. You then verify the restore by changing your project to the new restored namespace and verifying the resources.
Prerequisites
- You installed the OADP Operator.
- You have the backup of an application to be restored.
Procedure
Create a restore CR as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
test-restore- Specifies the name of the restore CR.
<backup_name>- Specifies the name of the backup.
<application_namespace>-
Specifies the target namespace to restore to.
namespaceMappingmaps the source application namespace to the target application namespace.test-restore-applicationis the name of target namespace where you want to restore the backup.
Apply the restore CR by running the following command:
oc apply -f <restore_cr_filename>
$ oc apply -f <restore_cr_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify that the restore is in the
Completedphase by running the following command:oc describe restores.velero.io <restore_name> -n openshift-adp
$ oc describe restores.velero.io <restore_name> -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Change to the restored namespace
test-restore-applicationby running the following command:oc project test-restore-application
$ oc project test-restore-applicationCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify the restored resources such as persistent volume claim (pvc), service (svc), deployment, secret, and config map by running the following command:
oc get pvc,svc,deployment,secret,configmap
$ oc get pvc,svc,deployment,secret,configmapCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.5.4. Including a self-signed CA certificate during backup Copy linkLink copied to clipboard!
You can include a self-signed Certificate Authority (CA) certificate in the Data Protection Application (DPA) and then back up an application. You store the backup in a NooBaa bucket provided by Red Hat OpenShift Data Foundation (ODF).
1.5.4.1. Backing up an application and its self-signed CA certificate Copy linkLink copied to clipboard!
The s3.openshift-storage.svc service, provided by ODF, uses a Transport Layer Security protocol (TLS) certificate that is signed with the self-signed service CA.
To prevent a certificate signed by unknown authority error, you must include a self-signed CA certificate in the backup storage location (BSL) section of DataProtectionApplication custom resource (CR). For this situation, you must complete the following tasks:
- Request a NooBaa bucket by creating an object bucket claim (OBC).
- Extract the bucket details.
-
Include a self-signed CA certificate in the
DataProtectionApplicationCR. - Back up an application.
Prerequisites
- You installed the OADP Operator.
- You installed the ODF Operator.
- You have an application with a database running in a separate namespace.
Procedure
Create an OBC manifest to request a NooBaa bucket as shown in the following example:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
test-obc- Specifies the name of the object bucket claim.
test-backup-bucket- Specifies the name of the bucket.
Create the OBC by running the following command:
oc create -f <obc_file_name>
$ oc create -f <obc_file_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow When you create an OBC, ODF creates a
secretand aConfigMapwith the same name as the object bucket claim. Thesecretobject contains the bucket credentials, and theConfigMapobject contains information to access the bucket. To get the bucket name and bucket host from the generated config map, run the following command:oc extract --to=- cm/test-obc
$ oc extract --to=- cm/test-obcCopy to Clipboard Copied! Toggle word wrap Toggle overflow test-obcis the name of the OBC.Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To get the bucket credentials from the
secretobject, run the following command:oc extract --to=- secret/test-obc
$ oc extract --to=- secret/test-obcCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
# AWS_ACCESS_KEY_ID ebYR....xLNMc # AWS_SECRET_ACCESS_KEY YXf...+NaCkdyC3QPym
# AWS_ACCESS_KEY_ID ebYR....xLNMc # AWS_SECRET_ACCESS_KEY YXf...+NaCkdyC3QPymCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
cloud-credentialsfile with the object bucket credentials by using the following example configuration:[default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
[default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
cloud-credentialssecret with thecloud-credentialsfile content by running the following command:oc create secret generic \ cloud-credentials \ -n openshift-adp \ --from-file cloud=cloud-credentials
$ oc create secret generic \ cloud-credentials \ -n openshift-adp \ --from-file cloud=cloud-credentialsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the service CA certificate from the
openshift-service-ca.crtconfig map by running the following command. Ensure that you encode the certificate inBase64format and note the value to use in the next step.oc get cm/openshift-service-ca.crt \ -o jsonpath='{.data.service-ca\.crt}' | base64 -w0; echo$ oc get cm/openshift-service-ca.crt \ -o jsonpath='{.data.service-ca\.crt}' | base64 -w0; echoCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0... ....gpwOHMwaG9CRmk5a3....FLS0tLS0K
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0... ....gpwOHMwaG9CRmk5a3....FLS0tLS0KCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the
DataProtectionApplicationCR manifest file with the bucket name and CA certificate as shown in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
insecureSkipTLSVerify-
Specifies whether SSL/TLS security is enabled. If set to
true, SSL/TLS security is disabled. If set tofalse, SSL/TLS security is enabled. <bucket_name>- Specifies the name of the bucket extracted in an earlier step.
<ca_cert>-
Specifies the
Base64encoded certificate from the previous step.
Create the
DataProtectionApplicationCR by running the following command:oc apply -f <dpa_filename>
$ oc apply -f <dpa_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the
DataProtectionApplicationCR is created successfully by running the following command:oc get dpa -o yaml
$ oc get dpa -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the backup storage location (BSL) is available by running the following command:
oc get backupstoragelocations.velero.io -n openshift-adp
$ oc get backupstoragelocations.velero.io -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 3s 15s true
NAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 3s 15s trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the
BackupCR by using the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<application_namespace>- Specifies the namespace for the application to back up.
Create the
BackupCR by running the following command:oc apply -f <backup_cr_filename>
$ oc apply -f <backup_cr_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Verify that the
Backupobject is in theCompletedphase by running the following command:oc describe backup test-backup -n openshift-adp
$ oc describe backup test-backup -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.6. Installing and configuring OADP Copy linkLink copied to clipboard!
1.6.1. Installing OADP Copy linkLink copied to clipboard!
You can use OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS clusters to back up and restore application data.
Before installing OpenShift API for Data Protection (OADP), you must set up role and policy credentials for OADP so that it can use the Amazon Web Services API.
This process is performed in the following two stages:
- Prepare AWS credentials
- Install the OADP Operator and give it an IAM role
1.6.1.1. Preparing AWS credentials for OADP Copy linkLink copied to clipboard!
An Amazon Web Services account must be prepared and configured to accept an OpenShift API for Data Protection (OADP) installation.
Procedure
Create the following environment variables by running the following commands:
ImportantChange the cluster name to match your cluster, and ensure you are logged into the cluster as an administrator. Ensure that all fields are outputted correctly before continuing.
export CLUSTER_NAME=my-cluster
$ export CLUSTER_NAME=my-clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
my-cluster: Replacemy-clusterwith your cluster name.
export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id)$ export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id)Copy to Clipboard Copied! Toggle word wrap Toggle overflow export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)$ export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)Copy to Clipboard Copied! Toggle word wrap Toggle overflow export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')Copy to Clipboard Copied! Toggle word wrap Toggle overflow export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)Copy to Clipboard Copied! Toggle word wrap Toggle overflow export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.')$ export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.')Copy to Clipboard Copied! Toggle word wrap Toggle overflow export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"$ export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"Copy to Clipboard Copied! Toggle word wrap Toggle overflow export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"$ export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"Copy to Clipboard Copied! Toggle word wrap Toggle overflow mkdir -p ${SCRATCH}$ mkdir -p ${SCRATCH}Copy to Clipboard Copied! Toggle word wrap Toggle overflow echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"$ echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
On the AWS account, create an IAM policy to allow access to AWS S3:
Check to see if the policy exists by running the following command:
POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text)$ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text)Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
RosaOadp: ReplaceRosaOadpwith your policy name.
-
Enter the following command to create the policy JSON file and then create the policy:
NoteIf the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the
ifstatement intentionally skips the policy creation.Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
SCRATCH:SCRATCHis a name for a temporary directory created for the environment variables.
-
View the policy ARN by running the following command:
echo ${POLICY_ARN}$ echo ${POLICY_ARN}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Create an IAM role trust policy for the cluster:
Create the trust policy file by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the role by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow View the role ARN by running the following command:
echo ${ROLE_ARN}$ echo ${ROLE_ARN}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Attach the IAM policy to the IAM role by running the following command:
aws iam attach-role-policy --role-name "${ROLE_NAME}" \ --policy-arn ${POLICY_ARN}$ aws iam attach-role-policy --role-name "${ROLE_NAME}" \ --policy-arn ${POLICY_ARN}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.6.1.2. Installing the OADP Operator and providing the IAM role Copy linkLink copied to clipboard!
AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. Red Hat OpenShift Service on AWS with STS is the recommended credential mode. This document describes how to install OpenShift API for Data Protection (OADP) on clusters with AWS STS.
Restic is unsupported.
Kopia file system backup (FSB) is supported when backing up file systems that do not support Container Storage Interface (CSI) snapshots.
Example file systems include the following:
- Amazon Elastic File System (EFS)
- Network File System (NFS)
-
emptyDirvolumes - Local volumes
For backing up volumes, OADP on ROSA with AWS STS recommends native snapshots and Container Storage Interface (CSI) snapshots. Data Mover backups are supported, but can be slower than native snapshots.
In an Amazon ROSA cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.
Prerequisites
-
A Red Hat OpenShift Service on AWS cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including
ROLE_ARN, for each cluster.
Procedure
Create a Red Hat OpenShift Service on AWS secret from your AWS token file by entering the following commands:
Create the credentials file:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<aws_region>with the AWS region to use for the STS endpoint.
Create a namespace for OADP:
oc create namespace openshift-adp
$ oc create namespace openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create the Red Hat OpenShift Service on AWS secret:
oc -n openshift-adp create secret generic cloud-credentials \ --from-file=${SCRATCH}/credentials$ oc -n openshift-adp create secret generic cloud-credentials \ --from-file=${SCRATCH}/credentialsCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIn Red Hat OpenShift Service on AWS versions 4.15 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the Red Hat OpenShift Service on AWS web console, for more information see Installing from software catalog using the web console.
The preceding secret is created automatically by CCO.
Install the OADP Operator:
- In the Red Hat OpenShift Service on AWS web console, browse to Ecosystem → Software Catalog.
- Search for the OADP Operator.
- In the role_ARN field, paste the role_arn that you created previously and click Install.
Create AWS cloud storage using your AWS credentials by entering the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check your application’s storage default storage class by entering the following command:
oc get pvc -n <namespace>
$ oc get pvc -n <namespace>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE applog Bound pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8 1Gi RWO gp3-csi 4d19h mysql Bound pvc-16b8e009-a20a-4379-accc-bc81fedd0621 1Gi RWO gp3-csi 4d19h
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE applog Bound pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8 1Gi RWO gp3-csi 4d19h mysql Bound pvc-16b8e009-a20a-4379-accc-bc81fedd0621 1Gi RWO gp3-csi 4d19hCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the storage class by running the following command:
oc get storageclass
$ oc get storageclassCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 4d21h gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 4d21h gp2-csi ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21h gp3-csi (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 4d21hCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe following storage classes will work:
- gp3-csi
- gp2-csi
- gp3
- gp2
If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.
Create the
DataProtectionApplicationresource to configure the connection to the storage where the backups and volume snapshots are stored:If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Red Hat OpenShift Service on AWS supports internal image backup. Set this field to
falseif you do not want to use image backup. - 2
- See the important note regarding the
nodeAgentattribute at the end of this procedure. - 3
- The type of uploader. The built-in Data Mover uses Kopia as the default uploader mechanism regardless of the value of the
uploaderTypefield.
If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Red Hat OpenShift Service on AWS supports internal image backup. Set this field to
falseif you do not want to use image backup. - 2
- See the important note regarding the
nodeAgentattribute at the end of this procedure. - 3
- The
credentialsFilefield is the mounted location of the bucket credential on the pod. - 4
- The
enableSharedConfigfield allows thesnapshotLocationsto share or reuse the credential defined for the bucket. - 5
- Use the profile name set in the AWS credentials file.
- 6
- Specify
regionas your AWS region. This must be the same as the cluster region.
You are now ready to back up and restore Red Hat OpenShift Service on AWS applications, as described in Backing up applications.
The enable parameter of restic is set to false in this configuration, because OADP does not support Restic in Red Hat OpenShift Service on AWS environments.
If you use OADP 1.2, replace this configuration:
nodeAgent: enable: false uploaderType: restic
nodeAgent:
enable: false
uploaderType: restic
with the following configuration:
restic: enable: false
restic:
enable: false
If you want to use two different clusters for backing up and restoring, the two clusters must have the same AWS S3 storage names in both the cloud storage CR and the OADP DataProtectionApplication configuration.
1.6.1.3. Updating the IAM role ARN in the OADP Operator subscription Copy linkLink copied to clipboard!
While installing the OADP Operator on a Red Hat OpenShift Service on AWS cluster, if you provide an incorrect IAM role Amazon Resource Name (ARN), the openshift-adp-controller pod gives an error. The credential requests that are generated contain the wrong IAM role ARN. To update the credential requests object with the correct IAM role ARN, you can edit the OADP Operator subscription and patch the IAM role ARN with the correct value. By editing the OADP Operator subscription, you do not have to uninstall and reinstall OADP to update the IAM role ARN.
Prerequisites
- You have a Red Hat OpenShift Service on AWS cluster with the required access and tokens.
- You have installed OADP on the ROSA STS cluster.
Procedure
To verify that the OADP subscription has the wrong IAM role ARN environment variable set, run the following command:
oc get sub -o yaml redhat-oadp-operator
$ oc get sub -o yaml redhat-oadp-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example subscription
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Verify the value of
ROLEARNyou want to update.
Update the
ROLEARNfield of the subscription with the correct role ARN by running the following command:oc patch subscription redhat-oadp-operator -p '{"spec": {"config": {"env": [{"name": "ROLEARN", "value": "<role_arn>"}]}}}' --type='merge'$ oc patch subscription redhat-oadp-operator -p '{"spec": {"config": {"env": [{"name": "ROLEARN", "value": "<role_arn>"}]}}}' --type='merge'Copy to Clipboard Copied! Toggle word wrap Toggle overflow where:
<role_arn>-
Specifies the IAM role ARN to be updated. For example,
arn:aws:iam::160…..6956:role/oadprosa…..8wlf.
Verify that the
secretobject is updated with correct role ARN value by running the following command:oc get secret cloud-credentials -o jsonpath='{.data.credentials}' | base64 -d$ oc get secret cloud-credentials -o jsonpath='{.data.credentials}' | base64 -dCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
[default] sts_regional_endpoints = regional role_arn = arn:aws:iam::160.....6956:role/oadprosa.....8wlf web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
[default] sts_regional_endpoints = regional role_arn = arn:aws:iam::160.....6956:role/oadprosa.....8wlf web_identity_token_file = /var/run/secrets/openshift/serviceaccount/tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the
DataProtectionApplicationcustom resource (CR) manifest file as shown in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the
CloudStorageCR.
Create the
DataProtectionApplicationCR by running the following command:oc create -f <dpa_manifest_file>
$ oc create -f <dpa_manifest_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the
DataProtectionApplicationCR is reconciled and thestatusis set to"True"by running the following command:oc get dpa -n openshift-adp -o yaml
$ oc get dpa -n openshift-adp -o yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example
DataProtectionApplicationCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the
BackupStorageLocationCR is in an available state by running the following command:oc get backupstoragelocations.velero.io -n openshift-adp
$ oc get backupstoragelocations.velero.io -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example
BackupStorageLocationNAME PHASE LAST VALIDATED AGE DEFAULT ts-dpa-1 Available 3s 6s true
NAME PHASE LAST VALIDATED AGE DEFAULT ts-dpa-1 Available 3s 6s trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow
1.7. OADP backing up Copy linkLink copied to clipboard!
1.7.1. Backing up applications Copy linkLink copied to clipboard!
Frequent backups might consume storage on the backup storage location. Check the frequency of backups, retention time, and the amount of data of the persistent volumes (PVs) if using non-local backups, for example, S3 buckets. Because all taken backup remains until expired, also check the time to live (TTL) setting of the schedule.
You can back up applications by creating a Backup custom resource (CR). For more information, see Creating a Backup CR. The following are the different backup types for a Backup CR:
The Backup CR creates backup files for Kubernetes resources and internal images on S3 object storage.
1.7.1.1. Previewing resources before running backup and restore Copy linkLink copied to clipboard!
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete. To preview the resources in advance, you can do a dry run of the backup and restore operations.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
velero backup create <backup-name> --snapshot-volumes false
$ velero backup create <backup-name> --snapshot-volumes false1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the value of
--snapshot-volumesparameter asfalse.
To know more details about the backup resources, run the following command:
velero describe backup <backup_name> --details
$ velero describe backup <backup_name> --details1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
velero restore create --from-backup <backup-name>
$ velero restore create --from-backup <backup-name>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup created to review the backup resources.
ImportantThe
velero restore createcommand creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
velero describe restore <restore_name> --details
$ velero describe restore <restore_name> --details1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the restore.
You can create backup hooks to run commands before or after the backup operation. See Creating backup hooks.
You can schedule backups by creating a Schedule CR instead of a Backup CR. See Scheduling backups using Schedule CR.
1.7.1.2. Known issues Copy linkLink copied to clipboard!
Red Hat OpenShift Service on AWS 4 enforces a pod security admission (PSA) policy that can hinder the readiness of pods during a Restic restore process.
This issue has been resolved in the OADP 1.1.6 and OADP 1.2.2 releases, therefore it is recommended that users upgrade to these releases.
1.7.3. Creating backup hooks Copy linkLink copied to clipboard!
When performing a backup, it is possible to specify one or more commands to execute in a container within a pod, based on the pod being backed up.
The commands can be configured to performed before any custom action processing (Pre hooks), or after all custom actions have been completed and any additional items specified by the custom action have been backed up (Post hooks).
You create backup hooks to run commands in a container in a pod by editing the Backup custom resource (CR).
Procedure
Add a hook to the
spec.hooksblock of theBackupCR, as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Optional: You can specify namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Optional: You can specify namespaces to which the hook does not apply.
- 3
- Currently, pods are the only supported resource that hooks can apply to.
- 4
- Optional: You can specify resources to which the hook does not apply.
- 5
- Optional: This hook only applies to objects matching the label. If this value is not specified, the hook applies to all objects.
- 6
- Array of hooks to run before the backup.
- 7
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 8
- This is the entry point for the
initcontainer being added. - 9
- Allowed values for error handling are
FailandContinue. The default isFail. - 10
- Optional: How long to wait for the commands to run. The default is
30s. - 11
- This block defines an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.
1.7.4. Scheduling backups using Schedule CR Copy linkLink copied to clipboard!
The schedule operation allows you to create a backup of your data at a particular time, specified by a Cron expression.
You schedule backups by creating a Schedule custom resource (CR) instead of a Backup CR.
Leave enough time in your backup schedule for a backup to finish before another backup is created.
For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplicationCR must be in aReadystate.
Procedure
Retrieve the
backupStorageLocationsCRs:oc get backupStorageLocations -n openshift-adp
$ oc get backupStorageLocations -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31m
NAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT openshift-adp velero-sample-1 Available 11s 31mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
ScheduleCR, as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- 1
cronexpression to schedule the backup, for example,0 7 * * *to perform a backup every day at 7:00.NoteTo schedule a backup at specific intervals, enter the
<duration_in_minutes>in the following format:schedule: "*/10 * * * *"
schedule: "*/10 * * * *"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Enter the minutes value between quotation marks (
" ").- 2
- Array of namespaces to back up.
- 3
- Name of the
backupStorageLocationsCR. - 4
- Optional: In OADP version 1.2 and later, add the
defaultVolumesToFsBackup: truekey-value pair to your configuration when performing backups of volumes with Restic. In OADP version 1.1, add thedefaultVolumesToRestic: truekey-value pair when you back up volumes with Restic.
Verification
Verify that the status of the
ScheduleCR isCompletedafter the scheduled backup runs:oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'$ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.7.5. Deleting backups Copy linkLink copied to clipboard!
You can delete a backup by creating the DeleteBackupRequest custom resource (CR) or by running the velero backup delete command as explained in the following procedures.
The volume backup artifacts are deleted at different times depending on the backup method:
- Restic: The artifacts are deleted in the next full maintenance cycle, after the backup is deleted.
- Container Storage Interface (CSI): The artifacts are deleted immediately when the backup is deleted.
- Kopia: The artifacts are deleted after three full maintenance cycles of the Kopia repository, after the backup is deleted.
1.7.5.1. Deleting a backup by creating a DeleteBackupRequest CR Copy linkLink copied to clipboard!
You can delete a backup by creating a DeleteBackupRequest custom resource (CR).
Prerequisites
- You have run a backup of your application.
Procedure
Create a
DeleteBackupRequestCR manifest file:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup.
Apply the
DeleteBackupRequestCR to delete the backup:oc apply -f <deletebackuprequest_cr_filename>
$ oc apply -f <deletebackuprequest_cr_filename>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.7.5.2. Deleting a backup by using the Velero CLI Copy linkLink copied to clipboard!
You can delete a backup by using the Velero CLI.
Prerequisites
- You have run a backup of your application.
- You downloaded the Velero CLI and can access the Velero binary in your cluster.
Procedure
To delete the backup, run the following Velero command:
velero backup delete <backup_name> -n openshift-adp
$ velero backup delete <backup_name> -n openshift-adp1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup.
1.7.5.3. About Kopia repository maintenance Copy linkLink copied to clipboard!
There are two types of Kopia repository maintenance:
- Quick maintenance
- Runs every hour to keep the number of index blobs (n) low. A high number of indexes negatively affects the performance of Kopia operations.
- Does not delete any metadata from the repository without ensuring that another copy of the same metadata exists.
- Full maintenance
- Runs every 24 hours to perform garbage collection of repository contents that are no longer needed.
-
snapshot-gc, a full maintenance task, finds all files and directory listings that are no longer accessible from snapshot manifests and marks them as deleted. - A full maintenance is a resource-costly operation, as it requires scanning all directories in all snapshots that are active in the cluster.
1.7.5.3.1. Kopia maintenance in OADP Copy linkLink copied to clipboard!
The repo-maintain-job jobs are executed in the namespace where OADP is installed, as shown in the following example:
pod/repo-maintain-job-173...2527-2nbls 0/1 Completed 0 168m pod/repo-maintain-job-173....536-fl9tm 0/1 Completed 0 108m pod/repo-maintain-job-173...2545-55ggx 0/1 Completed 0 48m
pod/repo-maintain-job-173...2527-2nbls 0/1 Completed 0 168m
pod/repo-maintain-job-173....536-fl9tm 0/1 Completed 0 108m
pod/repo-maintain-job-173...2545-55ggx 0/1 Completed 0 48m
You can check the logs of the repo-maintain-job for more details about the cleanup and the removal of artifacts in the backup object storage. You can find a note, as shown in the following example, in the repo-maintain-job when the next full cycle maintenance is due:
not due for full maintenance cycle until 2024-00-00 18:29:4
not due for full maintenance cycle until 2024-00-00 18:29:4
Three successful executions of a full maintenance cycle are required for the objects to be deleted from the backup object storage. This means you can expect up to 72 hours for all the artifacts in the backup object storage to be deleted.
1.7.5.4. Deleting a backup repository Copy linkLink copied to clipboard!
After you delete the backup, and after the Kopia repository maintenance cycles to delete the related artifacts are complete, the backup is no longer referenced by any metadata or manifest objects. You can then delete the backuprepository custom resource (CR) to complete the backup deletion process.
Prerequisites
- You have deleted the backup of your application.
- You have waited up to 72 hours after the backup is deleted. This time frame allows Kopia to run the repository maintenance cycles.
Procedure
To get the name of the backup repository CR for a backup, run the following command:
oc get backuprepositories.velero.io -n openshift-adp
$ oc get backuprepositories.velero.io -n openshift-adpCopy to Clipboard Copied! Toggle word wrap Toggle overflow To delete the backup repository CR, run the following command:
oc delete backuprepository <backup_repository_name> -n openshift-adp
$ oc delete backuprepository <backup_repository_name> -n openshift-adp1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup repository from the earlier step.
1.8. OADP restoring Copy linkLink copied to clipboard!
1.8.1. Restoring applications Copy linkLink copied to clipboard!
You restore application backups by creating a Restore custom resource (CR). See Creating a Restore CR.
You can create restore hooks to run commands in a container in a pod by editing the Restore CR. See Creating restore hooks.
1.8.1.1. Previewing resources before running backup and restore Copy linkLink copied to clipboard!
OADP backs up application resources based on the type, namespace, or label. This means that you can view the resources after the backup is complete. Similarly, you can view the restored objects based on the namespace, persistent volume (PV), or label after a restore operation is complete. To preview the resources in advance, you can do a dry run of the backup and restore operations.
Prerequisites
- You have installed the OADP Operator.
Procedure
To preview the resources included in the backup before running the actual backup, run the following command:
velero backup create <backup-name> --snapshot-volumes false
$ velero backup create <backup-name> --snapshot-volumes false1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the value of
--snapshot-volumesparameter asfalse.
To know more details about the backup resources, run the following command:
velero describe backup <backup_name> --details
$ velero describe backup <backup_name> --details1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup.
To preview the resources included in the restore before running the actual restore, run the following command:
velero restore create --from-backup <backup-name>
$ velero restore create --from-backup <backup-name>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the backup created to review the backup resources.
ImportantThe
velero restore createcommand creates restore resources in the cluster. You must delete the resources created as part of the restore, after you review the resources.To know more details about the restore resources, run the following command:
velero describe restore <restore_name> --details
$ velero describe restore <restore_name> --details1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the name of the restore.
1.8.1.2. Creating a Restore CR Copy linkLink copied to clipboard!
You restore a Backup custom resource (CR) by creating a Restore CR.
When you restore a stateful application that uses the azurefile-csi storage class, the restore operation remains in the Finalizing phase.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplicationCR must be in aReadystate. -
You must have a Velero
BackupCR. - The persistent volume (PV) capacity must match the requested size at backup time. Adjust the requested size if needed.
Procedure
Create a
RestoreCR, as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Name of the
BackupCR. - 2
- Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example,
poforpods) or fully-qualified. If unspecified, all resources are included. - 3
- Optional: The
restorePVsparameter can be set tofalseto turn off restore ofPersistentVolumesfromVolumeSnapshotof Container Storage Interface (CSI) snapshots or from native snapshots whenVolumeSnapshotLocationis configured.
Verify that the status of the
RestoreCR isCompletedby entering the following command:oc get restores.velero.io -n openshift-adp <restore> -o jsonpath='{.status.phase}'$ oc get restores.velero.io -n openshift-adp <restore> -o jsonpath='{.status.phase}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the backup resources have been restored by entering the following command:
oc get all -n <namespace>
$ oc get all -n <namespace>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Namespace that you backed up.
If you restore
DeploymentConfigwith volumes or if you use post-restore hooks, run thedc-post-restore.shcleanup script by entering the following command:bash dc-restic-post-restore.sh -> dc-post-restore.sh
$ bash dc-restic-post-restore.sh -> dc-post-restore.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteDuring the restore process, the OADP Velero plug-ins scale down the
DeploymentConfigobjects and restore the pods as standalone pods. This is done to prevent the cluster from deleting the restoredDeploymentConfigpods immediately on restore and to allow the restore and post-restore hooks to complete their actions on the restored pods. The cleanup script shown below removes these disconnected pods and scales anyDeploymentConfigobjects back up to the appropriate number of replicas.dc-restic-post-restore.sh → dc-post-restore.shcleanup scriptCopy to Clipboard Copied! Toggle word wrap Toggle overflow
1.8.1.3. Creating restore hooks Copy linkLink copied to clipboard!
You create restore hooks to run commands in a container in a pod by editing the Restore custom resource (CR).
You can create two types of restore hooks:
An
inithook adds an init container to a pod to perform setup tasks before the application container starts.If you restore a Restic backup, the
restic-waitinit container is added before the restore hook init container.-
An
exechook runs commands or scripts in a container of a restored pod.
Procedure
Add a hook to the
spec.hooksblock of theRestoreCR, as in the following example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Optional: Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Currently, pods are the only supported resource that hooks can apply to.
- 3
- Optional: This hook only applies to objects matching the label selector.
- 4
- Optional: Timeout specifies the maximum length of time Velero waits for
initContainersto complete. - 5
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 6
- This is the entrypoint for the init container being added.
- 7
- Optional: How long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.
- 8
- Optional: How long to wait for the commands to run. The default is
30s. - 9
- Allowed values for error handling are
FailandContinue:-
Continue: Only command failures are logged. -
Fail: No more restore hooks run in any container in any pod. The status of theRestoreCR will bePartiallyFailed.
-
During a File System Backup (FSB) restore operation, a Deployment resource referencing an ImageStream is not restored properly. The restored pod that runs the FSB, and the postHook is terminated prematurely.
This happens because, during the restore operation, OpenShift controller updates the spec.template.spec.containers[0].image field in the Deployment resource with an updated ImageStreamTag hash. The update triggers the rollout of a new pod, terminating the pod on which velero runs the FSB and the post restore hook. For more information about image stream trigger, see "Triggering updates on image stream changes".
The workaround for this behavior is a two-step restore process:
First, perform a restore excluding the
Deploymentresources, for example:velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --exclude-resources=deployment.apps
$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --exclude-resources=deployment.appsCopy to Clipboard Copied! Toggle word wrap Toggle overflow After the first restore is successful, perform a second restore by including these resources, for example:
velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --include-resources=deployment.apps
$ velero restore create <RESTORE_NAME> \ --from-backup <BACKUP_NAME> \ --include-resources=deployment.appsCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Legal Notice
Copy linkLink copied to clipboard!
Copyright © 2025 Red Hat
OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
Modified versions must remove all Red Hat trademarks.
Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.
Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.