Chapter 4. Application backup and restore
4.1. OADP features and plug-ins
OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications.
The default plug-ins enable Velero to integrate with certain cloud providers and to back up and restore OpenShift Container Platform resources.
4.1.1. OADP features
OpenShift API for Data Protection (OADP) supports the following features:
- Backup
You can back up all resources in your cluster or you can filter the resources by type, namespace, or label.
OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.
- Restore
- You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the restored objects by namespace, PV, or label.
- Schedule
- You can schedule backups at specified intervals.
- Hooks
-
You can use hooks to run commands in a container on a pod, for example,
fsfreeze
to freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.
4.1.2. OADP plug-ins
The OpenShift API for Data Protection (OADP) provides default Velero plug-ins that are integrated with storage providers to support backup and snapshot operations. You can create custom plug-ins based on the Velero plug-ins.
OADP also provides plug-ins for OpenShift Container Platform resource backups and Container Storage Interface (CSI) snapshots.
OADP plug-in | Function | Storage location |
---|---|---|
| Backs up and restores Kubernetes objects by using object store. | AWS S3 |
Backs up and restores volumes by using snapshots. | AWS EBS | |
| Backs up and restores Kubernetes objects by using object store. | Microsoft Azure Blob storage |
Backs up and restores volumes by using snapshots. | Microsoft Azure Managed Disks | |
| Backs up and restores Kubernetes objects by using object store. | Google Cloud Storage |
Backs up and restores volumes by using snapshots. | Google Compute Engine Disks | |
| Backs up and restores OpenShift Container Platform resources by using object store. [1] | Object store |
| Backs up and restores volumes by using CSI snapshots. [2] | Cloud storage that supports CSI snapshots |
- Mandatory.
-
The
csi
plug-in uses the Velero CSI beta snapshot API.
4.1.3. About OADP Velero plug-ins
You can configure two types of plug-ins when you install Velero:
- Default cloud provider plug-ins
- Custom plug-ins
Both types of plug-in are optional, but most users configure at least one cloud provider plug-in.
4.1.3.1. Default Velero cloud provider plug-ins
You can install any of the following default Velero cloud provider plug-ins when you configure the oadp_v1alpha1_dpa.yaml
file during deployment:
-
aws
(Amazon Web Services) -
gcp
(Google Cloud Platform) -
azure
(Microsoft Azure) -
openshift
(OpenShift Velero plug-in) -
csi
(Container Storage Interface) -
kubevirt
(KubeVirt)
You specify the desired default plug-ins in the oadp_v1alpha1_dpa.yaml
file during deployment.
Example file
The following .yaml
file installs the openshift
, aws
, azure
, and gcp
plug-ins:
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample spec: configuration: velero: defaultPlugins: - openshift - aws - azure - gcp
4.1.3.2. Custom Velero plug-ins
You can install a custom Velero plug-in by specifying the plug-in image
and name
when you configure the oadp_v1alpha1_dpa.yaml
file during deployment.
You specify the desired custom plug-ins in the oadp_v1alpha1_dpa.yaml
file during deployment.
Example file
The following .yaml
file installs the default openshift
, azure
, and gcp
plug-ins and a custom plug-in that has the name custom-plugin-example
and the image quay.io/example-repo/custom-velero-plugin
:
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-sample spec: configuration: velero: defaultPlugins: - openshift - azure - gcp customPlugins: - name: custom-plugin-example image: quay.io/example-repo/custom-velero-plugin
4.2. Installing and configuring OADP
4.2.1. About installing OADP
As a cluster administrator, you install the OpenShift API for Data Protection (OADP) by installing the OADP Operator. The OADP Operator installs Velero 1.7.
To back up Kubernetes resources and internal images, you must have object storage as a backup location, such as one of the following storage types:
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- Multicloud Object Gateway
- S3-compatible object storage, such as Noobaa or Minio
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
You can back up persistent volumes (PVs) by using snapshots or Restic.
To back up PVs with snapshots, you must have a cloud provider that supports either a native snapshot API or Container Storage Interface (CSI) snapshots, such as one of the following cloud providers:
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- CSI snapshot-enabled cloud provider, such as OpenShift Container Storage
If your cloud provider does not support snapshots or if your storage is NFS, you can back up applications with Restic.
You create a Secret
object for your storage provider credentials and then you install the Data Protection Application.
Additional resources
- Overview of backup locations and snapshot locations in the Velero documentation.
4.2.2. Installing and configuring the OpenShift API for Data Protection with Amazon Web Services
You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) by installing the OADP Operator, configuring AWS for Velero, and then installing the Data Protection Application.
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.2.2.1. Installing the OADP Operator
You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.6 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.7.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
-
In the OpenShift Container Platform web console, click Operators
OperatorHub. - Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. -
Click Operators
Installed Operators to verify the installation.
4.2.2.2. Configuring Amazon Web Services S3
You can configure an Amazon Web Services (AWS) S3 storage bucket as a replication repository for the Migration Toolkit for Containers (MTC).
Prerequisites
- The AWS S3 storage bucket must be accessible to the source and target clusters.
- You must have the AWS CLI installed.
If you are using the snapshot copy method:
- You must have access to EC2 Elastic Block Storage (EBS).
- The source and target clusters must be in the same region.
- The source and target clusters must have the same storage class.
- The storage class must be compatible with snapshots.
Procedure
Create an AWS S3 bucket:
$ aws s3api create-bucket \ --bucket <bucket> \ 1 --region <bucket_region> 2
Create the IAM user
velero
:$ aws iam create-user --user-name velero
Create an EC2 EBS snapshot policy:
$ cat > velero-ec2-snapshot-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" } ] } EOF
Create an AWS S3 access policy for one or for all S3 buckets:
$ cat > velero-s3-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:DeleteObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::<bucket>/*" 1 ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::<bucket>" 2 ] } ] } EOF
Example output
"Resource": [ "arn:aws:s3:::*"
Attach the EC2 EBS policy to
velero
:$ aws iam put-user-policy \ --user-name velero \ --policy-name velero-ebs \ --policy-document file://velero-ec2-snapshot-policy.json
Attach the AWS S3 policy to
velero
:$ aws iam put-user-policy \ --user-name velero \ --policy-name velero-s3 \ --policy-document file://velero-s3-policy.json
Create an access key for
velero
:$ aws iam create-access-key --user-name velero { "AccessKey": { "UserName": "velero", "Status": "Active", "CreateDate": "2017-07-31T22:24:41.576Z", "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1 "AccessKeyId": <AWS_ACCESS_KEY_ID> 2 } }
Create a
credentials-velero
file:$ cat << EOF > ./credentials-velero [default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> EOF
You use the
credentials-velero
file to create aSecret
object for AWS before you install the Data Protection Application.
4.2.2.3. Creating a secret for backup and snapshot locations
You create a Secret
object for the backup and snapshot locations if they use the same credentials.
The default name of the Secret
is cloud-credentials
.
Prerequisites
- Your object storage and cloud storage must use the same credentials.
- You must configure object storage for Velero.
You must create a
credentials-velero
file for the object storage in the appropriate format.NoteThe
DataProtectionApplication
custom resource (CR) requires aSecret
for installation. If nospec.backupLocations.credential.name
value is specified, the default name is used.If you do not want to specify the backup locations or the snapshot locations, you must create a
Secret
with the default name by using an emptycredentials-velero
file.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.2.2.3.1. Configuring secrets for different backup and snapshot location credentials
If your backup and snapshot locations use different credentials, you create separate profiles in the credentials-velero
file.
Then, you create a Secret
object and specify the profiles in the DataProtectionApplication
custom resource (CR).
Procedure
Create a
credentials-velero
file with separate profiles for the backup and snapshot locations, as in the following example:[backupStorage] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> [volumeSnapshot] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
Create a
Secret
object with thecredentials-velero
file:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero 1
Add the profiles to the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket_name> prefix: <prefix> config: region: us-east-1 profile: "backupStorage" credential: key: cloud name: cloud-credentials snapshotLocations: - name: default velero: provider: aws config: region: us-west-2 profile: "volumeSnapshot"
4.2.2.4. Configuring the Data Protection Application
You can configure Velero resource allocations and enable self-signed CA certificates.
4.2.2.4.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... configuration: velero: podConfig: resourceAllocations: limits: cpu: "1" 1 memory: 512Mi 2 requests: cpu: 500m 3 memory: 256Mi 4
4.2.2.4.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 ...
4.2.2.5. Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create a
Secret
with the default name,cloud-credentials
, which contains separate profiles for the backup and snapshot location credentials.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
Procedure
-
Click Operators
Installed Operators and select the OADP Operator. - Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - openshift <.> - aws restic: enable: true <.> backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket_name> <.> prefix: <prefix> <.> config: region: <region> profile: "default" credential: key: cloud name: cloud-credentials <.> snapshotLocations: <.> - name: default velero: provider: aws config: region: <region> <.> profile: "default"
<.> The
openshift
plug-in is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster. <.> Set tofalse
if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node hasRestic
pods running. You configure Restic for backups by addingspec.defaultVolumesToRestic: true
to theBackup
CR. <.> Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix. <.> Specify a prefix for Velero backups, for example,velero
, if the bucket is used for multiple purposes. <.> Specify the name of theSecret
object that you created. If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. <.> You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs. <.> The snapshot location must be in the same region as the PVs.- Click Create.
Verify the installation by viewing the OADP resources:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/oadp-velero-sample-1-aws-registry-5d6968cbdd-d5w9k 1/1 Running 0 95s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/oadp-velero-sample-1-aws-registry-svc ClusterIP 172.30.130.230 <none> 5000/TCP 95s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/oadp-velero-sample-1-aws-registry 1/1 1 1 96s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/oadp-velero-sample-1-aws-registry-5d6968cbdd 1 1 1 96s replicaset.apps/velero-588db7f655 1 1 1 96s
4.2.2.5.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1 featureFlags: - EnableCSI 2
4.2.3. Installing and configuring the OpenShift API for Data Protection with Microsoft Azure
You install the OpenShift API for Data Protection (OADP) with Microsoft Azure by installing the OADP Operator, configuring Azure for Velero, and then installing the Data Protection Application.
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.2.3.1. Installing the OADP Operator
You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.6 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.7.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
-
In the OpenShift Container Platform web console, click Operators
OperatorHub. - Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. -
Click Operators
Installed Operators to verify the installation.
4.2.3.2. Configuring Microsoft Azure Blob
You can configure a Microsoft Azure Blob storage container as a replication repository for the Migration Toolkit for Containers (MTC).
Prerequisites
- You must have an Azure storage account.
- You must have the Azure CLI installed.
- The Azure Blob storage container must be accessible to the source and target clusters.
If you are using the snapshot copy method:
- The source and target clusters must be in the same region.
- The source and target clusters must have the same storage class.
- The storage class must be compatible with snapshots.
Procedure
Set the
AZURE_RESOURCE_GROUP
variable:$ AZURE_RESOURCE_GROUP=Velero_Backups
Create an Azure resource group:
$ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1
- 1
- Specify your location.
Set the
AZURE_STORAGE_ACCOUNT_ID
variable:$ AZURE_STORAGE_ACCOUNT_ID=velerobackups
Create an Azure storage account:
$ az storage account create \ --name $AZURE_STORAGE_ACCOUNT_ID \ --resource-group $AZURE_RESOURCE_GROUP \ --sku Standard_GRS \ --encryption-services blob \ --https-only true \ --kind BlobStorage \ --access-tier Hot
Set the
BLOB_CONTAINER
variable:$ BLOB_CONTAINER=velero
Create an Azure Blob storage container:
$ az storage container create \ -n $BLOB_CONTAINER \ --public-access off \ --account-name $AZURE_STORAGE_ACCOUNT_ID
Obtain the storage account access key:
$ AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list \ --account-name $AZURE_STORAGE_ACCOUNT_ID \ --query "[?keyName == 'key1'].value" -o tsv`
Create a
credentials-velero
file:$ cat << EOF > ./credentials-velero AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} AZURE_TENANT_ID=${AZURE_TENANT_ID} AZURE_CLIENT_ID=${AZURE_CLIENT_ID} AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET} AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} 1 AZURE_CLOUD_NAME=AzurePublicCloud EOF
- 1
- Mandatory. You cannot back up internal images if the
credentials-velero
file contains only the service principal credentials.
You use the
credentials-velero
file to create aSecret
object for Azure before you install the Data Protection Application.
4.2.3.3. Creating a secret for backup and snapshot locations
You create a Secret
object for the backup and snapshot locations if they use the same credentials.
The default name of the Secret
is cloud-credentials-azure
.
Prerequisites
- Your object storage and cloud storage must use the same credentials.
- You must configure object storage for Velero.
You must create a
credentials-velero
file for the object storage in the appropriate format.NoteThe
DataProtectionApplication
custom resource (CR) requires aSecret
for installation. If nospec.backupLocations.credential.name
value is specified, the default name is used.If you do not want to specify the backup locations or the snapshot locations, you must create a
Secret
with the default name by using an emptycredentials-velero
file.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.2.3.3.1. Configuring secrets for different backup and snapshot location credentials
If your backup and snapshot locations use different credentials, you create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials-azure
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: config: resourceGroup: <azure_resource_group> storageAccount: <azure_storage_account_id> subscriptionId: <azure_subscription_id> storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY credential: key: cloud name: <custom_secret> 1 provider: azure default: true objectStorage: bucket: <bucket_name> prefix: <prefix> snapshotLocations: - velero: config: resourceGroup: <azure_resource_group> subscriptionId: <azure_subscription_id> incremental: "true" name: default provider: azure
- 1
- Backup location
Secret
with custom name.
4.2.3.4. Configuring the Data Protection Application
You can configure Velero resource allocations and enable self-signed CA certificates.
4.2.3.4.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... configuration: velero: podConfig: resourceAllocations: limits: cpu: "1" 1 memory: 512Mi 2 requests: cpu: 500m 3 memory: 256Mi 4
4.2.3.4.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 ...
4.2.3.5. Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-azure
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with the default name,cloud-credentials-azure
, for the snapshot location. ThisSecret
is not referenced in theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
-
Procedure
-
Click Operators
Installed Operators and select the OADP Operator. - Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - azure - openshift <.> restic: enable: true <.> backupLocations: - velero: config: resourceGroup: <azure_resource_group> <.> storageAccount: <azure_storage_account_id> <.> subscriptionId: <azure_subscription_id> <.> storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY credential: key: cloud name: cloud-credentials-azure <.> provider: azure default: true objectStorage: bucket: <bucket_name> <.> prefix: <prefix> <.> snapshotLocations: <.> - velero: config: resourceGroup: <azure_resource_group> subscriptionId: <azure_subscription_id> incremental: "true" name: default provider: azure
<.> The
openshift
plug-in is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster. <.> Set tofalse
if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node hasRestic
pods running. You configure Restic for backups by addingspec.defaultVolumesToRestic: true
to theBackup
CR. <.> Specify the Azure resource group. <.> Specify the Azure storage account ID. <.> Specify the Azure subscription ID. <.> If you do not specify this value, the default name,cloud-credentials-azure
, is used. If you specify a custom name, the custom name is used for the backup location. <.> Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix. <.> Specify a prefix for Velero backups, for example,velero
, if the bucket is used for multiple purposes. <.> You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.- Click Create.
Verify the installation by viewing the OADP resources:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/oadp-velero-sample-1-aws-registry-5d6968cbdd-d5w9k 1/1 Running 0 95s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/oadp-velero-sample-1-aws-registry-svc ClusterIP 172.30.130.230 <none> 5000/TCP 95s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/oadp-velero-sample-1-aws-registry 1/1 1 1 96s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/oadp-velero-sample-1-aws-registry-5d6968cbdd 1 1 1 96s replicaset.apps/velero-588db7f655 1 1 1 96s
4.2.3.5.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1 featureFlags: - EnableCSI 2
4.2.4. Installing and configuring the OpenShift API for Data Protection with Google Cloud Platform
You install the OpenShift API for Data Protection (OADP) with Google Cloud Platform (GCP) by installing the OADP Operator, configuring GCP for Velero, and then installing the Data Protection Application.
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.
4.2.4.1. Installing the OADP Operator
You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.6 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.7.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
-
In the OpenShift Container Platform web console, click Operators
OperatorHub. - Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. -
Click Operators
Installed Operators to verify the installation.
4.2.4.2. Configuring Google Cloud Platform
You can configure a Google Cloud Platform (GCP) storage bucket as a replication repository for the Migration Toolkit for Containers (MTC).
Prerequisites
- The GCP storage bucket must be accessible to the source and target clusters.
-
You must have
gsutil
installed. If you are using the snapshot copy method:
- The source and target clusters must be in the same region.
- The source and target clusters must have the same storage class.
- The storage class must be compatible with snapshots.
Procedure
Log in to
gsutil
:$ gsutil init
Example output
Welcome! This command will take you through the configuration of gcloud. Your current configuration has been set to: [default] To continue, you must login. Would you like to login (Y/n)?
Set the
BUCKET
variable:$ BUCKET=<bucket> 1
- 1
- Specify your bucket name.
Create a storage bucket:
$ gsutil mb gs://$BUCKET/
Set the
PROJECT_ID
variable to your active project:$ PROJECT_ID=`gcloud config get-value project`
Create a
velero
IAM service account:$ gcloud iam service-accounts create velero \ --display-name "Velero Storage"
Create the
SERVICE_ACCOUNT_EMAIL
variable:$ SERVICE_ACCOUNT_EMAIL=`gcloud iam service-accounts list \ --filter="displayName:Velero Storage" \ --format 'value(email)'`
Create the
ROLE_PERMISSIONS
variable:$ ROLE_PERMISSIONS=( compute.disks.get compute.disks.create compute.disks.createSnapshot compute.snapshots.get compute.snapshots.create compute.snapshots.useReadOnly compute.snapshots.delete compute.zones.get )
Create the
velero.server
custom role:$ gcloud iam roles create velero.server \ --project $PROJECT_ID \ --title "Velero Server" \ --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
Add IAM policy binding to the project:
$ gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \ --role projects/$PROJECT_ID/roles/velero.server
Update the IAM service account:
$ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}
Save the IAM service account keys to the
credentials-velero
file in the current directory:$ gcloud iam service-accounts keys create credentials-velero \ --iam-account $SERVICE_ACCOUNT_EMAIL
You use the
credentials-velero
file to create aSecret
object for GCP before you install the Data Protection Application.
4.2.4.3. Creating a secret for backup and snapshot locations
You create a Secret
object for the backup and snapshot locations if they use the same credentials.
The default name of the Secret
is cloud-credentials-gcp
.
Prerequisites
- Your object storage and cloud storage must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.2.4.3.1. Configuring secrets for different backup and snapshot location credentials
If your backup and snapshot locations use different credentials, you create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials-gcp
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: provider: gcp default: true credential: key: cloud name: <custom_secret> 1 objectStorage: bucket: <bucket_name> prefix: <prefix> snapshotLocations: - velero: provider: gcp default: true config: project: <project> snapshotLocation: us-west1
- 1
- Backup location
Secret
with custom name.
4.2.4.4. Configuring the Data Protection Application
You can configure Velero resource allocations and enable self-signed CA certificates.
4.2.4.4.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... configuration: velero: podConfig: resourceAllocations: limits: cpu: "1" 1 memory: 512Mi 2 requests: cpu: 500m 3 memory: 256Mi 4
4.2.4.4.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 ...
4.2.4.5. Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials-gcp
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with the default name,cloud-credentials-gcp
, for the snapshot location. ThisSecret
is not referenced in theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
-
Procedure
-
Click Operators
Installed Operators and select the OADP Operator. - Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - gcp - openshift <.> restic: enable: true <.> backupLocations: - velero: provider: gcp default: true credential: key: cloud name: cloud-credentials-gcp <.> objectStorage: bucket: <bucket_name> <.> prefix: <prefix> <.> snapshotLocations: <.> - velero: provider: gcp default: true config: project: <project> snapshotLocation: us-west1 <.>
<.> The
openshift
plug-in is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster. <.> Set tofalse
if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node hasRestic
pods running. You configure Restic for backups by addingspec.defaultVolumesToRestic: true
to theBackup
CR. <.> If you do not specify this value, the default name,cloud-credentials-gcp
, is used. If you specify a custom name, the custom name is used for the backup location. <.> Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix. <.> Specify a prefix for Velero backups, for example,velero
, if the bucket is used for multiple purposes. <.> You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs. <.> The snapshot location must be in the same region as the PVs.- Click Create.
Verify the installation by viewing the OADP resources:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/oadp-velero-sample-1-aws-registry-5d6968cbdd-d5w9k 1/1 Running 0 95s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/oadp-velero-sample-1-aws-registry-svc ClusterIP 172.30.130.230 <none> 5000/TCP 95s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/oadp-velero-sample-1-aws-registry 1/1 1 1 96s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/oadp-velero-sample-1-aws-registry-5d6968cbdd 1 1 1 96s replicaset.apps/velero-588db7f655 1 1 1 96s
4.2.4.5.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1 featureFlags: - EnableCSI 2
4.2.5. Installing and configuring the OpenShift API for Data Protection with Multicloud Object Gateway
You install the OpenShift API for Data Protection (OADP) with Multicloud Object Gateway (MCG) by installing the OADP Operator, creating a Secret
object, and then installing the Data Protection Application.
MCG is a component of OpenShift Container Storage (OCS). You configure MCG as a backup location in the DataProtectionApplication
custom resource (CR).
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
If your cloud provider has a native snapshot API, configure a snapshot location. If your cloud provider does not support snapshots or if your storage is NFS, you can create backups with Restic.
You do not need to specify a snapshot location in the DataProtectionApplication
CR for Restic or Container Storage Interface (CSI) snapshots.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.
4.2.5.1. Installing the OADP Operator
You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.6 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.7.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
-
In the OpenShift Container Platform web console, click Operators
OperatorHub. - Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. -
Click Operators
Installed Operators to verify the installation.
4.2.5.2. Configuring the Multicloud Object Gateway
You can configure the Multicloud Object Gateway (MCG) as a replication repository for the Migration Toolkit for Containers (MTC). MCG is a component of OpenShift Container Storage.
Procedure
- Deploy OpenShift Container Storage by using the appropriate OpenShift Container Storage deployment guide.
Obtain the S3 endpoint,
AWS_ACCESS_KEY_ID
, andAWS_SECRET_ACCESS_KEY
by running thedescribe
command on theNooBaa
custom resource.These values are required in order to add MCG as a replication repository to the MTC web console.
4.2.5.3. Creating a secret for backup and snapshot locations
You create a Secret
object for the backup and snapshot locations if they use the same credentials.
The default name of the Secret
is cloud-credentials
.
Prerequisites
- Your object storage and cloud storage must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.2.5.3.1. Configuring secrets for different backup and snapshot location credentials
If your backup and snapshot locations use different credentials, you create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - aws - openshift restic: enable: true backupLocations: - velero: config: profile: "default" region: minio s3Url: <url> insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" provider: aws default: true credential: key: cloud name: <custom_secret> 1 objectStorage: bucket: <bucket_name> prefix: <prefix>
- 1
- Backup location
Secret
with custom name.
4.2.5.4. Configuring the Data Protection Application
You can configure Velero resource allocations and enable self-signed CA certificates.
4.2.5.4.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... configuration: velero: podConfig: resourceAllocations: limits: cpu: "1" 1 memory: 512Mi 2 requests: cpu: 500m 3 memory: 256Mi 4
4.2.5.4.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 ...
4.2.5.5. Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with the default name,cloud-credentials
, for the snapshot location. ThisSecret
is not referenced in theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
-
Procedure
-
Click Operators
Installed Operators and select the OADP Operator. - Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - aws - openshift <.> restic: enable: true <.> backupLocations: - velero: config: profile: "default" region: minio s3Url: <url> <.> insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" provider: aws default: true credential: key: cloud name: cloud-credentials <.> objectStorage: bucket: <bucket_name> <.> prefix: <prefix> <.>
<.> The
openshift
plug-in is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster. <.> Set tofalse
if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node hasRestic
pods running. You configure Restic for backups by addingspec.defaultVolumesToRestic: true
to theBackup
CR. <.> Specify the URL of the S3 endpoint. <.> If you do not specify this value, the default name,cloud-credentials
, is used. If you specify a custom name, the custom name is used for the backup location. <.> Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix. <.> Specify a prefix for Velero backups, for example,velero
, if the bucket is used for multiple purposes.- Click Create.
Verify the installation by viewing the OADP resources:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/oadp-velero-sample-1-aws-registry-5d6968cbdd-d5w9k 1/1 Running 0 95s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/oadp-velero-sample-1-aws-registry-svc ClusterIP 172.30.130.230 <none> 5000/TCP 95s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/oadp-velero-sample-1-aws-registry 1/1 1 1 96s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/oadp-velero-sample-1-aws-registry-5d6968cbdd 1 1 1 96s replicaset.apps/velero-588db7f655 1 1 1 96s
4.2.5.5.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1 featureFlags: - EnableCSI 2
4.2.6. Installing and configuring the OpenShift API for Data Protection with OpenShift Container Storage
You install the OpenShift API for Data Protection (OADP) with OpenShift Container Storage (OCS) by installing the OADP Operator and configuring a backup location and a snapshot location. Then, you install the Data Protection Application.
You can configure Multicloud Object Gateway or any S3-compatible object storage as a backup location in the DataProtectionApplication
custom resource (CR).
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
If the cloud provider has a native snapshot API, you can configure cloud storage as a snapshot location in the DataProtectionApplication
CR. You do not need to specify a snapshot location for Restic or Container Storage Interface (CSI) snapshots.
To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.
4.2.6.1. Installing the OADP Operator
You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.6 by using Operator Lifecycle Manager (OLM).
The OADP Operator installs Velero 1.7.
Prerequisites
-
You must be logged in as a user with
cluster-admin
privileges.
Procedure
-
In the OpenShift Container Platform web console, click Operators
OperatorHub. - Use the Filter by keyword field to find the OADP Operator.
- Select the OADP Operator and click Install.
-
Click Install to install the Operator in the
openshift-adp
project. -
Click Operators
Installed Operators to verify the installation.
After you install the OADP Operator, you configure object storage as a backup location and cloud storage as a snapshot location, if the cloud provider supports a native snapshot API.
If the cloud provider does not support snapshots or if your storage is NFS, you can create backups with Restic. Restic does not require a snapshot location.
4.2.6.2. Creating a secret for backup and snapshot locations
You create a Secret
object for the backup and snapshot locations if they use the same credentials.
The default name of the Secret
is cloud-credentials
, unless you specify a default plug-in for the backup storage provider.
Prerequisites
- Your object storage and cloud storage must use the same credentials.
- You must configure object storage for Velero.
-
You must create a
credentials-velero
file for the object storage in the appropriate format.
Procedure
Create a
Secret
with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
The Secret
is referenced in the spec.backupLocations.credential
block of the DataProtectionApplication
CR when you install the Data Protection Application.
4.2.6.2.1. Configuring secrets for different backup and snapshot location credentials
If your backup and snapshot locations use different credentials, you create two Secret
objects:
-
Backup location
Secret
with a custom name. The custom name is specified in thespec.backupLocations
block of theDataProtectionApplication
custom resource (CR). -
Snapshot location
Secret
with the default name,cloud-credentials
. ThisSecret
is not specified in theDataProtectionApplication
CR.
Procedure
-
Create a
credentials-velero
file for the snapshot location in the appropriate format for your cloud provider. Create a
Secret
for the snapshot location with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
-
Create a
credentials-velero
file for the backup location in the appropriate format for your object storage. Create a
Secret
for the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
Add the
Secret
with the custom name to theDataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - csi - openshift featureFlags: - EnableCSI restic: enable: true backupLocations: - velero: provider: gcp default: true credential: key: cloud name: <custom_secret> 1 objectStorage: bucket: <bucket_name> prefix: <prefix>
- 1
- Backup location
Secret
with custom name.
4.2.6.3. Configuring the Data Protection Application
You can configure Velero resource allocations and enable self-signed CA certificates.
4.2.6.3.1. Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero
pod by editing the DataProtectionApplication
custom resource (CR) manifest.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocations
block of theDataProtectionApplication
CR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... configuration: velero: podConfig: resourceAllocations: limits: cpu: "1" 1 memory: 512Mi 2 requests: cpu: 500m 3 memory: 256Mi 4
4.2.6.3.2. Enabling self-signed CA certificates
You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication
custom resource (CR) manifest to prevent a certificate signed by unknown authority
error.
Prerequisites
- You must have the OpenShift API for Data Protection (OADP) Operator installed.
Procedure
Edit the
spec.backupLocations.velero.objectStorage.caCert
parameter andspec.backupLocations.velero.config
parameters of theDataProtectionApplication
CR manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: ... backupLocations: - name: default velero: provider: aws default: true objectStorage: bucket: <bucket> prefix: <prefix> caCert: <base64_encoded_cert_string> 1 config: insecureSkipTLSVerify: "false" 2 ...
4.2.6.4. Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication
API.
Prerequisites
- You must install the OADP Operator.
- You must configure object storage as a backup location.
- If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secret
with the default name,cloud-credentials
. If the backup and snapshot locations use different credentials, you must create two
Secrets
:-
Secret
with a custom name for the backup location. You add thisSecret
to theDataProtectionApplication
CR. Secret
with the default name,cloud-credentials
, for the snapshot location. ThisSecret
is not referenced in theDataProtectionApplication
CR.NoteIf you do not want to specify backup or snapshot locations during the installation, you can create a default
Secret
with an emptycredentials-velero
file. If there is no defaultSecret
, the installation will fail.
-
Procedure
-
Click Operators
Installed Operators and select the OADP Operator. - Under Provided APIs, click Create instance in the DataProtectionApplication box.
Click YAML View and update the parameters of the
DataProtectionApplication
manifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - gcp <.> - csi <.> - openshift <.> restic: enable: true <.> backupLocations: - velero: provider: gcp <.> default: true credential: key: cloud name: <default_secret> <.> objectStorage: bucket: <bucket_name> <.> prefix: <prefix> <.>
<.> Specify the default plug-in for the backup provider, for example,
gcp
, if appropriate. <.> Specify thecsi
default plug-in if you use CSI snapshots to back up PVs. Thecsi
plug-in uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location. <.> Theopenshift
plug-in is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster. <.> Set tofalse
if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node hasRestic
pods running. You configure Restic for backups by addingspec.defaultVolumesToRestic: true
to theBackup
CR. <.> Specify the backup provider. <.> If you use a default plug-in for the backup provider, you must specify the correct default name for theSecret
, for example,cloud-credentials-gcp
. If you specify a custom name, the custom name is used for the backup location. If you do not specify aSecret
name, the default name is used. <.> Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix. <.> Specify a prefix for Velero backups, for example,velero
, if the bucket is used for multiple purposes.- Click Create.
Verify the installation by viewing the OADP resources:
$ oc get all -n openshift-adp
Example output
NAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/oadp-velero-sample-1-aws-registry-5d6968cbdd-d5w9k 1/1 Running 0 95s pod/restic-9cq4q 1/1 Running 0 94s pod/restic-m4lts 1/1 Running 0 94s pod/restic-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/oadp-velero-sample-1-aws-registry-svc ClusterIP 172.30.130.230 <none> 5000/TCP 95s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/restic 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/oadp-velero-sample-1-aws-registry 1/1 1 1 96s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/oadp-velero-sample-1-aws-registry-5d6968cbdd 1 1 1 96s replicaset.apps/velero-588db7f655 1 1 1 96s
4.2.6.4.1. Enabling CSI in the DataProtectionApplication CR
You enable the Container Storage Interface (CSI) in the DataProtectionApplication
custom resource (CR) in order to back up persistent volumes with CSI snapshots.
Prerequisites
- The cloud provider must support CSI snapshots.
Procedure
Edit the
DataProtectionApplication
CR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication ... spec: configuration: velero: defaultPlugins: - openshift - csi 1 featureFlags: - EnableCSI 2
4.2.7. Uninstalling the OpenShift API for Data Protection
You uninstall the OpenShift API for Data Protection (OADP) by deleting the OADP Operator. See Deleting Operators from a cluster for details.
4.3. Backing up and restoring
4.3.1. Backing up applications
You back up applications by creating a Backup
custom resource (CR).
The Backup
CR creates backup files for Kubernetes resources and internal images, on S3 object storage, and snapshots for persistent volumes (PVs), if the cloud provider uses a native snapshot API or the Container Storage Interface (CSI) to create snapshots, such as OpenShift Container Storage 4. For more information, see CSI volume snapshots.
The CloudStorage
API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
If your cloud provider has a native snapshot API or supports Container Storage Interface (CSI) snapshots, the Backup
CR backs up persistent volumes by creating snapshots. For more information, see the Overview of CSI volume snapshots in the OpenShift Container Platform documentation.
If your cloud provider does not support snapshots or if your applications are on NFS data volumes, you can create backups by using Restic.
You can create backup hooks to run commands before or after the backup operation.
You can schedule backups by creating a Schedule
CR instead of a Backup
CR.
4.3.1.1. Creating a Backup CR
You back up Kubernetes images, internal images, and persistent volumes (PVs) by creating a Backup
custom resource (CR).
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state. Backup location prerequisites:
- You must have S3 object storage configured for Velero.
-
You must have a backup location configured in the
DataProtectionApplication
CR.
Snapshot location prerequisites:
- Your cloud provider must have a native snapshot API or support Container Storage Interface (CSI) snapshots.
-
For CSI snapshots, you must create a
VolumeSnapshotClass
CR to register the CSI driver. -
You must have a volume location configured in the
DataProtectionApplication
CR.
Procedure
Retrieve the
backupStorageLocations
CRs:$ oc get backupStorageLocations
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT velero-sample-1 Available 11s 31m
Create a
Backup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> labels: velero.io/storage-location: default namespace: openshift-adp spec: hooks: {} includedNamespaces: - <namespace> 1 storageLocation: <velero-sample-1> 2 ttl: 720h0m0s
Verify that the status of the
Backup
CR isCompleted
:$ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'
4.3.1.2. Backing up persistent volumes with CSI snapshots
You back up persistent volumes with Container Storage Interface (CSI) snapshots by creating a VolumeSnapshotClass
custom resource (CR) to register the CSI driver before you create the Backup
CR.
Prerequisites
- The cloud provider must support CSI snapshots.
-
You must enable CSI in the
DataProtectionApplication
CR.
Procedure
Create a
VolumeSnapshotClass
CR, as in the following examples:Ceph RBD
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass deletionPolicy: Retain metadata: name: <volume_snapshot_class_name> labels: velero.io/csi-volumesnapshot-class: "true" snapshotter: openshift-storage.rbd.csi.ceph.com driver: openshift-storage.rbd.csi.ceph.com parameters: clusterID: openshift-storage csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
Ceph FS
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: <volume_snapshot_class_name> labels: velero.io/csi-volumesnapshot-class: "true" driver: openshift-storage.cephfs.csi.ceph.com deletionPolicy: Retain parameters: clusterID: openshift-storage csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
Other cloud providers
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: <volume_snapshot_class_name> labels: velero.io/csi-volumesnapshot-class: "true" driver: <csi_driver> deletionPolicy: Retain
You can now create a Backup
CR.
4.3.1.3. Backing up applications with Restic
You back up Kubernetes resources, internal images, and persistent volumes with Restic by editing the Backup
custom resource (CR).
You do not need to specify a snapshot location in the DataProtectionApplication
CR.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
You must not disable the default Restic installation by setting
spec.configuration.restic.enable
tofalse
in theDataProtectionApplication
CR. -
The
DataProtectionApplication
CR must be in aReady
state.
Procedure
Edit the
Backup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> labels: velero.io/storage-location: default namespace: openshift-adp spec: defaultVolumesToRestic: true 1 ...
- 1
- Add
defaultVolumesToRestic: true
to thespec
block.
4.3.1.4. Creating backup hooks
You create backup hooks to run commands in a container in a pod by editing the Backup
custom resource (CR).
Pre hooks run before the pod is backed up. Post hooks run after the backup.
Procedure
Add a hook to the
spec.hooks
block of theBackup
CR, as in the following example:apiVersion: velero.io/v1 kind: Backup metadata: name: <backup> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> 1 excludedNamespaces: - <namespace> includedResources: - pods 2 excludedResources: [] labelSelector: 3 matchLabels: app: velero component: server pre: 4 - exec: container: <container> 5 command: - /bin/uname 6 - -a onError: Fail 7 timeout: 30s 8 post: 9 ...
- 1
- Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Currently, pods are the only supported resource.
- 3
- Optional: This hook only applies to objects matching the label selector.
- 4
- Array of hooks to run before the backup.
- 5
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 6
- Array of commands that the hook runs.
- 7
- Allowed values for error handling are
Fail
andContinue
. The default isFail
. - 8
- Optional: How long to wait for the commands to run. The default is
30s
. - 9
- This block defines an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.
4.3.1.5. Scheduling backups
You schedule backups by creating a Schedule
custom resource (CR) instead of a Backup
CR.
Leave enough time in your backup schedule for a backup to finish before another backup is created.
For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state.
Procedure
Retrieve the
backupStorageLocations
CRs:$ oc get backupStorageLocations
Example output
NAME PHASE LAST VALIDATED AGE DEFAULT velero-sample-1 Available 11s 31m
Create a
Schedule
CR, as in the following example:$ cat << EOF | oc apply -f - apiVersion: velero.io/v1 kind: Schedule metadata: name: <schedule> namespace: openshift-adp spec: schedule: 0 7 * * * 1 template: hooks: {} includedNamespaces: - <namespace> 2 storageLocation: <velero-sample-1> 3 defaultVolumesToRestic: true 4 ttl: 720h0m0s EOF
Verify that the status of the
Schedule
CR isCompleted
after the scheduled backup runs:$ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'
4.3.2. Restoring applications
You restore application backups by creating a Restore
custom resources (CRs).
You can create restore hooks to run commands in init containers, before the application container starts, or in the application container itself.
4.3.2.1. Creating a Restore CR
You restore a Backup
custom resource (CR) by creating a Restore
CR.
Prerequisites
- You must install the OpenShift API for Data Protection (OADP) Operator.
-
The
DataProtectionApplication
CR must be in aReady
state. -
You must have a Velero
Backup
CR. - Adjust the requested size so the persistent volume (PV) capacity matches the requested size at backup time.
Procedure
Create a
Restore
CR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: backupName: <backup> 1 excludedResources: - nodes - events - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io restorePVs: true
- 1
- Name of the
Backup
CR.
Verify that the status of the
Restore
CR isCompleted
:$ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
Verify that the backup resources have been restored:
$ oc get all -n <namespace> 1
- 1
- Namespace that you backed up.
4.3.2.2. Creating restore hooks
You create restore hooks to run commands in a container in a pod while restoring your application by editing the Restore
custom resource (CR).
You can create two types of restore hooks:
An
init
hook adds an init container to a pod to perform setup tasks before the application container starts.If you restore a Restic backup, the
restic-wait
init container is added before the restore hook init container.-
An
exec
hook runs commands or scripts in a container of a restored pod.
Procedure
Add a hook to the
spec.hooks
block of theRestore
CR, as in the following example:apiVersion: velero.io/v1 kind: Restore metadata: name: <restore> namespace: openshift-adp spec: hooks: resources: - name: <hook_name> includedNamespaces: - <namespace> 1 excludedNamespaces: - <namespace> includedResources: - pods 2 excludedResources: [] labelSelector: 3 matchLabels: app: velero component: server postHooks: - init: initContainers: - name: restore-hook-init image: alpine:latest volumeMounts: - mountPath: /restores/pvc1-vm name: pvc1-vm command: - /bin/ash - -c - exec: container: <container> 4 command: - /bin/bash 5 - -c - "psql < /backup/backup.sql" waitTimeout: 5m 6 execTimeout: 1m 7 onError: Continue 8
- 1
- Optional: Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
- 2
- Currently, pods are the only supported resource.
- 3
- Optional: This hook only applies to objects matching the label selector.
- 4
- Optional: If the container is not specified, the command runs in the first container in the pod.
- 5
- Array of commands that the hook runs.
- 6
- Optional: If the
waitTimeout
is not specified, the restore waits indefinitely. You can specify how long to wait for a container to start and for preceding hooks in the container to complete. The wait timeout starts when the container is restored and might require time for the container to pull the image and mount the volumes. - 7
- Optional: How long to wait for the commands to run. The default is
30s
. - 8
- Allowed values for error handling are
Fail
andContinue
:-
Continue
: Only command failures are logged. -
Fail
: No more restore hooks run in any container in any pod. The status of theRestore
CR will bePartiallyFailed
.
-
4.4. Troubleshooting
You can debug Velero custom resources (CRs) by using the OpenShift CLI tool or the Velero CLI tool. The Velero CLI tool provides more detailed logs and information.
You can check installation issues, backup and restore CR issues, and Restic issues.
You can collect logs, CR information, and Prometheus metric data by using the must-gather
tool.
You can obtain the Velero CLI tool by:
- Downloading the Velero CLI tool
- Accessing the Velero binary in the Velero deployment in the cluster
4.4.1. Downloading the Velero CLI tool
You can download and install the Velero CLI tool by following the instructions on the Velero documentation page.
The page includes instructions for:
- macOS by using Homebrew
- GitHub
- Windows by using Chocolatey
Prerequisites
- You have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
-
You have installed
kubectl
locally.
Procedure
- Open a browser and navigate to "Install the CLI" on the Verleo website.
- Follow the appropriate procedure for macOS, GitHub, or Windows.
Download the Velero version appropriate for your version of OADP, according to the table that follows:
Table 4.2. OADP-Velero version relationship OADP version Velero version 0.2.6
1.6.0
0.5.5
1.7.1
1.0.0
1.7.1
1.0.1
1.7.1
1.0.2
1.7.1
1.0.3
1.7.1
4.4.2. Accessing the Velero binary in the Velero deployment in the cluster
You can use a shell command to access the Velero binary in the Velero deployment in the cluster.
Prerequisites
-
Your
DataProtectionApplication
custom resource has a status ofReconcile complete
.
Procedure
Enter the following command to set the needed alias:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
4.4.3. Debugging Velero resources with the OpenShift CLI tool
You can debug a failed backup or restore by checking Velero custom resources (CRs) and the Velero
pod log with the OpenShift CLI tool.
Velero CRs
Use the oc describe
command to retrieve a summary of warnings and errors associated with a Backup
or Restore
CR:
$ oc describe <velero_cr> <cr_name>
Velero pod logs
Use the oc logs
command to retrieve the Velero
pod logs:
$ oc logs pod/<velero>
Velero pod debug logs
You can specify the Velero log level in the DataProtectionApplication
resource as shown in the following example.
This option is available starting from OADP 1.0.3.
apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: velero-sample spec: configuration: velero: logLevel: warning
The following logLevel
values are available:
-
trace
-
debug
-
info
-
warning
-
error
-
fatal
-
panic
It is recommended to use debug
for most logs.
4.4.4. Debugging Velero resources with the Velero CLI tool
You can debug Backup
and Restore
custom resources (CRs) and retrieve logs with the Velero CLI tool.
The Velero CLI tool provides more detailed information than the OpenShift CLI tool.
Syntax
Use the oc exec
command to run a Velero CLI command:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ <backup_restore_cr> <command> <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Help option
Use the velero --help
option to list all Velero CLI commands:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ --help
Describe command
Use the velero describe
command to retrieve a summary of warnings and errors associated with a Backup
or Restore
CR:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ <backup_restore_cr> describe <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Logs command
Use the velero logs
command to retrieve the logs of a Backup
or Restore
CR:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ <backup_restore_cr> logs <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \ restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
4.4.5. Installation issues
You might encounter issues caused by using invalid directories or incorrect credentials when you install the Data Protection Application.
4.4.5.1. Backup storage contains invalid directories
The Velero
pod log displays the error message, Backup storage contains invalid top-level directories
.
Cause
The object storage contains top-level directories that are not Velero directories.
Solution
If the object storage is not dedicated to Velero, you must specify a prefix for the bucket by setting the spec.backupLocations.velero.objectStorage.prefix
parameter in the DataProtectionApplication
manifest.
4.4.5.2. Incorrect AWS credentials
The oadp-aws-registry
pod log displays the error message, InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
The Velero
pod log displays the error message, NoCredentialProviders: no valid providers in chain
.
Cause
The credentials-velero
file used to create the Secret
object is incorrectly formatted.
Solution
Ensure that the credentials-velero
file is correctly formatted, as in the following example:
Example credentials-velero
file
[default] 1 aws_access_key_id=AKIAIOSFODNN7EXAMPLE 2 aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
4.4.6. Backup and Restore CR issues
You might encounter these common issues with Backup
and Restore
custom resources (CRs).
4.4.6.1. Backup CR cannot retrieve volume
The Backup
CR displays the error message, InvalidVolume.NotFound: The volume ‘vol-xxxx’ does not exist
.
Cause
The persistent volume (PV) and the snapshot locations are in different regions.
Solution
-
Edit the value of the
spec.snapshotLocations.velero.config.region
key in theDataProtectionApplication
manifest so that the snapshot location is in the same region as the PV. -
Create a new
Backup
CR.
4.4.6.2. Backup CR status remains in progress
The status of a Backup
CR remains in the InProgress
phase and does not complete.
Cause
If a backup is interrupted, it cannot be resumed.
Solution
Retrieve the details of the
Backup
CR:$ oc -n {namespace} exec deployment/velero -c velero -- ./velero \ backup describe <backup>
Delete the
Backup
CR:$ oc delete backup <backup> -n openshift-adp
You do not need to clean up the backup location because a
Backup
CR in progress has not uploaded files to object storage.-
Create a new
Backup
CR.
4.4.7. Restic issues
You might encounter these issues when you back up applications with Restic.
4.4.7.1. Restic permission error for NFS data volumes with root_squash enabled
The Restic
pod log displays the error message, controller=pod-volume-backup error="fork/exec/usr/bin/restic: permission denied"
.
Cause
If your NFS data volumes have root_squash
enabled, Restic
maps to nfsnobody
and does not have permission to create backups.
Solution
You can resolve this issue by creating a supplemental group for Restic
and adding the group ID to the DataProtectionApplication
manifest:
-
Create a supplemental group for
Restic
on the NFS data volume. -
Set the
setgid
bit on the NFS directories so that group ownership is inherited. Add the
spec.configuration.restic.supplementalGroups
parameter and the group ID to theDataProtectionApplication
manifest, as in the following example:spec: configuration: restic: enable: true supplementalGroups: - <group_id> 1
- 1
- Specify the supplemental group ID.
-
Wait for the
Restic
pods to restart so that the changes are applied.
4.4.7.2. Restore CR of Restic backup is "PartiallyFailed", "Failed", or remains "InProgress"
The Restore
CR of a Restic backup completes with a PartiallyFailed
or Failed
status or it remains InProgress
and does not complete.
If the status is PartiallyFailed
or Failed
, the Velero
pod log displays the error message, level=error msg="unable to successfully complete restic restores of pod’s volumes"
.
If the status is InProgress
, the Restore
CR logs are unavailable and no errors appear in the Restic
pod logs.
Cause
The DeploymentConfig
object redeploys the Restore
pod, causing the Restore
CR to fail.
Solution
Create a
Restore
CR that excludes theReplicationController
,DeploymentConfig
, andTemplateInstances
resources:$ velero restore create --from-backup=<backup> -n openshift-adp \ 1 --include-namespaces <namespace> \ 2 --exclude-resources replicationcontroller,deploymentconfig,templateinstances.template.openshift.io \ --restore-volumes=true
Verify that the status of the
Restore
CR isCompleted
:$ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
Create a
Restore
CR that includes theReplicationController
andDeploymentConfig
resources:$ velero restore create --from-backup=<backup> -n openshift-adp \ --include-namespaces <namespace> \ --include-resources replicationcontroller,deploymentconfig \ --restore-volumes=true
Verify that the status of the
Restore
CR isCompleted
:$ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
Verify that the backup resources have been restored:
$ oc get all -n <namespace>
4.4.7.3. Restic Backup CR cannot be recreated after bucket is emptied
If you create a Restic Backup
CR for a namespace, empty the S3 bucket, and then recreate the Backup
CR for the same namespace, the recreated Backup
CR fails.
The velero
pod log displays the error message, msg="Error checking repository for stale locks"
.
Cause
Velero does not create the Restic repository from the ResticRepository
manifest if the Restic directories are deleted on object storage. See (Velero issue 4421) for details.
4.4.8. Using the must-gather tool
You can collect logs, metrics, and information about OADP custom resources by using the must-gather
tool.
The must-gather
data must be attached to all customer cases.
You can run the must-gather
tool with the following data collection options:
-
Full
must-gather
data collection collects Prometheus metrics, pod logs, and Velero CR information for all namespaces where the OADP Operator is installed. -
Essential
must-gather
data collection collects pod logs and Velero CR information for a specific duration of time, for example, one hour or 24 hours. Prometheus metrics and duplicate logs are not included. -
must-gather
data collection with timeout. Data collection can take a long time if there are many failedBackup
CRs. You can improve performance by setting a timeout value. - Prometheus metrics data dump downloads an archive file containing the metrics data collected by Prometheus.
Prerequisites
-
You must be logged in to the OpenShift Container Platform cluster as a user with the
cluster-admin
role. - You must have the OpenShift CLI installed.
Procedure
-
Navigate to the directory where you want to store the
must-gather
data. Run the
oc adm must-gather
command for one of the following data collection options:Full
must-gather
data collection, including Prometheus metrics:$ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0
The data is saved as
must-gather/must-gather.tar.gz
. You can upload this file to a support case on the Red Hat Customer Portal.Essential
must-gather
data collection, without Prometheus metrics, for a specific time duration:$ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \ -- /usr/bin/gather_<time>_essential 1
- 1
- Specify the time in hours. Allowed values are
1h
,6h
,24h
,72h
, orall
, for example,gather_1h_essential
orgather_all_essential
.
must-gather
data collection with timeout:$ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \ -- /usr/bin/gather_with_timeout <timeout> 1
- 1
- Specify a timeout value in seconds.
Prometheus metrics data dump:
$ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \ -- /usr/bin/gather_metrics_dump
This operation can take a long time. The data is saved as
must-gather/metrics/prom_data.tar.gz
.
Viewing metrics data with the Prometheus console
You can view the metrics data with the Prometheus console.
Procedure
Decompress the
prom_data.tar.gz
file:$ tar -xvzf must-gather/metrics/prom_data.tar.gz
Create a local Prometheus instance:
$ make prometheus-run
The command outputs the Prometheus URL.
Output
Started Prometheus on http://localhost:9090
- Launch a web browser and navigate to the URL to view the data by using the Prometheus web console.
After you have viewed the data, delete the Prometheus instance and data:
$ make prometheus-cleanup