Red Hat Summit Red Hat SummitSupportKonsoleEntwicklerDemo startenKontakt

Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.

Chapter 4. Application backup and restore

4.1. OADP release notes
Link kopieren

The release notes for OpenShift API for Data Protection (OADP) describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.

4.1.1. OADP 1.1.2 release notes
Link kopieren

The OADP 1.1.2 release notes include product recommendations, a list of fixed bugs and descriptions of known issues.

4.1.1.1. Product recommendations
Link kopieren

VolSync

To prepare for the upgrade from VolSync 0.5.1 to the latest version available from the VolSync stable channel, you must add this annotation in the openshift-adp namespace by running the following command: 

$ oc annotate --overwrite namespace/openshift-adp volsync.backube/privileged-movers='true'
Copy to Clipboard Toggle word wrap

Velero

In this release, Velero has been upgraded from version 1.9.2 to version 1.9.5.

Restic

In this release, Restic has been upgraded from version 0.13.1 to version 0.14.0.

4.1.1.2. Fixed bugs
Link kopieren

The following bugs have been fixed in this release:

4.1.1.3. Known issues
Link kopieren

This release has the following known issues:

  • OADP currently does not support backup and restore of AWS EFS volumes using restic in Velero (OADP-778).

  • CSI backups might fail due to a Ceph limitation of VolumeSnapshotContent snapshots per PVC.

    You can create many snapshots of the same persistent volume claim (PVC) but cannot schedule periodic creation of snapshots:

    • For CephFS, you can create up to 100 snapshots per PVC. (OADP-804)
    • For RADOS Block Device (RBD), you can create up to 512 snapshots for each PVC. (OADP-975)

    For more information, see Volume Snapshots.

4.1.2. OADP 1.1.1 release notes
Link kopieren

The OADP 1.1.1 release notes include product recommendations and descriptions of known issues.

4.1.2.1. Product recommendations
Link kopieren

Before you install OADP 1.1.1, it is recommended to either install VolSync 0.5.1 or to upgrade to it.

4.1.2.2. Known issues
Link kopieren

This release has the following known issues:

  • OADP currently does not support backup and restore of AWS EFS volumes using restic in Velero (OADP-778).

  • CSI backups might fail due to a Ceph limitation of VolumeSnapshotContent snapshots per PVC.

    You can create many snapshots of the same persistent volume claim (PVC) but cannot schedule periodic creation of snapshots:

    • For CephFS, you can create up to 100 snapshots per PVC.

    • For RADOS Block Device (RBD), you can create up to 512 snapshots for each PVC. (OADP-804) and (OADP-975)

      For more information, see Volume Snapshots.

4.2. OADP features and plugins
Link kopieren

OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications.

The default plugins enable Velero to integrate with certain cloud providers and to back up and restore OpenShift Container Platform resources.

4.2.1. OADP features
Link kopieren

OpenShift API for Data Protection (OADP) supports the following features:

Backup

You can back up all resources in your cluster or you can filter the resources by type, namespace, or label.

OADP backs up Kubernetes objects and internal images by saving them as an archive file on object storage. OADP backs up persistent volumes (PVs) by creating snapshots with the native cloud snapshot API or with the Container Storage Interface (CSI). For cloud providers that do not support snapshots, OADP backs up resources and PV data with Restic.

Restore
You can restore resources and PVs from a backup. You can restore all objects in a backup or filter the restored objects by namespace, PV, or label.
Schedule
You can schedule backups at specified intervals.
Hooks
You can use hooks to run commands in a container on a pod, for example, fsfreeze to freeze a file system. You can configure a hook to run before or after a backup or restore. Restore hooks can run in an init container or in the application container.

4.2.2. OADP plugins
Link kopieren

The OpenShift API for Data Protection (OADP) provides default Velero plugins that are integrated with storage providers to support backup and snapshot operations. You can create custom plugins based on the Velero plugins.

OADP also provides plugins for OpenShift Container Platform resource backups and Container Storage Interface (CSI) snapshots.

Expand
Table 4.1. OADP plugins
OADP pluginFunctionStorage location

aws

Backs up and restores Kubernetes objects by using object store.

AWS S3

Backs up and restores volumes by using snapshots.

AWS EBS

azure

Backs up and restores Kubernetes objects by using object store.

Microsoft Azure Blob storage

Backs up and restores volumes by using snapshots.

Microsoft Azure Managed Disks

gcp

Backs up and restores Kubernetes objects by using object store.

Google Cloud Storage

Backs up and restores volumes by using snapshots.

Google Compute Engine Disks

openshift

Backs up and restores OpenShift Container Platform resources by using object store. [1]

Object store

csi

Backs up and restores volumes by using CSI snapshots. [2]

Cloud storage that supports CSI snapshots

  1. Mandatory.
  2. The csi plugin uses the Velero CSI beta snapshot API.

4.2.3. About OADP Velero plugins
Link kopieren

You can configure two types of plugins when you install Velero:

  • Default cloud provider plugins
  • Custom plugins

Both types of plugin are optional, but most users configure at least one cloud provider plugin.

4.2.3.1. Default Velero cloud provider plugins
Link kopieren

You can install any of the following default Velero cloud provider plugins when you configure the oadp_v1alpha1_dpa.yaml file during deployment:

  • aws (Amazon Web Services)
  • gcp (Google Cloud Platform)
  • azure (Microsoft Azure)
  • openshift (OpenShift Velero plugin)
  • csi (Container Storage Interface)
  • kubevirt (KubeVirt)

You specify the desired default plugins in the oadp_v1alpha1_dpa.yaml file during deployment.

Example file

The following .yaml file installs the openshift, aws, azure, and gcp plugins: 

 apiVersion: oadp.openshift.io/v1alpha1
 kind: DataProtectionApplication
 metadata:
   name: dpa-sample
 spec:
   configuration:
     velero:
       defaultPlugins:
       - openshift
       - aws
       - azure
       - gcp
Copy to Clipboard Toggle word wrap

4.2.3.2. Custom Velero plugins
Link kopieren

You can install a custom Velero plugin by specifying the plugin image and name when you configure the oadp_v1alpha1_dpa.yaml file during deployment.

You specify the desired custom plugins in the oadp_v1alpha1_dpa.yaml file during deployment.

Example file

The following .yaml file installs the default openshift, azure, and gcp plugins and a custom plugin that has the name custom-plugin-example and the image quay.io/example-repo/custom-velero-plugin: 

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-sample
spec:
 configuration:
   velero:
     defaultPlugins:
     - openshift
     - azure
     - gcp
     customPlugins:
     - name: custom-plugin-example
       image: quay.io/example-repo/custom-velero-plugin
Copy to Clipboard Toggle word wrap

4.3. Installing and configuring OADP
Link kopieren

4.3.1. About installing OADP
Link kopieren

As a cluster administrator, you install the OpenShift API for Data Protection (OADP) by installing the OADP Operator. The OADP Operator installs Velero 1.7.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

To back up Kubernetes resources and internal images, you must have object storage as a backup location, such as one of the following storage types:

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can back up persistent volumes (PVs) by using snapshots or Restic.

To back up PVs with snapshots, you must have a cloud provider that supports either a native snapshot API or Container Storage Interface (CSI) snapshots, such as one of the following cloud providers:

If your cloud provider does not support snapshots or if your storage is NFS, you can back up applications with Restic.

You create a Secret object for your storage provider credentials and then you install the Data Protection Application.

If you use cluster storage for your NooBaa bucket backupStorageLocation on OpenShift Container Storage, configure NooBaa as an external object store.

Warning

Failure to configure NooBaa as an external object store might lead to backups not being available.

Procedure

/// Module included in the following assemblies:

4.3.1.2. About OADP update channels
Link kopieren

When you install an OADP Operator, you choose an update channel. This channel determines which upgrades to the OADP Operator and to Velero you receive. You can switch channels at any time.

There are three update channels:

  • The stable channel contains the latest minor updates (y-stream updates) and patches (z-stream updates) of OADP ClusterServiceVersion`. As each new release is published, the available ClusterServiceVersion of the OADP Operator will be appended with the latest available minor patch.
  • The stable-1.0 channel contains oadp.v1.0.z, the most recent OADP 1.0 ClusterServiceVersion.
  • The stable-1.1 channel contains oadp.v1.1.z, the most recent OADP 1.1 ClusterServiceVersion.

Which update channel is right for you?

  • Choose the stable update channel to install the latest stable OADP version and to receive both minor updates and patches. If you choose this channel, you will receive all y-stream and all z-stream updates for version x.y.z.
  • Choose the stable-1.y update channel to install OADP 1.y and to continue receiving patches for it. If you choose this channel, you will receive all z-stream patches for version 1.y.z.

When must you switch update channels?

  • If you have OADP 1.y installed and you want to receive patches only for that y-stream, you must switch from the stable update channel to the stable-1.y update channel. You will then receive all z-stream patches for version 1.y.z.
  • If you have OADP 1.0 installed, want to upgrade to OADP 1.1, and then receive patches only for OADP 1.1, you must switch from the stable-1.0 update channel to the stable-1.1 update channel. You will then receive all z-stream patches for version 1.1.z.
  • If you have OADP 1.y installed, with y greater than 0, and want to switch to OADP 1.0, you must uninstall your OADP Operator and then reinstall it using the stable-1.0 update channel. You will then receive all z-stream patches for version 1.0.z.
Note

You cannot switch from OADP 1.y to OADP 1.0 by switching update channels. You must uninstall the Operator and then reinstall it.

You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) by installing the OADP Operator, configuring AWS for Velero, and then installing the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.

4.3.2.1. Installing the OADP Operator
Link kopieren

You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.9 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.7.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click Operators OperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click Operators Installed Operators to verify the installation.

4.3.2.2. Configuring Amazon Web Services
Link kopieren

You configure Amazon Web Services (AWS) for the OpenShift API for Data Protection (OADP).

Prerequisites

  • You must have the AWS CLI installed.

Procedure

  1. Set the BUCKET variable: 

    $ BUCKET=<your_bucket>
    Copy to Clipboard Toggle word wrap

  2. Set the REGION variable: 

    $ REGION=<your_region>
    Copy to Clipboard Toggle word wrap

  3. Create an AWS S3 bucket: 

    $ aws s3api create-bucket \
    --bucket $BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION
    1
    Copy to Clipboard Toggle word wrap
    1
    us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

  4. Create an IAM user: 

    $ aws iam create-user --user-name velero
    1
    Copy to Clipboard Toggle word wrap
    1
    If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

  5. Create a velero-policy.json file: 

    $ cat > velero-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF
    Copy to Clipboard Toggle word wrap

  6. Attach the policies to give the velero user the minimum necessary permissions: 

    $ aws iam put-user-policy \
  --user-name velero \
  --policy-name velero \
  --policy-document file://velero-policy.json
    Copy to Clipboard Toggle word wrap

  7. Create an access key for the velero user: 

    $ aws iam create-access-key --user-name velero
    Copy to Clipboard Toggle word wrap

    Example output

    {
  "AccessKey": {
        "UserName": "velero",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}
    Copy to Clipboard Toggle word wrap

  8. Create a credentials-velero file: 

    $ cat << EOF > ./credentials-velero
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF
    Copy to Clipboard Toggle word wrap

    You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

4.3.2.3. Creating a secret for backup and snapshot locations
Link kopieren

You create a Secret object for the backup and snapshot locations if they use the same credentials.

The default name of the Secret is cloud-credentials.

Prerequisites

  • Your object storage and cloud storage must use the same credentials.
  • You must configure object storage for Velero.

  • You must create a credentials-velero file for the object storage in the appropriate format.

    Note

    The DataProtectionApplication custom resource (CR) requires a Secret for installation. If no spec.backupLocations.credential.name value is specified, the default name is used.

    If you do not want to specify the backup locations or the snapshot locations, you must create a Secret with the default name by using an empty credentials-velero file.

Procedure

  • Create a Secret with the default name: 

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

If your backup and snapshot locations use different credentials, you create separate profiles in the credentials-velero file.

Then, you create a Secret object and specify the profiles in the DataProtectionApplication custom resource (CR).

Procedure

  1. Create a credentials-velero file with separate profiles for the backup and snapshot locations, as in the following example: 

    [backupStorage]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

[volumeSnapshot]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    Copy to Clipboard Toggle word wrap

  2. Create a Secret object with the credentials-velero file: 

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
    1
    Copy to Clipboard Toggle word wrap

  3. Add the profiles to the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <prefix>
        config:
          region: us-east-1
          profile: "backupStorage"
        credential:
          key: cloud
          name: cloud-credentials
  snapshotLocations:
    - name: default
      velero:
        provider: aws
        config:
          region: us-west-2
          profile: "volumeSnapshot"
    Copy to Clipboard Toggle word wrap

4.3.2.4. Configuring the Data Protection Application
Link kopieren

You can configure Velero resource allocations and enable self-signed CA certificates.

4.3.2.4.1. Setting Velero CPU and memory resource allocations
Link kopieren

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node selector>
    1 
    
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
    Copy to Clipboard Toggle word wrap
    1 1
    Specify the node selector to be supplied to Velero podSpec
4.3.2.4.2. Enabling self-signed CA certificates
Link kopieren

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string>
    1 
    
        config:
          insecureSkipTLSVerify: "false"
    2 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Specify the Base46-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.

4.3.2.5. Installing the Data Protection Application
Link kopieren

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

  • If the backup and snapshot locations use different credentials, you must create a Secret with the default name, cloud-credentials, which contains separate profiles for the backup and snapshot location credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Operators Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

  3. Click YAML View and update the parameters of the DataProtectionApplication manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift
    1 
    
        - aws
    restic:
      enable: true
    2 
    
      podConfig:
        nodeSelector: <node selector>
    3 
    
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name>
    4 
    
          prefix: <prefix>
    5 
    
        config:
          region: <region>
          profile: "default"
        credential:
          key: cloud
          name: cloud-credentials
    6 
    
  snapshotLocations:
    7 
    
    - name: default
      velero:
        provider: aws
        config:
          region: <region>
    8 
    
          profile: "default"
    Copy to Clipboard Toggle word wrap
    1
    The openshift plugin is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster.
    2
    Set to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.
    3
    Specify the node selector to be supplied to Restic podSpec.
    4
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    5
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    6
    Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.
    7
    You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
    8
    The snapshot location must be in the same region as the PVs.
  4. Click Create.

  5. Verify the installation by viewing the OADP resources: 

    $ oc get all -n openshift-adp
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s
    Copy to Clipboard Toggle word wrap

4.3.2.5.1. Enabling CSI in the DataProtectionApplication CR
Link kopieren

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
    1
    Copy to Clipboard Toggle word wrap
    1
    Add the csi default plugin.

You install the OpenShift API for Data Protection (OADP) with Microsoft Azure by installing the OADP Operator, configuring Azure for Velero, and then installing the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.

4.3.3.1. Installing the OADP Operator
Link kopieren

You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.9 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.7.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click Operators OperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click Operators Installed Operators to verify the installation.

4.3.3.2. Configuring Microsoft Azure
Link kopieren

You configure a Microsoft Azure for the OpenShift API for Data Protection (OADP).

Prerequisites

Procedure

  1. Log in to Azure: 

    $ az login
    Copy to Clipboard Toggle word wrap

  2. Set the AZURE_RESOURCE_GROUP variable: 

    $ AZURE_RESOURCE_GROUP=Velero_Backups
    Copy to Clipboard Toggle word wrap

  3. Create an Azure resource group: 

    $ az group create -n $AZURE_RESOURCE_GROUP --location CentralUS
    1
    Copy to Clipboard Toggle word wrap
    1
    Specify your location.

  4. Set the AZURE_STORAGE_ACCOUNT_ID variable: 

    $ AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
    Copy to Clipboard Toggle word wrap

  5. Create an Azure storage account: 

    $ az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $AZURE_RESOURCE_GROUP \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot
    Copy to Clipboard Toggle word wrap

  6. Set the BLOB_CONTAINER variable: 

    $ BLOB_CONTAINER=velero
    Copy to Clipboard Toggle word wrap

  7. Create an Azure Blob storage container: 

    $ az storage container create \
  -n $BLOB_CONTAINER \
  --public-access off \
  --account-name $AZURE_STORAGE_ACCOUNT_ID
    Copy to Clipboard Toggle word wrap

  8. Obtain the storage account access key: 

    $ AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list \
  --account-name $AZURE_STORAGE_ACCOUNT_ID \
  --query "[?keyName == 'key1'].value" -o tsv`
    Copy to Clipboard Toggle word wrap

  9. Create a custom role that has the minimum required permissions: 

    AZURE_ROLE=Velero
az role definition create --role-definition '{
   "Name": "'$AZURE_ROLE'",
   "Description": "Velero related permissions to perform backups, restores and deletions",
   "Actions": [
       "Microsoft.Compute/disks/read",
       "Microsoft.Compute/disks/write",
       "Microsoft.Compute/disks/endGetAccess/action",
       "Microsoft.Compute/disks/beginGetAccess/action",
       "Microsoft.Compute/snapshots/read",
       "Microsoft.Compute/snapshots/write",
       "Microsoft.Compute/snapshots/delete",
       "Microsoft.Storage/storageAccounts/listkeys/action",
       "Microsoft.Storage/storageAccounts/regeneratekey/action"
   ],
   "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"]
   }'
    Copy to Clipboard Toggle word wrap

  10. Create a credentials-velero file: 

    $ cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY}
    1 
    
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
    Copy to Clipboard Toggle word wrap
    1
    Mandatory. You cannot back up internal images if the credentials-velero file contains only the service principal credentials.

    You use the credentials-velero file to create a Secret object for Azure before you install the Data Protection Application.

4.3.3.3. Creating a secret for backup and snapshot locations
Link kopieren

You create a Secret object for the backup and snapshot locations if they use the same credentials.

The default name of the Secret is cloud-credentials-azure.

Prerequisites

  • Your object storage and cloud storage must use the same credentials.
  • You must configure object storage for Velero.

  • You must create a credentials-velero file for the object storage in the appropriate format.

    Note

    The DataProtectionApplication custom resource (CR) requires a Secret for installation. If no spec.backupLocations.credential.name value is specified, the default name is used.

    If you do not want to specify the backup locations or the snapshot locations, you must create a Secret with the default name by using an empty credentials-velero file.

Procedure

  • Create a Secret with the default name: 

    $ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

If your backup and snapshot locations use different credentials, you create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials-azure. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.

  2. Create a Secret for the snapshot location with the default name: 

    $ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.

  4. Create a Secret for the backup location with a custom name: 

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
...
  backupLocations:
    - velero:
        config:
          resourceGroup: <azure_resource_group>
          storageAccount: <azure_storage_account_id>
          subscriptionId: <azure_subscription_id>
          storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
        credential:
          key: cloud
          name: <custom_secret>
    1 
    
        provider: azure
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <prefix>
  snapshotLocations:
    - velero:
        config:
          resourceGroup: <azure_resource_group>
          subscriptionId: <azure_subscription_id>
          incremental: "true"
        name: default
        provider: azure
    Copy to Clipboard Toggle word wrap
    1
    Backup location Secret with custom name.

4.3.3.4. Configuring the Data Protection Application
Link kopieren

You can configure Velero resource allocations and enable self-signed CA certificates.

4.3.3.4.1. Setting Velero CPU and memory resource allocations
Link kopieren

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node selector>
    1 
    
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
    Copy to Clipboard Toggle word wrap
    1
    Specify the node selector to be supplied to Velero podSpec
4.3.3.4.2. Enabling self-signed CA certificates
Link kopieren

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string>
    1 
    
        config:
          insecureSkipTLSVerify: "false"
    2 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Specify the Base46-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.

4.3.3.5. Installing the Data Protection Application
Link kopieren

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials-azure.

  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.

    • Secret with the default name, cloud-credentials-azure, for the snapshot location. This Secret is not referenced in the DataProtectionApplication CR.

      Note

      If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Operators Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

  3. Click YAML View and update the parameters of the DataProtectionApplication manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - azure
        - openshift
    1 
    
    restic:
      enable: true
    2 
    
      podConfig:
        nodeSelector: <node selector>
    3 
    
  backupLocations:
    - velero:
        config:
          resourceGroup: <azure_resource_group>
    4 
    
          storageAccount: <azure_storage_account_id>
    5 
    
          subscriptionId: <azure_subscription_id>
    6 
    
          storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
        credential:
          key: cloud
          name: cloud-credentials-azure
    7 
    
        provider: azure
        default: true
        objectStorage:
          bucket: <bucket_name>
    8 
    
          prefix: <prefix>
    9 
    
  snapshotLocations:
    10 
    
    - velero:
        config:
          resourceGroup: <azure_resource_group>
          subscriptionId: <azure_subscription_id>
          incremental: "true"
        name: default
        provider: azure
    Copy to Clipboard Toggle word wrap
    1
    The openshift plugin is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster.
    2
    Set to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.
    3
    Specify the node selector to be supplied to Restic podSpec.
    4
    Specify the Azure resource group.
    5
    Specify the Azure storage account ID.
    6
    Specify the Azure subscription ID.
    7
    If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.
    8
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    9
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    10
    You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
  4. Click Create.

  5. Verify the installation by viewing the OADP resources: 

    $ oc get all -n openshift-adp
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s
    Copy to Clipboard Toggle word wrap

4.3.3.5.1. Enabling CSI in the DataProtectionApplication CR
Link kopieren

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
    1
    Copy to Clipboard Toggle word wrap
    1
    Add the csi default plugin.

You install the OpenShift API for Data Protection (OADP) with Google Cloud Platform (GCP) by installing the OADP Operator, configuring GCP for Velero, and then installing the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.

4.3.4.1. Installing the OADP Operator
Link kopieren

You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.9 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.7.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click Operators OperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click Operators Installed Operators to verify the installation.

4.3.4.2. Configuring Google Cloud Platform
Link kopieren

You configure Google Cloud Platform (GCP) for the OpenShift API for Data Protection (OADP).

Prerequisites

Procedure

  1. Log in to GCP: 

    $ gcloud auth login
    Copy to Clipboard Toggle word wrap

  2. Set the BUCKET variable: 

    $ BUCKET=<bucket>
    1
    Copy to Clipboard Toggle word wrap
    1
    Specify your bucket name.

  3. Create the storage bucket: 

    $ gsutil mb gs://$BUCKET/
    Copy to Clipboard Toggle word wrap

  4. Set the PROJECT_ID variable to your active project: 

    $ PROJECT_ID=$(gcloud config get-value project)
    Copy to Clipboard Toggle word wrap

  5. Create a service account: 

    $ gcloud iam service-accounts create velero \
    --display-name "Velero service account"
    Copy to Clipboard Toggle word wrap

  6. List your service accounts: 

    $ gcloud iam service-accounts list
    Copy to Clipboard Toggle word wrap

  7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value: 

    $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
    --filter="displayName:Velero service account" \
    --format 'value(email)')
    Copy to Clipboard Toggle word wrap

  8. Attach the policies to give the velero user the minimum necessary permissions: 

    $ ROLE_PERMISSIONS=(
    compute.disks.get
    compute.disks.create
    compute.disks.createSnapshot
    compute.snapshots.get
    compute.snapshots.create
    compute.snapshots.useReadOnly
    compute.snapshots.delete
    compute.zones.get
)
    Copy to Clipboard Toggle word wrap

  9. Create the velero.server custom role: 

    $ gcloud iam roles create velero.server \
    --project $PROJECT_ID \
    --title "Velero Server" \
    --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"
    Copy to Clipboard Toggle word wrap

  10. Add IAM policy binding to the project: 

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
    --role projects/$PROJECT_ID/roles/velero.server
    Copy to Clipboard Toggle word wrap

  11. Update the IAM service account: 

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}
    Copy to Clipboard Toggle word wrap

  12. Save the IAM service account keys to the credentials-velero file in the current directory: 

    $ gcloud iam service-accounts keys create credentials-velero \
    --iam-account $SERVICE_ACCOUNT_EMAIL
    Copy to Clipboard Toggle word wrap

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

4.3.4.3. Creating a secret for backup and snapshot locations
Link kopieren

You create a Secret object for the backup and snapshot locations if they use the same credentials.

The default name of the Secret is cloud-credentials-gcp.

Prerequisites

  • Your object storage and cloud storage must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name: 

    $ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

If your backup and snapshot locations use different credentials, you create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials-gcp. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.

  2. Create a Secret for the snapshot location with the default name: 

    $ oc create secret generic cloud-credentials-gcp -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.

  4. Create a Secret for the backup location with a custom name: 

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
...
  backupLocations:
    - velero:
        provider: gcp
        default: true
        credential:
          key: cloud
          name: <custom_secret>
    1 
    
        objectStorage:
          bucket: <bucket_name>
          prefix: <prefix>
  snapshotLocations:
    - velero:
        provider: gcp
        default: true
        config:
          project: <project>
          snapshotLocation: us-west1
    Copy to Clipboard Toggle word wrap
    1
    Backup location Secret with custom name.

4.3.4.4. Configuring the Data Protection Application
Link kopieren

You can configure Velero resource allocations and enable self-signed CA certificates.

4.3.4.4.1. Setting Velero CPU and memory resource allocations
Link kopieren

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node selector>
    1 
    
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
    Copy to Clipboard Toggle word wrap
    1
    Specify the node selector to be supplied to Velero podSpec
4.3.4.4.2. Enabling self-signed CA certificates
Link kopieren

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string>
    1 
    
        config:
          insecureSkipTLSVerify: "false"
    2 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Specify the Base46-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.

4.3.4.5. Installing the Data Protection Application
Link kopieren

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials-gcp.

  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.

    • Secret with the default name, cloud-credentials-gcp, for the snapshot location. This Secret is not referenced in the DataProtectionApplication CR.

      Note

      If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Operators Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

  3. Click YAML View and update the parameters of the DataProtectionApplication manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - gcp
        - openshift
    1 
    
    restic:
      enable: true
    2 
    
      podConfig:
        nodeSelector: <node selector>
    3 
    
  backupLocations:
    - velero:
        provider: gcp
        default: true
        credential:
          key: cloud
          name: cloud-credentials-gcp
    4 
    
        objectStorage:
          bucket: <bucket_name>
    5 
    
          prefix: <prefix>
    6 
    
  snapshotLocations:
    7 
    
    - velero:
        provider: gcp
        default: true
        config:
          project: <project>
          snapshotLocation: us-west1
    8
    Copy to Clipboard Toggle word wrap
    1
    The openshift plugin is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster.
    2
    Set to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.
    3
    Specify the node selector to be supplied to Restic podSpec.
    4
    If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.
    5
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    6
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
    7
    You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.
    8
    The snapshot location must be in the same region as the PVs.
  4. Click Create.

  5. Verify the installation by viewing the OADP resources: 

    $ oc get all -n openshift-adp
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s
    Copy to Clipboard Toggle word wrap

4.3.4.5.1. Enabling CSI in the DataProtectionApplication CR
Link kopieren

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
    1
    Copy to Clipboard Toggle word wrap
    1
    Add the csi default plugin.

You install the OpenShift API for Data Protection (OADP) with Multicloud Object Gateway (MCG) by installing the OADP Operator, creating a Secret object, and then installing the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

MCG is a component of OpenShift Container Storage (OCS). You configure MCG as a backup location in the DataProtectionApplication custom resource (CR).

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If your cloud provider has a native snapshot API, configure a snapshot location. If your cloud provider does not support snapshots or if your storage is NFS, you can create backups with Restic.

You do not need to specify a snapshot location in the DataProtectionApplication CR for Restic or Container Storage Interface (CSI) snapshots.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.

4.3.5.1. Installing the OADP Operator
Link kopieren

You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.9 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.7.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click Operators OperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click Operators Installed Operators to verify the installation.

4.3.5.2. Retrieving Multicloud Object Gateway credentials
Link kopieren

You must retrieve the Multicloud Object Gateway (MCG) credentials in order to create a Secret custom resource (CR) for the OpenShift API for Data Protection (OADP).

MCG is a component of OpenShift Container Storage.

Prerequisites

Procedure

  1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

  2. Create a credentials-velero file: 

    $ cat << EOF > ./credentials-velero
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF
    Copy to Clipboard Toggle word wrap

    You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

4.3.5.3. Creating a secret for backup and snapshot locations
Link kopieren

You create a Secret object for the backup and snapshot locations if they use the same credentials.

The default name of the Secret is cloud-credentials.

Prerequisites

  • Your object storage and cloud storage must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name: 

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

If your backup and snapshot locations use different credentials, you create two Secret objects:

  • Backup location Secret with a custom name. The custom name is specified in the spec.backupLocations block of the DataProtectionApplication custom resource (CR).
  • Snapshot location Secret with the default name, cloud-credentials. This Secret is not specified in the DataProtectionApplication CR.

Procedure

  1. Create a credentials-velero file for the snapshot location in the appropriate format for your cloud provider.

  2. Create a Secret for the snapshot location with the default name: 

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap
  3. Create a credentials-velero file for the backup location in the appropriate format for your object storage.

  4. Create a Secret for the backup location with a custom name: 

    $ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

  5. Add the Secret with the custom name to the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - aws
        - openshift
    restic:
      enable: true
  backupLocations:
    - velero:
        config:
          profile: "default"
          region: minio
          s3Url: <url>
          insecureSkipTLSVerify: "true"
          s3ForcePathStyle: "true"
        provider: aws
        default: true
        credential:
          key: cloud
          name:  <custom_secret>
    1 
    
        objectStorage:
          bucket: <bucket_name>
          prefix: <prefix>
    Copy to Clipboard Toggle word wrap
    1
    Backup location Secret with custom name.

4.3.5.4. Configuring the Data Protection Application
Link kopieren

You can configure Velero resource allocations and enable self-signed CA certificates.

4.3.5.4.1. Setting Velero CPU and memory resource allocations
Link kopieren

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node selector>
    1 
    
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
    Copy to Clipboard Toggle word wrap
    1
    Specify the node selector to be supplied to Velero podSpec
4.3.5.4.2. Enabling self-signed CA certificates
Link kopieren

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string>
    1 
    
        config:
          insecureSkipTLSVerify: "false"
    2 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Specify the Base46-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.

4.3.5.5. Installing the Data Protection Application
Link kopieren

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

  • If the backup and snapshot locations use different credentials, you must create two Secrets:

    • Secret with a custom name for the backup location. You add this Secret to the DataProtectionApplication CR.

    • Secret with the default name, cloud-credentials, for the snapshot location. This Secret is not referenced in the DataProtectionApplication CR.

      Note

      If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Operators Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

  3. Click YAML View and update the parameters of the DataProtectionApplication manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - aws
        - openshift
    1 
    
    restic:
      enable: true
    2 
    
      podConfig:
        nodeSelector: <node selector>
    3 
    
  backupLocations:
    - velero:
        config:
          profile: "default"
          region: minio
          s3Url: <url>
    4 
    
          insecureSkipTLSVerify: "true"
          s3ForcePathStyle: "true"
        provider: aws
        default: true
        credential:
          key: cloud
          name: cloud-credentials
    5 
    
        objectStorage:
          bucket: <bucket_name>
    6 
    
          prefix: <prefix>
    7
    Copy to Clipboard Toggle word wrap
    1
    The openshift plugin is mandatory in order to back up and restore namespaces on an OpenShift Container Platform cluster.
    2
    Set to false if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.
    3
    Specify the node selector to be supplied to Restic podSpec.
    4
    Specify the URL of the S3 endpoint.
    5
    If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.
    6
    Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    7
    Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
  4. Click Create.

  5. Verify the installation by viewing the OADP resources: 

    $ oc get all -n openshift-adp
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s
    Copy to Clipboard Toggle word wrap

4.3.5.5.1. Enabling CSI in the DataProtectionApplication CR
Link kopieren

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
    1
    Copy to Clipboard Toggle word wrap
    1
    Add the csi default plugin.

You install the OpenShift API for Data Protection (OADP) with OpenShift Container Storage (OCS) by installing the OADP Operator and configuring a backup location and a snapshot location. Then, you install the Data Protection Application.

Note

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

You can configure Multicloud Object Gateway or any S3-compatible object storage as a backup location in the DataProtectionApplication custom resource (CR).

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If the cloud provider has a native snapshot API, you can configure cloud storage as a snapshot location in the DataProtectionApplication CR. You do not need to specify a snapshot location for Restic or Container Storage Interface (CSI) snapshots.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. For details, see Using Operator Lifecycle Manager on restricted networks.

4.3.6.1. Installing the OADP Operator
Link kopieren

You install the OpenShift API for Data Protection (OADP) Operator on OpenShift Container Platform 4.9 by using Operator Lifecycle Manager (OLM).

The OADP Operator installs Velero 1.7.

Prerequisites

  • You must be logged in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, click Operators OperatorHub.
  2. Use the Filter by keyword field to find the OADP Operator.
  3. Select the OADP Operator and click Install.
  4. Click Install to install the Operator in the openshift-adp project.
  5. Click Operators Installed Operators to verify the installation.
Note

After you install the OADP Operator, you configure object storage as a backup location and cloud storage as a snapshot location, if the cloud provider supports a native snapshot API.

If the cloud provider does not support snapshots or if your storage is NFS, you can create backups with Restic. Restic does not require a snapshot location.

4.3.6.2. Creating a secret for backup and snapshot locations
Link kopieren

You create a Secret object for the backup and snapshot locations if they use the same credentials.

Prerequisites

  • Your object storage and cloud storage must use the same credentials.
  • You must configure object storage for Velero.
  • You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

  • Create a Secret with the default name: 

    $ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
    Copy to Clipboard Toggle word wrap

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

4.3.6.3. Configuring the Data Protection Application
Link kopieren

You can configure Velero resource allocations and enable self-signed CA certificates.

4.3.6.3.1. Setting Velero CPU and memory resource allocations
Link kopieren

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node selector>
    1 
    
        resourceAllocations:
          limits:
            cpu: "1"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 256Mi
    Copy to Clipboard Toggle word wrap
    1
    Specify the node selector to be supplied to Velero podSpec
4.3.6.3.2. Enabling self-signed CA certificates
Link kopieren

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string>
    1 
    
        config:
          insecureSkipTLSVerify: "false"
    2 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Specify the Base46-encoded CA certificate string.
    2
    The insecureSkipTLSVerify configuration can be set to either "true" or "false". If set to "true", SSL/TLS security is disabled. If set to "false", SSL/TLS security is enabled.

4.3.6.4. Installing the Data Protection Application
Link kopieren

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.

  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Operators Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:
  4. Click Create.

  5. Verify the installation by viewing the OADP resources: 

    $ oc get all -n openshift-adp
    Copy to Clipboard Toggle word wrap

    Example output

    NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s
    Copy to Clipboard Toggle word wrap

If you use cluster storage for your NooBaa bucket backupStorageLocation on OpenShift Container Storage, configure NooBaa as an external object store.

Warning

Failure to configure NooBaa as an external object store might lead to backups not being available.

Procedure

4.3.6.4.2. Enabling CSI in the DataProtectionApplication CR
Link kopieren

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

  • The cloud provider must support CSI snapshots.

Procedure

  • Edit the DataProtectionApplication CR, as in the following example: 

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi
    1
    Copy to Clipboard Toggle word wrap
    1
    Add the csi default plugin.

4.3.7. Uninstalling the OpenShift API for Data Protection
Link kopieren

You uninstall the OpenShift API for Data Protection (OADP) by deleting the OADP Operator. See Deleting Operators from a cluster for details.

4.4. Backing up and restoring
Link kopieren

4.4.1. Backing up applications
Link kopieren

You back up applications by creating a Backup custom resource (CR).

The Backup CR creates backup files for Kubernetes resources and internal images, on S3 object storage, and snapshots for persistent volumes (PVs), if the cloud provider uses a native snapshot API or the Container Storage Interface (CSI) to create snapshots, such as OpenShift Container Storage 4. For more information, see CSI volume snapshots.

Important

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If your cloud provider has a native snapshot API or supports Container Storage Interface (CSI) snapshots, the Backup CR backs up persistent volumes by creating snapshots. For more information, see the Overview of CSI volume snapshots in the OpenShift Container Platform documentation.

If your cloud provider does not support snapshots or if your applications are on NFS data volumes, you can create backups by using Restic.

You can create backup hooks to run commands before or after the backup operation.

You can schedule backups by creating a Schedule CR instead of a Backup CR.

4.4.1.1. Creating a Backup CR
Link kopieren

You back up Kubernetes images, internal images, and persistent volumes (PVs) by creating a Backup custom resource (CR).

Prerequisites

  • You must install the OpenShift API for Data Protection (OADP) Operator.
  • The DataProtectionApplication CR must be in a Ready state.

  • Backup location prerequisites:

    • You must have S3 object storage configured for Velero.
    • You must have a backup location configured in the DataProtectionApplication CR.

  • Snapshot location prerequisites:

    • Your cloud provider must have a native snapshot API or support Container Storage Interface (CSI) snapshots.
    • For CSI snapshots, you must create a VolumeSnapshotClass CR to register the CSI driver.
    • You must have a volume location configured in the DataProtectionApplication CR.

Procedure

  1. Retrieve the backupStorageLocations CRs by entering the following command: 

    $ oc get backupStorageLocations
    Copy to Clipboard Toggle word wrap

    Example output

    NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
velero-sample-1   Available   11s              31m
    Copy to Clipboard Toggle word wrap

  2. Create a Backup CR, as in the following example: 

    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  labels:
    velero.io/storage-location: default
  namespace: openshift-adp
spec:
  hooks: {}
  includedNamespaces:
  - <namespace>
    1 
    
  includedResources: []
    2 
    
  excludedResources: []
    3 
    
  storageLocation: <velero-sample-1>
    4 
    
  ttl: 720h0m0s
  labelSelector:
    5 
    
  - matchLabels:
      app=<label_1>
  - matchLabels:
      app=<label_2>
  - matchLabels:
      app=<label_3>
  orlabelSelectors:
    6 
    
  - matchLabels:
      app=<label_1>
  - matchLabels:
      app=<label_2>
  - matchLabels:
      app=<label_3>
    Copy to Clipboard Toggle word wrap
    1
    Specify an array of namespaces to back up.
    2
    Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.
    3
    Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.
    4
    Specify the name of the backupStorageLocations CR.
    5
    Backup resources that have all of the specified labels.
    6
    Backup resources that have one or more of the specified labels.

  3. Verify that the status of the Backup CR is Completed: 

    $ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'
    Copy to Clipboard Toggle word wrap

4.4.1.2. Backing up persistent volumes with CSI snapshots
Link kopieren

You back up persistent volumes with Container Storage Interface (CSI) snapshots by creating a VolumeSnapshotClass custom resource (CR) to register the CSI driver before you create the Backup CR.

Prerequisites

  • The cloud provider must support CSI snapshots.
  • You must enable CSI in the DataProtectionApplication CR.

Procedure

  • Create a VolumeSnapshotClass CR, as in the following examples:

    Ceph RBD

    apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
deletionPolicy: Retain
metadata:
  name: <volume_snapshot_class_name>
  labels:
    velero.io/csi-volumesnapshot-class: "true"
    snapshotter: openshift-storage.rbd.csi.ceph.com
driver: openshift-storage.rbd.csi.ceph.com
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
    Copy to Clipboard Toggle word wrap

    Ceph FS

    apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: <volume_snapshot_class_name>
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: openshift-storage.cephfs.csi.ceph.com
deletionPolicy: Retain
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
    Copy to Clipboard Toggle word wrap

    Other cloud providers

    apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: <volume_snapshot_class_name>
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: <csi_driver>
deletionPolicy: Retain
    Copy to Clipboard Toggle word wrap

You can now create a Backup CR.

4.4.1.3. Backing up applications with Restic
Link kopieren

You back up Kubernetes resources, internal images, and persistent volumes with Restic by editing the Backup custom resource (CR).

You do not need to specify a snapshot location in the DataProtectionApplication CR.

Important

Restic does not support backing up hostPath volumes. For more information, see additional Rustic limitations.

Prerequisites

  • You must install the OpenShift API for Data Protection (OADP) Operator.
  • You must not disable the default Restic installation by setting spec.configuration.restic.enable to false in the DataProtectionApplication CR.
  • The DataProtectionApplication CR must be in a Ready state.

Procedure

  • Edit the Backup CR, as in the following example: 

    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  labels:
    velero.io/storage-location: default
  namespace: openshift-adp
spec:
  defaultVolumesToRestic: true
    1 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Add defaultVolumesToRestic: true to the spec block.

4.4.1.4. Using Data Mover for CSI snapshots
Link kopieren

Important

Data Mover for CSI snapshots is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The OADP 1.1.0 Data Mover enables customers to back up container storage interface (CSI) volume snapshots to a remote object store. When Data Mover is enabled, you can restore stateful applications from the store if a failure, accidental deletion, or corruption of the cluster occurs. The OADP 1.1.0 Data Mover solution uses the Restic option of VolSync.

Note

Data Mover supports backup and restore of CSI volume snapshots only.

Currently, Data Mover does not support Google Cloud Storage (GCS) buckets.

Prerequisites

  • You have verified that the StorageClass and VolumeSnapshotClass custom resources (CRs) support CSI.
  • You have verified that only one volumeSnapshotClass CR has the annotation snapshot.storage.kubernetes.io/is-default-class: true.
  • You have verified that only one storageClass CR has the annotation storageclass.kubernetes.io/is-default-class: true.
  • You have included the label velero.io/csi-volumesnapshot-class: 'true' in your VolumeSnapshotClass CR.

  • You have installed the VolSync Operator by using the Operator Lifecycle Manager (OLM).

    Note

    The VolSync Operator is required only for use with the Technology Preview Data Mover. The Operator is not required for using OADP production features.

  • You have installed the OADP operator by using OLM.

Procedure

  1. Configure a Restic secret by creating a .yaml file as following: 

    apiVersion: v1
kind: Secret
metadata:
  name: <secret_name>
  namespace: openshift-adp
type: Opaque
stringData:
  RESTIC_PASSWORD: <secure_restic_password>
    Copy to Clipboard Toggle word wrap
    Note

    By default, the Operator looks for a secret named dm-credential. If you are using a different name, you need to specify the name through a Data Protection Application (DPA) CR using dpa.spec.features.dataMover.credentialName.

  2. Create a DPA CR similar to the following example. The default plugins include CSI.

    Example Data Protection Application (DPA) CR

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  features:
    dataMover:
      enable: true
      credentialName: <secret_name>
    1 
    
  backupLocations:
    - velero:
        config:
          profile: default
          region: us-east-1
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <bucket_prefix>
        provider: aws
  configuration:
    restic:
      enable: <true_or_false>
    velero:
      defaultPlugins:
        - openshift
        - aws
        - csi
    Copy to Clipboard Toggle word wrap

    1
    Add the Restic secret name from the previous step. If this is not done, the default secret name dm-credential is used.

    The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

    Example VolumeSnapshotBackup CRD

    apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotBackup
metadata:
  name: <vsb_name>
  namespace: <namespace_name>
    1 
    
spec:
  volumeSnapshotContent:
    name: <snapcontent_name>
  protectedNamespace: <adp_namespace>
  resticSecretRef:
    name: <restic_secret_name>
    Copy to Clipboard Toggle word wrap

    1
    Specify the namespace where the volume snapshot exists.

    Example VolumeSnapshotRestore CRD

    apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotRestore
metadata:
  name: <vsr_name>
  namespace: <namespace_name>
    1 
    
spec:
  protectedNamespace: <protected_ns>
    2 
    
  resticSecretRef:
    name: <restic_secret_name>
  volumeSnapshotMoverBackupRef:
    sourcePVCData:
      name: <source_pvc_name>
      size: <source_pvc_size>
    resticrepository: <your_restic_repo>
    volumeSnapshotClassName: <vsclass_name>
    Copy to Clipboard Toggle word wrap

    1
    Specify the namespace where the volume snapshot exists.
    2
    Specify the namespace where the Operator is installed. The default is openshift-adp.

  3. You can back up a volume snapshot by performing the following steps:

    1. Create a backup CR: 

      apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup_name>
  namespace: <protected_ns>
      1 
      
spec:
  includedNamespaces:
  - <app_ns>
  storageLocation: velero-sample-1
      Copy to Clipboard Toggle word wrap
      1
      Specify the namespace where the Operator is installed. The default namespace is openshift-adp.

    2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands: 

      $ oc get vsb -n <app_ns>
      Copy to Clipboard Toggle word wrap 
      $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"
      Copy to Clipboard Toggle word wrap

      A snapshot is created in the object store was configured in the DPA.

      Note

      If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.

  4. You can restore a volume snapshot by performing the following steps:

    1. Delete the application namespace and the volumeSnapshotContent that was created by the Velero CSI plugin.

    2. Create a Restore CR and set restorePVs to true.

      Example Restore CR

      apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore_name>
  namespace: <protected_ns>
spec:
  backupName: <previous_backup_name>
  restorePVs: true
      Copy to Clipboard Toggle word wrap

    3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command: 

      $ oc get vsr -n <app_ns>
      Copy to Clipboard Toggle word wrap 
      $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"
      Copy to Clipboard Toggle word wrap

    4. Check whether your application data and resources have been restored.

      Note

      If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

4.4.1.5. Creating backup hooks
Link kopieren

You create backup hooks to run commands in a container in a pod by editing the Backup custom resource (CR).

Pre hooks run before the pod is backed up. Post hooks run after the backup.

Procedure

  • Add a hook to the spec.hooks block of the Backup CR, as in the following example: 

    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  namespace: openshift-adp
spec:
  hooks:
    resources:
      - name: <hook_name>
        includedNamespaces:
        - <namespace>
    1 
    
        excludedNamespaces:
    2 
    
        - <namespace>
        includedResources: []
        - pods
    3 
    
        excludedResources: []
    4 
    
        labelSelector:
    5 
    
          matchLabels:
            app: velero
            component: server
        pre:
    6 
    
          - exec:
              container: <container>
    7 
    
              command:
              - /bin/uname
    8 
    
              - -a
              onError: Fail
    9 
    
              timeout: 30s
    10 
    
        post:
    11 
    
...
    Copy to Clipboard Toggle word wrap
    1
    Optional: You can specify namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
    2
    Optional: You can specify namespaces to which the hook does not apply.
    3
    Currently, pods are the only supported resource that hooks can apply to.
    4
    Optional: You can specify resources to which the hook does not apply.
    5
    Optional: This hook only applies to objects matching the label. If this value is not specified, the hook applies to all namespaces.
    6
    Array of hooks to run before the backup.
    7
    Optional: If the container is not specified, the command runs in the first container in the pod.
    8
    This is the entrypoint for the init container being added.
    9
    Allowed values for error handling are Fail and Continue. The default is Fail.
    10
    Optional: How long to wait for the commands to run. The default is 30s.
    11
    This block defines an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.

4.4.1.6. Scheduling backups
Link kopieren

You schedule backups by creating a Schedule custom resource (CR) instead of a Backup CR.

Warning

Leave enough time in your backup schedule for a backup to finish before another backup is created.

For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.

Prerequisites

  • You must install the OpenShift API for Data Protection (OADP) Operator.
  • The DataProtectionApplication CR must be in a Ready state.

Procedure

  1. Retrieve the backupStorageLocations CRs: 

    $ oc get backupStorageLocations
    Copy to Clipboard Toggle word wrap

    Example output

    NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
velero-sample-1   Available   11s              31m
    Copy to Clipboard Toggle word wrap

  2. Create a Schedule CR, as in the following example: 

    $ cat << EOF | oc apply -f -
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: <schedule>
  namespace: openshift-adp
spec:
  schedule: 0 7 * * *
    1 
    
  template:
    hooks: {}
    includedNamespaces:
    - <namespace>
    2 
    
    storageLocation: <velero-sample-1>
    3 
    
    defaultVolumesToRestic: true
    4 
    
    ttl: 720h0m0s
EOF
    Copy to Clipboard Toggle word wrap
    1
    cron expression to schedule the backup, for example, 0 7 * * * to perform a backup every day at 7:00.
    2
    Array of namespaces to back up.
    3
    Name of the backupStorageLocations CR.
    4
    Optional: Add the defaultVolumesToRestic: true key-value pair if you are backing up volumes with Restic.

  3. Verify that the status of the Schedule CR is Completed after the scheduled backup runs: 

    $ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'
    Copy to Clipboard Toggle word wrap

4.4.1.7. Deleting backups
Link kopieren

You can remove backup files by deleting the Backup custom resource (CR).

Warning

After you delete the Backup CR and the associated object storage data, you cannot recover the deleted data.

Prerequisites

  • You created a Backup CR.
  • You know the name of the Backup CR and the namespace that contains it.
  • You downloaded the Velero CLI tool.
  • You can access the Velero binary in your cluster.

Procedure

  • Choose one of the following actions to delete the Backup CR:

    • To delete the Backup CR and keep the associated object storage data, issue the following command: 

      $ oc delete backup <backup_CR_name> -n <velero_namespace>
      Copy to Clipboard Toggle word wrap

    • To delete the Backup CR and delete the associated object storage data, issue the following command: 

      $ velero backup delete <backup_CR_name> -n <velero_namespace>
      Copy to Clipboard Toggle word wrap

      Where:

      <backup_CR_name>
      Specifies the name of the Backup custom resource.
      <velero_namespace>
      Specifies the namespace that contains the Backup custom resource.

4.4.2. Restoring applications
Link kopieren

You restore application backups by creating a Restore custom resources (CRs).

You can create restore hooks to run commands in init containers, before the application container starts, or in the application container itself.

4.4.2.1. Creating a Restore CR
Link kopieren

You restore a Backup custom resource (CR) by creating a Restore CR.

Prerequisites

  • You must install the OpenShift API for Data Protection (OADP) Operator.
  • The DataProtectionApplication CR must be in a Ready state.
  • You must have a Velero Backup CR.
  • Adjust the requested size so the persistent volume (PV) capacity matches the requested size at backup time.

Procedure

  1. Create a Restore CR, as in the following example: 

    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  backupName: <backup>
    1 
    
  includedResources: []
    2 
    
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  restorePVs: true
    Copy to Clipboard Toggle word wrap
    1
    Name of the Backup CR.
    2
    Optional. Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.

  2. Verify that the status of the Restore CR is Completed by entering the following command: 

    $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'
    Copy to Clipboard Toggle word wrap

  3. Verify that the backup resources have been restored by entering the following command: 

    $ oc get all -n <namespace>
    1
    Copy to Clipboard Toggle word wrap
    1
    Namespace that you backed up.

  4. If you use Restic to restore DeploymentConfig objects or if you use post-restore hooks, run the dc-restic-post-restore.sh cleanup script by entering the following command: 

    $ bash dc-restic-post-restore.sh <restore-name>
    Copy to Clipboard Toggle word wrap
    Note

    In the course of the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow Restic and post-restore hooks to complete their actions on the restored pods. The cleanup script removes these disconnected pods and scale any DeploymentConfig objects back up to the appropriate number of replicas.

    Example 4.1. dc-restic-post-restore.sh cleanup script

    #!/bin/bash
set -e

# if sha256sum exists, use it to check the integrity of the file
if command -v sha256sum >/dev/null 2>&1; then
  CHECKSUM_CMD="sha256sum"
else
  CHECKSUM_CMD="shasum -a 256"
fi

label_name () {
    if [ "${#1}" -le "63" ]; then
	echo $1
	return
    fi
    sha=$(echo -n $1|$CHECKSUM_CMD)
    echo "${1:0:57}${sha:0:6}"
}

OADP_NAMESPACE=${OADP_NAMESPACE:=openshift-adp}

if [[ $# -ne 1 ]]; then
    echo "usage: ${BASH_SOURCE} restore-name"
    exit 1
fi

echo using OADP Namespace $OADP_NAMESPACE
echo restore: $1

label=$(label_name $1)
echo label: $label

echo Deleting disconnected restore pods
oc delete pods -l oadp.openshift.io/disconnected-from-dc=$label

for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
do
    IFS=',' read -ra dc_arr <<< "$dc"
    if [ ${#dc_arr[0]} -gt 0 ]; then
	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
spec:
  replicas: ${dc_arr[2]}
  paused: ${dc_arr[3]}
EOF
    fi
done
    Copy to Clipboard Toggle word wrap

4.4.2.2. Creating restore hooks
Link kopieren

You create restore hooks to run commands in a container in a pod while restoring your application by editing the Restore custom resource (CR).

You can create two types of restore hooks:

  • An init hook adds an init container to a pod to perform setup tasks before the application container starts.

    If you restore a Restic backup, the restic-wait init container is added before the restore hook init container.

  • An exec hook runs commands or scripts in a container of a restored pod.

Procedure

  • Add a hook to the spec.hooks block of the Restore CR, as in the following example: 

    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: <restore>
  namespace: openshift-adp
spec:
  hooks:
    resources:
      - name: <hook_name>
        includedNamespaces:
        - <namespace>
    1 
    
        excludedNamespaces:
        - <namespace>
        includedResources:
        - pods
    2 
    
        excludedResources: []
        labelSelector:
    3 
    
          matchLabels:
            app: velero
            component: server
        postHooks:
        - init:
            initContainers:
            - name: restore-hook-init
              image: alpine:latest
              volumeMounts:
              - mountPath: /restores/pvc1-vm
                name: pvc1-vm
              command:
              - /bin/ash
              - -c
            timeout:
    4 
    
        - exec:
            container: <container>
    5 
    
            command:
            - /bin/bash
    6 
    
            - -c
            - "psql < /backup/backup.sql"
            waitTimeout: 5m
    7 
    
            execTimeout: 1m
    8 
    
            onError: Continue
    9
    Copy to Clipboard Toggle word wrap
    1
    Optional: Array of namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
    2
    Currently, pods are the only supported resource that hooks can apply to.
    3
    Optional: This hook only applies to objects matching the label selector.
    4
    Optional: Timeout specifies the maximum amount of time Velero waits for initContainers to complete.
    5
    Optional: If the container is not specified, the command runs in the first container in the pod.
    6
    This is the entrypoint for the init container being added.
    7
    Optional: How long to wait for a container to become ready. This should be long enough for the container to start and for any preceding hooks in the same container to complete. If not set, the restore process waits indefinitely.
    8
    Optional: How long to wait for the commands to run. The default is 30s.
    9
    Allowed values for error handling are Fail and Continue:
    • Continue: Only command failures are logged.
    • Fail: No more restore hooks run in any container in any pod. The status of the Restore CR will be PartiallyFailed.

4.5. Troubleshooting
Link kopieren

You can debug Velero custom resources (CRs) by using the OpenShift CLI tool or the Velero CLI tool. The Velero CLI tool provides more detailed logs and information.

You can check installation issues, backup and restore CR issues, and Restic issues.

You can collect logs, CR information, and Prometheus metric data by using the must-gather tool.

You can obtain the Velero CLI tool by:

  • Downloading the Velero CLI tool
  • Accessing the Velero binary in the Velero deployment in the cluster

4.5.1. Downloading the Velero CLI tool
Link kopieren

You can download and install the Velero CLI tool by following the instructions on the Velero documentation page.

The page includes instructions for:

  • macOS by using Homebrew
  • GitHub
  • Windows by using Chocolatey

Prerequisites

  • You have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
  • You have installed kubectl locally.

Procedure

  1. Open a browser and navigate to "Install the CLI" on the Verleo website.
  2. Follow the appropriate procedure for macOS, GitHub, or Windows.

  3. Download the Velero version appropriate for your version of OADP and OpenShift Container Platform according to the table that follows:

    Expand
    Table 4.2. OADP-Velero-OpenShift Container Platform version relationship
    OADP versionVelero versionOpenShift Container Platform version

    1.0.0

    1.7

    4.6 and later

    1.0.1

    1.7

    4.6 and later

    1.0.2

    1.7

    4.6 and later

    1.0.3

    1.7

    4.6 and later

    1.1.0

    {velero-version}

    4.9 and later

    1.1.1

    {velero-version}

    4.9 and later

    1.1.2

    {velero-version}

    4.9 and later

You can use a shell command to access the Velero binary in the Velero deployment in the cluster.

Prerequisites

  • Your DataProtectionApplication custom resource has a status of Reconcile complete.

Procedure

  • Enter the following command to set the needed alias: 

    $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
    Copy to Clipboard Toggle word wrap

4.5.3. Debugging Velero resources with the OpenShift CLI tool
Link kopieren

You can debug a failed backup or restore by checking Velero custom resources (CRs) and the Velero pod log with the OpenShift CLI tool.

Velero CRs

Use the oc describe command to retrieve a summary of warnings and errors associated with a Backup or Restore CR: 

$ oc describe <velero_cr> <cr_name>
Copy to Clipboard Toggle word wrap
Velero pod logs

Use the oc logs command to retrieve the Velero pod logs: 

$ oc logs pod/<velero>
Copy to Clipboard Toggle word wrap
Velero pod debug logs

You can specify the Velero log level in the DataProtectionApplication resource as shown in the following example.

Note

This option is available starting from OADP 1.0.3. 

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
spec:
  configuration:
    velero:
      logLevel: warning
Copy to Clipboard Toggle word wrap

The following logLevel values are available:

  • trace
  • debug
  • info
  • warning
  • error
  • fatal
  • panic

It is recommended to use debug for most logs.

4.5.4. Debugging Velero resources with the Velero CLI tool
Link kopieren

You can debug Backup and Restore custom resources (CRs) and retrieve logs with the Velero CLI tool.

The Velero CLI tool provides more detailed information than the OpenShift CLI tool.

Syntax

Use the oc exec command to run a Velero CLI command: 

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> <command> <cr_name>
Copy to Clipboard Toggle word wrap

Example

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Copy to Clipboard Toggle word wrap

Help option

Use the velero --help option to list all Velero CLI commands: 

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  --help
Copy to Clipboard Toggle word wrap
Describe command

Use the velero describe command to retrieve a summary of warnings and errors associated with a Backup or Restore CR: 

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> describe <cr_name>
Copy to Clipboard Toggle word wrap

Example

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Copy to Clipboard Toggle word wrap

Logs command

Use the velero logs command to retrieve the logs of a Backup or Restore CR: 

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> logs <cr_name>
Copy to Clipboard Toggle word wrap

Example

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
Copy to Clipboard Toggle word wrap

4.5.5. Issues with Velero and admission webhooks
Link kopieren

Velero has limited abilities to resolve admission webhook issues during a restore. If you have workloads with admission webhooks, you might need to use an additional Velero plugin or make changes to how you restore the workload.

Typically, workloads with admission webhooks require you to create a resource of a specific kind first. This is especially true if your workload has child resources because admission webhooks typically block child resources.

For example, creating or restoring a top-level object such as service.serving.knative.dev typically creates child resources automatically. If you do this first, you will not need to use Velero to create and restore these resources. This avoids the problem of child resources being blocked by an admission webhook that Velero might use.

This section describes the additional steps required to restore resources for several types of Velero backups that use admission webhooks.

4.5.5.1.1. Restoring Knative resources
Link kopieren

You might encounter problems using Velero to back up Knative resources that use admission webhooks.

You can avoid such problems by restoring the top level Service resource first whenever you back up and restore Knative resources that use admission webhooks.

Procedure

  • Restore the top level service.serving.knavtive.dev Service resource: 

    $ velero restore <restore_name> \
  --from-backup=<backup_name> --include-resources \
  service.serving.knavtive.dev
    Copy to Clipboard Toggle word wrap
4.5.5.1.2. Restoring IBM AppConnect resources
Link kopieren

If you experience issues when you use Velero to a restore an IBM AppConnect resource that has an admission webhook, you can run the checks in this procedure.

Procedure

  1. Check if you have any mutating admission plugins of kind: MutatingWebhookConfiguration in the cluster: 

    $ oc get mutatingwebhookconfigurations
    Copy to Clipboard Toggle word wrap
  2. Examine the YAML file of each kind: MutatingWebhookConfiguration to ensure that none of its rules block creation of the objects that are experiencing issues. For more information, see the official Kuberbetes documentation.
  3. Check that any spec.version in type: Configuration.appconnect.ibm.com/v1beta1 used at backup time is supported by the installed Operator.

4.5.6. Installation issues
Link kopieren

You might encounter issues caused by using invalid directories or incorrect credentials when you install the Data Protection Application.

4.5.6.1. Backup storage contains invalid directories
Link kopieren

The Velero pod log displays the error message, Backup storage contains invalid top-level directories.

Cause

The object storage contains top-level directories that are not Velero directories.

Solution

If the object storage is not dedicated to Velero, you must specify a prefix for the bucket by setting the spec.backupLocations.velero.objectStorage.prefix parameter in the DataProtectionApplication manifest.

4.5.6.2. Incorrect AWS credentials
Link kopieren

The oadp-aws-registry pod log displays the error message, InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

The Velero pod log displays the error message, NoCredentialProviders: no valid providers in chain.

Cause

The credentials-velero file used to create the Secret object is incorrectly formatted.

Solution

Ensure that the credentials-velero file is correctly formatted, as in the following example:

Example credentials-velero file

[default]
1 

aws_access_key_id=AKIAIOSFODNN7EXAMPLE
2 

aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Copy to Clipboard Toggle word wrap

1
AWS default profile.
2
Do not enclose the values with quotation marks (", ').

4.5.7. Backup and Restore CR issues
Link kopieren

You might encounter these common issues with Backup and Restore custom resources (CRs).

4.5.7.1. Backup CR cannot retrieve volume
Link kopieren

The Backup CR displays the error message, InvalidVolume.NotFound: The volume ‘vol-xxxx’ does not exist.

Cause

The persistent volume (PV) and the snapshot locations are in different regions.

Solution

  1. Edit the value of the spec.snapshotLocations.velero.config.region key in the DataProtectionApplication manifest so that the snapshot location is in the same region as the PV.
  2. Create a new Backup CR.

4.5.7.2. Backup CR status remains in progress
Link kopieren

The status of a Backup CR remains in the InProgress phase and does not complete.

Cause

If a backup is interrupted, it cannot be resumed.

Solution

  1. Retrieve the details of the Backup CR: 

    $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
  backup describe <backup>
    Copy to Clipboard Toggle word wrap

  2. Delete the Backup CR: 

    $ oc delete backup <backup> -n openshift-adp
    Copy to Clipboard Toggle word wrap

    You do not need to clean up the backup location because a Backup CR in progress has not uploaded files to object storage.

  3. Create a new Backup CR.

4.5.7.3. Backup CR status remains in PartiallyFailed
Link kopieren

The status of a Backup CR without Restic in use remains in the PartiallyFailed phase and does not complete. A snapshot of the affiliated PVC is not created.

Cause

If the backup is created based on the CSI snapshot class, but the label is missing, CSI snapshot plugin fails to create a snapshot. As a result, the Velero pod logs an error similar to the following:

+ 

time="2023-02-17T16:33:13Z" level=error msg="Error backing up item" backup=openshift-adp/user1-backup-check5 error="error executing custom action (groupResource=persistentvolumeclaims, namespace=busy1, name=pvc1-user1): rpc error: code = Unknown desc = failed to get volumesnapshotclass for storageclass ocs-storagecluster-ceph-rbd: failed to get volumesnapshotclass for provisioner openshift-storage.rbd.csi.ceph.com, ensure that the desired volumesnapshot class has the velero.io/csi-volumesnapshot-class label" logSource="/remote-source/velero/app/pkg/backup/backup.go:417" name=busybox-79799557b5-vprq
Copy to Clipboard Toggle word wrap

Solution

  1. Delete the Backup CR: 

    $ oc delete backup <backup> -n openshift-adp
    Copy to Clipboard Toggle word wrap
  2. If required, clean up the stored data on the BackupStorageLocation to free up space.

  3. Apply label velero.io/csi-volumesnapshot-class=true to the VolumeSnapshotClass object: 

    $ oc label volumesnapshotclass/<snapclass_name> velero.io/csi-volumesnapshot-class=true
    Copy to Clipboard Toggle word wrap
  4. Create a new Backup CR.

4.5.8. Restic issues
Link kopieren

You might encounter these issues when you back up applications with Restic.

The Restic pod log displays the error message: controller=pod-volume-backup error="fork/exec/usr/bin/restic: permission denied".

Cause

If your NFS data volumes have root_squash enabled, Restic maps to nfsnobody and does not have permission to create backups.

Solution

You can resolve this issue by creating a supplemental group for Restic and adding the group ID to the DataProtectionApplication manifest:

  1. Create a supplemental group for Restic on the NFS data volume.
  2. Set the setgid bit on the NFS directories so that group ownership is inherited.

  3. Add the spec.configuration.restic.supplementalGroups parameter and the group ID to the DataProtectionApplication manifest, as in the following example: 

    spec:
  configuration:
    restic:
      enable: true
      supplementalGroups:
      - <group_id>
    1
    Copy to Clipboard Toggle word wrap
    1
    Specify the supplemental group ID.
  4. Wait for the Restic pods to restart so that the changes are applied.

4.5.8.2. Restic Backup CR cannot be recreated after bucket is emptied
Link kopieren

If you create a Restic Backup CR for a namespace, empty the object storage bucket, and then recreate the Backup CR for the same namespace, the recreated Backup CR fails.

The velero pod log displays the following error message: stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?.

Cause

Velero does not recreate or update the Restic repository from the ResticRepository manifest if the Restic directories are deleted from object storage. See Velero issue 4421 for more information.

Solution

  • Remove the related Restic repository from the namespace by running the following command: 

    $ oc delete resticrepository openshift-adp <name_of_the_restic_repository>
    Copy to Clipboard Toggle word wrap

    In the following error log, mysql-persistent is the problematic Restic repository. The name of the repository appears in italics for clarity. 

     time="2021-12-29T18:29:14Z" level=info msg="1 errors
 encountered backup up item" backup=velero/backup65
 logSource="pkg/backup/backup.go:431" name=mysql-7d99fc949-qbkds
 time="2021-12-29T18:29:14Z" level=error msg="Error backing up item"
 backup=velero/backup65 error="pod volume backup failed: error running
 restic backup, stderr=Fatal: unable to open config file: Stat: The
 specified key does not exist.\nIs there a repository at the following
 location?\ns3:http://minio-minio.apps.mayap-oadp-
 veleo-1234.qe.devcluster.openshift.com/mayapvelerooadp2/velero1/
 restic/mysql-persistent\n: exit status 1" error.file="/remote-source/
 src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:184"
 error.function="github.com/vmware-tanzu/velero/
 pkg/restic.(*backupper).BackupPodVolumes"
 logSource="pkg/backup/backup.go:435" name=mysql-7d99fc949-qbkds
    Copy to Clipboard Toggle word wrap

4.5.9. Using the must-gather tool
Link kopieren

You can collect logs, metrics, and information about OADP custom resources by using the must-gather tool.

The must-gather data must be attached to all customer cases.

Prerequisites

  • You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.
  • You must have the OpenShift CLI (oc) installed.

Procedure

  1. Navigate to the directory where you want to store the must-gather data.

  2. Run the oc adm must-gather command for one of the following data collection options: 

    $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.1
    Copy to Clipboard Toggle word wrap

    The data is saved as must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal. 

    $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.1 \
  -- /usr/bin/gather_metrics_dump
    Copy to Clipboard Toggle word wrap

    This operation can take a long time. The data is saved as must-gather/metrics/prom_data.tar.gz.

Viewing metrics data with the Prometheus console

You can view the metrics data with the Prometheus console.

Procedure

  1. Decompress the prom_data.tar.gz file: 

    $ tar -xvzf must-gather/metrics/prom_data.tar.gz
    Copy to Clipboard Toggle word wrap

  2. Create a local Prometheus instance: 

    $ make prometheus-run
    Copy to Clipboard Toggle word wrap

    The command outputs the Prometheus URL.

    Output

    Started Prometheus on http://localhost:9090
    Copy to Clipboard Toggle word wrap

  3. Launch a web browser and navigate to the URL to view the data by using the Prometheus web console.

  4. After you have viewed the data, delete the Prometheus instance and data: 

    $ make prometheus-cleanup
    Copy to Clipboard Toggle word wrap

4.6. APIs used with OADP
Link kopieren

The document provides information about the following APIs that you can use with OADP:

  • Velero API
  • OADP API

4.6.1. Velero API
Link kopieren

Velero API documentation is maintained by Velero, not by Red Hat. It can be found at Velero API types.

4.6.2. OADP API
Link kopieren

The following tables provide the structure of the OADP API:

Expand
Table 4.3. DataProtectionApplicationSpec
PropertyTypeDescription

backupLocations

[] BackupLocation

Defines the list of configurations to use for BackupStorageLocations.

snapshotLocations

[] SnapshotLocation

Defines the list of configurations to use for VolumeSnapshotLocations.

unsupportedOverrides

map [ UnsupportedImageKey ] string

Can be used to override the deployed dependent images for development. Options are veleroImageFqin, awsPluginImageFqin, openshiftPluginImageFqin, azurePluginImageFqin, gcpPluginImageFqin, csiPluginImageFqin, dataMoverImageFqin, resticRestoreImageFqin, kubevirtPluginImageFqin, and operator-type.

podAnnotations

map [ string ] string

Used to add annotations to pods deployed by Operators.

podDnsPolicy

DNSPolicy

Defines the configuration of the DNS of a pod.

podDnsConfig

PodDNSConfig

Defines the DNS parameters of a pod in addition to those generated from DNSPolicy.

backupImages

*bool

Used to specify whether or not you want to deploy a registry for enabling backup and restore of images.

configuration

*ApplicationConfig

Used to define the data protection application’s server configuration.

features

*Features

Defines the configuration for the DPA to enable the Technology Preview features.

Complete schema definitions for the OADP API.

Expand
Table 4.4. BackupLocation
PropertyTypeDescription

velero

*velero.BackupStorageLocationSpec

Location to store volume snapshots, as described in Backup Storage Location.

bucket

*CloudStorageLocation

[Technology Preview] Automates creation of a bucket at some cloud storage providers for use as a backup storage location.

Important

The bucket parameter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Complete schema definitions for the type BackupLocation.

Expand
Table 4.5. SnapshotLocation
PropertyTypeDescription

velero

*VolumeSnapshotLocationSpec

Location to store volume snapshots, as described in Volume Snapshot Location.

Complete schema definitions for the type SnapshotLocation.

Expand
Table 4.6. ApplicationConfig
PropertyTypeDescription

velero

*VeleroConfig

Defines the configuration for the Velero server.

restic

*ResticConfig

Defines the configuration for the Restic server.

Complete schema definitions for the type ApplicationConfig.

Expand
Table 4.7. VeleroConfig
PropertyTypeDescription

featureFlags

[] string

Defines the list of features to enable for the Velero instance.

defaultPlugins

[] string

The following types of default Velero plugins can be installed: aws,azure, csi, gcp, kubevirt, and openshift.

customPlugins

[]CustomPlugin

Used for installation of custom Velero plugins.

Default and custom plugins are described in OADP plugins

restoreResourcesVersionPriority

string

Represents a config map that is created if defined for use in conjunction with the EnableAPIGroupVersions feature flag. Defining this field automatically adds EnableAPIGroupVersions to the Velero server feature flag.

noDefaultBackupLocation

bool

To install Velero without a default backup storage location, you must set the noDefaultBackupLocation flag in order to confirm installation.

podConfig

*PodConfig

Defines the configuration of the Velero pod.

logLevel

string

Velero server’s log level (use debug for the most granular logging, leave unset for Velero default). Valid options are trace, debug, info, warning, error, fatal, and panic.

Complete schema definitions for the type VeleroConfig.

Expand
Table 4.8. CustomPlugin
PropertyTypeDescription

name

string

Name of custom plugin.

image

string

Image of custom plugin.

Complete schema definitions for the type CustomPlugin.

Expand
Table 4.9. ResticConfig
PropertyTypeDescription

enable

*bool

If set to true, enables backup and restore using Restic. If set to false, snapshots are needed.

supplementalGroups

[]int64

Defines the Linux groups to be applied to the Restic pod.

timeout

string

A user-supplied duration string that defines the Restic timeout. Default value is 1hr (1 hour). A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as 300ms, -1.5h` or 2h45m. Valid time units are ns, us (or µs), ms, s, m, and h.

podConfig

*PodConfig

Defines the configuration of the Restic pod.

Complete schema definitions for the type ResticConfig.

Expand
Table 4.10. PodConfig
PropertyTypeDescription

nodeSelector

map [ string ] string

Defines the nodeSelector to be supplied to a Velero podSpec or a Restic podSpec.

tolerations

[]Toleration

Defines the list of tolerations to be applied to a Velero deployment or a Restic daemonset.

resourceAllocations

ResourceRequirements

Set specific resource limits and requests for a Velero pod or a Restic pod as described in Setting Velero CPU and memory resource allocations.

labels

map [ string ] string

Labels to add to pods.

Complete schema definitions for the type PodConfig.

Expand
Table 4.11. Features
PropertyTypeDescription

dataMover

*DataMover

Defines the configuration of the Data Mover.

Complete schema definitions for the type Features.

Expand
Table 4.12. DataMover
PropertyTypeDescription

enable

bool

If set to true, deploys the volume snapshot mover controller and a modified CSI Data Mover plugin. If set to false, these are not deployed.

credentialName

string

User-supplied Restic Secret name for Data Mover.

timeout

string

A user-supplied duration string for VolumeSnapshotBackup and VolumeSnapshotRestore to complete. Default is 10m (10 minutes). A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as 300ms, -1.5h` or 2h45m. Valid time units are ns, us (or µs), ms, s, m, and h.

The OADP API is more fully detailed in OADP Operator.

4.7. Advanced OADP features and functionalities
Link kopieren

This document provides information on advanced features and functionalities of OpenShift API for Data Protection (OADP).

4.7.1.1. Listing the Kubernetes API group versions on a cluster
Link kopieren

A source cluster might offer multiple versions of an API, where one of these versions is the preferred API version. For example, a source cluster with an API named Example might be available in the example.com/v1 and example.com/v1beta2 API groups.

If you use Velero to back up and restore such a source cluster, Velero backs up only the version of that resource that uses the preferred version of its Kubernetes API.

To return to the above example, if example.com/v1 is the preferred API, then Velero only backs up the version of a resource that uses example.com/v1. Moreover, the target cluster needs to have example.com/v1 registered in its set of available API resources in order for Velero to restore the resource on the target cluster.

Therefore, you need to generate a list of the Kubernetes API group versions on your target cluster to be sure the prefered API version is registered in its set of available API resources.

Procedure

  • Enter the following command: 
$ oc api-resources
Copy to Clipboard Toggle word wrap

4.7.1.2. About Enable API Group Versions
Link kopieren

By default, Velero only backs up resources that use the preferred version of the Kubernetes API. However, Velero also includes a feature, Enable API Group Versions, that overcomes this limitation. When enabled on the source cluster, this feature causes Velero to back up all Kubernetes API group versions that are supported on the cluster, not only the preferred one. After the versions are stored in the backup .tar file, they are available to be restored on the destination cluster.

For example, a source cluster with an API named Example might be available in the example.com/v1 and example.com/v1beta2 API groups, with example.com/v1 being the preferred API.

Without the Enable API Group Versions feature enabled, Velero backs up only the preferred API group version for Example, which is example.com/v1. With the feature enabled, Velero also backs up example.com/v1beta2.

When the Enable API Group Versions feature is enabled on the destination cluster, Velero selects the version to restore on the basis of the order of priority of API group versions.

Note

Enable API Group Versions is still in beta.

Velero uses the following algorithm to assign priorities to API versions, with 1 as the top priority:

  1. Preferred version of the destination cluster
  2. Preferred version of the source_ cluster
  3. Common non-preferred supported version with the highest Kubernetes version priority

4.7.1.3. Using Enable API Group Versions
Link kopieren

You can use Velero’s Enable API Group Versions feature to back up all Kubernetes API group versions that are supported on a cluster, not only the preferred one.

Note

Enable API Group Versions is still in beta.

Procedure

  • Configure the EnableAPIGroupVersions feature flag: 
apiVersion: oadp.openshift.io/vialpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      featureFlags:
      - EnableAPIGroupVersions
Copy to Clipboard Toggle word wrap
Nach oben
Red Hat logoGithubredditYoutubeTwitter

Lernen

Testen, kaufen und verkaufen

Communitys

Über Red Hat Dokumentation

Wir helfen Red Hat Benutzern, mit unseren Produkten und Diensten innovativ zu sein und ihre Ziele zu erreichen – mit Inhalten, denen sie vertrauen können. Entdecken Sie unsere neuesten Updates.

Mehr Inklusion in Open Source

Red Hat hat sich verpflichtet, problematische Sprache in unserem Code, unserer Dokumentation und unseren Web-Eigenschaften zu ersetzen. Weitere Einzelheiten finden Sie in Red Hat Blog.

Über Red Hat

Wir liefern gehärtete Lösungen, die es Unternehmen leichter machen, plattform- und umgebungsübergreifend zu arbeiten, vom zentralen Rechenzentrum bis zum Netzwerkrand.

Theme

© 2025 Red Hat