Backing up and restoring applications


Red Hat OpenShift Service on AWS 4

Backing up and restoring of applications data

Red Hat OpenShift Documentation Team

Abstract

This document provides information about backing up applications.

Chapter 1. Backing up applications

You can employ OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS (ROSA) clusters to backup and restore application data. Before installing OADP, you must set up role and policy credentials for OADP so that it can use the AWS API.

This is a two stage process:

  1. Prepare AWS credentials.
  2. Install the OADP Operator and provide it with the IAM role.

1.1. Preparing AWS credentials

An AWS account must be ready to accept an OADP installation.

Procedure

  1. Create the following environment variables by running the following commands:

    Note

    Change the cluster name to match your ROSA cluster, and ensure you are logged into the cluster as an administrator. Ensure that all fields are outputted correctly before continuing.

    $ export CLUSTER_NAME=my-cluster 1
    export ROSA_CLUSTER_ID=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .id)
    export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)
    export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')
    export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    export CLUSTER_VERSION=$(rosa describe cluster -c ${CLUSTER_NAME} -o json | jq -r .version.raw_id | cut -f -2 -d '.')
    export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"
    export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"
    mkdir -p ${SCRATCH}
    echo "Cluster ID: ${ROSA_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint:
    ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"
    1
    Replace my-cluster with your ROSA cluster name.
  2. On the AWS account, create an IAM policy to allow access to S3.

    1. Check to see if the policy exists by running the following command:

      $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text) 1
      1
      Replace RosaOadp with your policy name.
    2. Use the following command to create the policy JSON file and then create the policy in ROSA.

      Note

      If the policy ARN is not found, the command will create the policy. If the policy ARN already exists, the if statement will intentionally skip the policy creation.

      $ if [[ -z "${POLICY_ARN}" ]]; then
      cat << EOF > ${SCRATCH}/policy.json 1
      {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:CreateBucket",
            "s3:DeleteBucket",
            "s3:PutBucketTagging",
            "s3:GetBucketTagging",
            "s3:PutEncryptionConfiguration",
            "s3:GetEncryptionConfiguration",
            "s3:PutLifecycleConfiguration",
            "s3:GetLifecycleConfiguration",
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:ListBucketMultipartUploads",
            "s3:AbortMultipartUpload",
            "s3:ListMultipartUploadParts",
            "ec2:DescribeSnapshots",
            "ec2:DescribeVolumes",
            "ec2:DescribeVolumeAttribute",
            "ec2:DescribeVolumesModifications",
            "ec2:DescribeVolumeStatus",
            "ec2:CreateTags",
            "ec2:CreateVolume",
            "ec2:CreateSnapshot",
            "ec2:DeleteSnapshot"
          ],
          "Resource": "*"
        }
       ]}
      EOF
      
      POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \
      --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \
      --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \
      --output text)
      fi
      1
      SCRATCH is a name for a temporary directory created for the environment variables.
    3. View the policy ARN by running the following command:

      $ echo ${POLICY_ARN}
  3. Create an IAM role trust policy for the cluster:

    1. Create the trust policy file by running the following command:

      $ cat <<EOF > ${SCRATCH}/trust-policy.json
      {
          "Version": "2012-10-17",
          "Statement": [{
            "Effect": "Allow",
            "Principal": {
              "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
              "StringEquals": {
                "${OIDC_ENDPOINT}:sub": [
                  "system:serviceaccount:openshift-adp:openshift-adp-controller-manager",
                  "system:serviceaccount:openshift-adp:velero"]
              }
            }
          }]
      }
      EOF
    2. Create the role by running the following command:

      $ ROLE_ARN=$(aws iam create-role --role-name \
        "${ROLE_NAME}" \
        --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
        --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID}
      Key=rosa_openshift_version,Value=${CLUSTER_VERSION}
      Key=rosa_role_prefix,Value=ManagedOpenShift
      Key=operator_namespace,Value=openshift-adp
      Key=operator_name,Value=openshift-oadp \
         --query Role.Arn --output text)
    3. View the role ARN by running the following command:

      $ echo ${ROLE_ARN}
  4. Attach the IAM policy to the IAM role by running the following command:

    $ aws iam attach-role-policy --role-name "${ROLE_NAME}" \
      --policy-arn ${POLICY_ARN}

Next steps

  • Continue to Installing the OADP Operator and providing the IAM role.

1.2. Installing the OADP Operator and providing the IAM role

AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. Red Hat OpenShift Service on AWS (ROSA) with STS is the recommended credential mode for ROSA clusters. This document describes how to install OpenShift API for Data Protection (OADP) on ROSA with AWS STS.

Important

Restic is unsupported.

Kopia file system backup (FSB) is supported when backing up file systems that do not have Container Storage Interface (CSI) snapshotting support.

Example file systems include the following:

  • Amazon Elastic File System (EFS)
  • Network File System (NFS)
  • emptyDir volumes
  • Local volumes

For backing up volumes, OADP on ROSA with AWS STS supports only native snapshots and Container Storage Interface (CSI) snapshots.

In an Amazon ROSA cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.

The Data Mover feature is not currently supported in ROSA clusters. You can use native AWS S3 tools for moving data.

Prerequisites

  • An Red Hat OpenShift Service on AWS ROSA cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including ROLE_ARN, for each cluster.

Procedure

  1. Create an Red Hat OpenShift Service on AWS secret from your AWS token file by entering the following commands:

    1. Create the credentials file:

      $ cat <<EOF > ${SCRATCH}/credentials
        [default]
        role_arn = ${ROLE_ARN}
        web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
      EOF
    2. Create a namespace for OADP:

      $ oc create namespace openshift-adp
    3. Create the Red Hat OpenShift Service on AWS secret:

      $ oc -n openshift-adp create secret generic cloud-credentials \
        --from-file=${SCRATCH}/credentials
      Note

      In Red Hat OpenShift Service on AWS versions 4.15 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the Red Hat OpenShift Service on AWS web console, for more information see Installing from OperatorHub using the web console.

      The preceding secret is created automatically by CCO.

  2. Install the OADP Operator:

    1. In the Red Hat OpenShift Service on AWS web console, browse to OperatorsOperatorHub.
    2. Search for the OADP Operator.
    3. In the role_ARN field, paste the role_arn that you created previously and click Install.
  3. Create AWS cloud storage using your AWS credentials by entering the following command:

    $ cat << EOF | oc create -f -
      apiVersion: oadp.openshift.io/v1alpha1
      kind: CloudStorage
      metadata:
        name: ${CLUSTER_NAME}-oadp
        namespace: openshift-adp
      spec:
        creationSecret:
          key: credentials
          name: cloud-credentials
        enableSharedConfig: true
        name: ${CLUSTER_NAME}-oadp
        provider: aws
        region: $REGION
    EOF
  4. Check your application’s storage default storage class by entering the following command:

    $ oc get pvc -n <namespace>

    Example output

    NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    applog   Bound    pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8   1Gi        RWO            gp3-csi        4d19h
    mysql    Bound    pvc-16b8e009-a20a-4379-accc-bc81fedd0621   1Gi        RWO            gp3-csi        4d19h

  5. Get the storage class by running the following command:

    $ oc get storageclass

    Example output

    NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   4d21h
    gp2-csi             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
    gp3                 ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
    gp3-csi (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h

    Note

    The following storage classes will work:

    • gp3-csi
    • gp2-csi
    • gp3
    • gp2

    If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.

  6. Create the DataProtectionApplication resource to configure the connection to the storage where the backups and volume snapshots are stored:

    1. If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:

      $ cat << EOF | oc create -f -
        apiVersion: oadp.openshift.io/v1alpha1
        kind: DataProtectionApplication
        metadata:
          name: ${CLUSTER_NAME}-dpa
          namespace: openshift-adp
        spec:
          backupImages: true 1
          features:
            dataMover:
              enable: false
          backupLocations:
          - bucket:
              cloudStorageRef:
                name: ${CLUSTER_NAME}-oadp
              credential:
                key: credentials
                name: cloud-credentials
              prefix: velero
              default: true
              config:
                region: ${REGION}
          configuration:
            velero:
              defaultPlugins:
              - openshift
              - aws
              - csi
            restic:
              enable: false
      EOF
      1
      ROSA supports internal image backup. Set this field to false if you do not want to use image backup.
  1. If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:

    $ cat << EOF | oc create -f -
      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        name: ${CLUSTER_NAME}-dpa
        namespace: openshift-adp
      spec:
        backupImages: true 1
        features:
          dataMover:
             enable: false
        backupLocations:
        - bucket:
            cloudStorageRef:
              name: ${CLUSTER_NAME}-oadp
            credential:
              key: credentials
              name: cloud-credentials
            prefix: velero
            default: true
            config:
              region: ${REGION}
        configuration:
          velero:
            defaultPlugins:
            - openshift
            - aws
          nodeAgent: 2
            enable: false
            uploaderType: restic
        snapshotLocations:
          - velero:
              config:
                credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials 3
                enableSharedConfig: "true" 4
                profile: default 5
                region: ${REGION} 6
              provider: aws
    EOF
    1
    ROSA supports internal image backup. Set this field to false if you do not want to use image backup.
    2
    See the important note regarding the nodeAgent attribute.
    3
    The credentialsFile field is the mounted location of the bucket credential on the pod.
    4
    The enableSharedConfig field allows the snapshotLocations to share or reuse the credential defined for the bucket.
    5
    Use the profile name set in the AWS credentials file.
    6
    Specify region as your AWS region. This must be the same as the cluster region.

    You are now ready to back up and restore Red Hat OpenShift Service on AWS applications, as described in Backing up applications.

Important

The enable parameter of restic is set to false in this configuration, because OADP does not support Restic in ROSA environments.

If you use OADP 1.2, replace this configuration:

nodeAgent:
  enable: false
  uploaderType: restic

with the following configuration:

restic:
  enable: false

If you want to use two different clusters for backing up and restoring, the two clusters must have the same AWS S3 storage names in both the cloud storage CR and the OADP DataProtectionApplication configuration.

Additional resources

1.3. Backing up workloads on OADP with ROSA STS

1.3.1. Performing a backup with OADP and ROSA STS

The following example hello-world application has no persistent volumes (PVs) attached. Perform a backup with OpenShift API for Data Protection (OADP) with Red Hat OpenShift Service on AWS (ROSA) STS.

Either Data Protection Application (DPA) configuration will work.

  1. Create a workload to back up by running the following commands:

    $ oc create namespace hello-world
    $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift
  2. Expose the route by running the following command:

    $ oc expose service/hello-openshift -n hello-world
  3. Check that the application is working by running the following command:

    $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

    Example output

    Hello OpenShift!

  4. Back up the workload by running the following command:

    $ cat << EOF | oc create -f -
      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: hello-world
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - hello-world
        storageLocation: ${CLUSTER_NAME}-dpa-1
        ttl: 720h0m0s
    EOF
  5. Wait until the backup is completed and then run the following command:

    $ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"

    Example output

    {
      "completionTimestamp": "2022-09-07T22:20:44Z",
      "expiration": "2022-10-07T22:20:22Z",
      "formatVersion": "1.1.0",
      "phase": "Completed",
      "progress": {
        "itemsBackedUp": 58,
        "totalItems": 58
      },
      "startTimestamp": "2022-09-07T22:20:22Z",
      "version": 1
    }

  6. Delete the demo workload by running the following command:

    $ oc delete ns hello-world
  7. Restore the workload from the backup by running the following command:

    $ cat << EOF | oc create -f -
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: hello-world
        namespace: openshift-adp
      spec:
        backupName: hello-world
    EOF
  8. Wait for the Restore to finish by running the following command:

    $ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"

    Example output

    {
      "completionTimestamp": "2022-09-07T22:25:47Z",
      "phase": "Completed",
      "progress": {
        "itemsRestored": 38,
        "totalItems": 38
      },
      "startTimestamp": "2022-09-07T22:25:28Z",
      "warnings": 9
    }

  9. Check that the workload is restored by running the following command:

    $ oc -n hello-world get pods

    Example output

    NAME                              READY   STATUS    RESTARTS   AGE
    hello-openshift-9f885f7c6-kdjpj   1/1     Running   0          90s

  10. Check the JSONPath by running the following command:

    $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

    Example output

    Hello OpenShift!

Note

For troubleshooting tips, see the OADP team’s troubleshooting documentation.

1.3.2. Cleaning up a cluster after a backup with OADP and ROSA STS

If you need to uninstall the OpenShift API for Data Protection (OADP) Operator together with the backups and the S3 bucket from this example, follow these instructions.

Procedure

  1. Delete the workload by running the following command:

    $ oc delete ns hello-world
  2. Delete the Data Protection Application (DPA) by running the following command:

    $ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa
  3. Delete the cloud storage by running the following command:

    $ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp
    Warning

    If this command hangs, you might need to delete the finalizer by running the following command:

    $ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge
  4. If the Operator is no longer required, remove it by running the following command:

    $ oc -n openshift-adp delete subscription oadp-operator
  5. Remove the namespace from the Operator:

    $ oc delete ns openshift-adp
  6. If the backup and restore resources are no longer required, remove them from the cluster by running the following command:

    $ oc delete backup hello-world
  7. To delete backup, restore and remote objects in AWS S3 run the following command:

    $ velero backup delete hello-world
  8. If you no longer need the Custom Resource Definitions (CRD), remove them from the cluster by running the following command:

    $ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done
  9. Delete the AWS S3 bucket by running the following commands:

    $ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive
    $ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp
  10. Detach the policy from the role by running the following command:

    $ aws iam detach-role-policy --role-name "${ROLE_NAME}"  --policy-arn "${POLICY_ARN}"
  11. Delete the role by running the following command:

    $ aws iam delete-role --role-name "${ROLE_NAME}"

1.4. Known issues

1.5. Additional resources

Legal Notice

Copyright © 2024 Red Hat, Inc.

OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

Modified versions must remove all Red Hat trademarks.

Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.