Chapter 11. Backing up and restoring Red Hat Quay on an OpenShift Container Platform deployment

Procedure

1. Backup the QuayRegistry custom resource by exporting it:

   $ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml > quay-registry.yaml

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

3. Backup the managed keys secret:

   Note

   If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Quay for the first time. These are stored in a secret called <quay-registry-name>-quay-registry-managed-secret-keys in the QuayRegistry namespace.

   $ oc get secret -n <quay-namespace> <quay-registry-name>-quay-registry-managed-secret-keys -o yaml > managed-secret-keys.yaml

4. Edit the the resulting managed-secret-keys.yaml file and remove all owner references. Your managed-secret-keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quayname>-quay-registry-managed-secret-keys
     namespace: <quay-namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   All information under the data property should remain the same.

5. Backup the current Quay configuration:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay-registry-name> -n <quay-namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay-pod-name -- cat /conf/stack/config.yaml > quay-config.yaml

7. Scale down the Quay the Quay Operator:

   $  oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

8. Scale down the Quay namespace:

   $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

9. Wait for the registry-quay-app pods to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-mirror-758fc68ff7-5wxlp          1/1     Running            0          8m29s
   registry-quay-mirror-758fc68ff7-lbl82          1/1     Running            0          8m29s
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

10. Identify the Quay PostgreSQL pod name:

    $ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

    Exampe output:

1. Obtain the Quay database name:

   $ oc -n <quay-namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay-namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
   quayregistry-quay-database

2. Download a backup database:

   $ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

3. Decode and export the AWS_ACCESS_KEY_ID:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

4. Decode and export the AWS_SECRET_ACCESS_KEY_ID:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

5. Create a new directory and copy all blobs to it:

   $ mkdir blobs

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

1. Scale up the Quay the Quay Operator:

   $  oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

2. Scale up the Quay namespace:

   $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

3. Check the status of the Operator:

   $ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: example-registry
     namespace: <quay-namespace>
     ...
   spec:
     components:
     - kind: quay
       managed: true
     ...
     - kind: clairpostgres
       managed: true
     configBundleSecret: init-config-bundle-secret
   status:
     configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24
     configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org
     currentVersion: 3.7.0
     lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC
     registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org
        0          5d21h

Procedure

1. Restore the backed up Quay configuration and the randomly generated keys:

   $ oc create -f ./config-bundle.yaml

   $ oc create -f ./managed-secret-keys.yaml

   Note

   If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your exist resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

2. Restore the QuayRegistry custom resource:

   $ oc create -f ./quay-registry.yaml

3. Scale down the Quay the Quay Operator:

   $  oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

4. Scale down the Quay namespace:

   $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

5. Identify your Quay database pod:

   $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

6. Upload the backup by copying it from the local environment and into the pod:

   $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

7. Open a remote terminal to the database:

   $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm

8. Enter psql:

   bash-4.4$ psql

9. You can list the database by running the following command:

   postgres=# \l

   Example output:

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

10. Drop the database:

    postgres=# DROP DATABASE "quayregistry-quay-database";

    Example output:

    DROP DATABASE

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Redirect your PostgreSQL database to your backup database:

    sh-4.4$ psql < /tmp/backup.sql

13. Exit bash:

    sh-4.4$ exit

14. Export the AWS_ACCESS_KEY_ID:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

15. Export the AWS_SECRET_ACCESS_KEY:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

16. Upload all blobs to the bucket by running the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')

17. Scale up the Quay the Quay Operator:

    $  oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

18. Scale up the Quay namespace:

    $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>

19. Check the status of the Operator and ensure it has come back online:

    $ oc get quayregistry -n <quay-namespace> <registry-name> -o yaml

    Example output:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      ...
      name: example-registry
      namespace: quay-enterprise
      ...
    spec:
      components:
      - kind: quay
        managed: true
      ...
      - kind: clairpostgres
        managed: true
      configBundleSecret: init-config-bundle-secret
    status:
      configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24
      configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org
      currentVersion: 3.7.0
      lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC
      registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org
         0          5d21h