Chapter 5. Backing up and restoring Red Hat Quay managed by the Red Hat Quay Operator

Procedure

1. Enter the following command to obtain a list of Red Hat Quay pods:

   $ oc get pods -n <namespace>

   Example output

   example-registry-clair-app-7dc7ff5844-4skw5           0/1     Error                    0             70d
   example-registry-clair-app-7dc7ff5844-nvn4f           1/1     Running                  0             31d
   example-registry-clair-app-7dc7ff5844-x4smw           0/1     ContainerStatusUnknown   6 (70d ago)   70d
   example-registry-clair-app-7dc7ff5844-xjnvt           1/1     Running                  0             60d
   example-registry-clair-postgres-547d75759-75c49       1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-52wjz            1/1     Running                  0             70d
   example-registry-quay-app-76c8f55467-hwz4c            1/1     Running                  0             70d
   example-registry-quay-app-upgrade-57ghs               0/1     Completed                1             70d
   example-registry-quay-database-7c55899f89-hmnm6       1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-btsnb          1/1     Running                  0             70d
   example-registry-quay-mirror-6cccbd76d-x8g42          1/1     Running                  0             70d
   example-registry-quay-redis-85cbdf96bf-4vk5m          1/1     Running                  0             70d

2. Open a remote shell session to the Quay container by entering the following command:

   $ oc rsh example-registry-quay-app-76c8f55467-52wjz

3. Create the necessary service keys by entering the following command:

   sh-4.4$ python3 tools/generatekeypair.py quay-readonly

   Example output

   Writing public key to quay-readonly.jwk
   Writing key ID to quay-readonly.kid
   Writing private key to quay-readonly.pem

Procedure

1. Enter the following command to enter your Red Hat Quay database environment:

   $ oc rsh example-registry-quay-database-76c8f55467-52wjz psql -U <database_username> -d <database_name>

2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

   quay=# select * from servicekeyapproval;

   Example output

    id | approver_id |          approval_type           |       approved_date        | notes
   ----+-------------+----------------------------------+----------------------------+-------
     1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
     2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
     3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
     4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
     5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
   ...

3. Add the service key to your Red Hat Quay database by entering the following query:

   quay=# INSERT INTO servicekey
     (name, service, metadata, kid, jwk, created_date, expiration_date)
     VALUES ('quay-readonly',
              'quay',
              '{}',
              '{<contents_of_.kid_file>}',
              '{<contents_of_.jwk_file>}',
              '{<created_date_of_read-only>}',
              '{<expiration_date_of_read-only>}');

   Example output

   INSERT 0 1

4. Next, add the key approval with the following query:

   quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
     VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
              {include_notes_here_on_why_this_is_being_added});

   Example output

   INSERT 0 1

5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

   UPDATE servicekey
   SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
   WHERE name = 'quay-readonly';

   Example output

   UPDATE 1

Procedure

1. Read the secret name of your Red Hat Quay on OpenShift Container Platform deployment by entering the following command:

   $ oc get deployment -o yaml <quay_main_app_deployment_name>

2. Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files by entering the following commands:

   $ base64 -w0 quay-readonly.kid

   Example output

   ZjUyNDFm...

   $ base64 -w0 quay-readonly.pem

   Example output

   LS0tLS1CRUdJTiBSU0E...

3. Obtain the current configuration bundle and secret by entering the following command. Save the output to a file called config.yaml:

   $ oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml

4. Edit the config.yaml file and add the following information to enable read-only mode:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

5. Save the file and base64 encode it by entering the following command:

   $ base64 -w0 quay-config.yaml

6. Scale down the Red Hat Quay Operator pods to 0 by entering the following command. This ensures that the Operator does not reconcile the secret after editing it.

   $ oc scale --replicas=0 deployment quay-operator -n openshift-operators

7. Edit the secret to include the new content by entering the following command:

   $ oc edit secret quay-config-secret-name -n quay-namespace

   # ...
   data:
     "quay-readonly.kid": "ZjUyNDFm..."
     "quay-readonly.pem": "LS0tLS1CRUdJTiBSU0E..."
     "config.yaml": "QUNUSU9OX0xPR19..."
   # ...

   With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

Procedure

1. Edit the config.yaml file and remove the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

2. Scale the Red Hat Quay Operator back up by entering the following command:

   $ oc scale --replicas=1 deployment quay-operator -n openshift-operators

Procedure

1. To back the QuayRegistry custom resource by exporting it, enter the following command:

   $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml

2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

     metadata.creationTimestamp
     metadata.finalizers
     metadata.generation
     metadata.resourceVersion
     metadata.uid

3. Backup the managed keys secret by entering the following command:

   Note

   If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called <quay_registry_name>-quay-registry-managed-secret-keys in the namespace of the QuayRegistry resource.

   $ oc get secret -n <quay_namespace> <quay_registry_name>-quay-registry-managed-secret-keys -o yaml > managed_secret_keys.yaml

4. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: <quay_registry_name>-quay-registry-managed-secret-keys
     namespace: <quay_namespace>
   data:
     CONFIG_EDITOR_PW: <redacted>
     DATABASE_SECRET_KEY: <redacted>
     DB_ROOT_PW: <redacted>
     DB_URI: <redacted>
     SECRET_KEY: <redacted>
     SECURITY_SCANNER_V4_PSK: <redacted>

   All information under the data property should remain the same.

5. Redirect the current Quay configuration file by entering the following command:

   $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml

6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

   $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml

7. Obtain the Quay database name:

   $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'

   Example output

   quayregistry-quay-database

Procedure

1. Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false
       - kind: quay
         managed: true
         overrides:
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   where:

   managed: false:: Disables auto scaling of Quay, Clair and Mirroring workers.

   overrides:: Sets the replica count to 0 for components accessing the database and objectstorage.

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by entering the following command:

   $ oc get pods -n <quay_namespace>

   Example output:

   $ oc get pod

   Example output

   quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
   quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
   quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
   quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
   quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

Procedure

1. Identify the Red Hat Quay PostgreSQL pod name by entering the following command:

   $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

2. Download a backup database by entering the following command:

   $ oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

Procedure

1. Decode and export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

2. Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

3. Create a new directory by entering the following command:

   $ mkdir blobs

4. Copy all blobs to the directory by entering the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

Procedure

1. Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: true 

   1

   
       - kind: quay 

   2

   
         managed: true
       - kind: clair
         managed: true
       - kind: mirror
         managed: true
       …

   where:

   managed: true:: Re-enables auto scaling of Quay, Clair and Mirroring workers again.

   overrides:: Replica overrides are removed again to scale the Quay components back up.

2. Check the status of the Red Hat Quay deployment by entering the following command:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>

   Example output:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     ...
     name: registry
     namespace: <quay-namespace>
     ...
   spec:
     ...
   status:
     - lastTransitionTime: '2022-06-20T05:31:17Z'
       lastUpdateTime: '2022-06-20T17:31:13Z'
       message: All components reporting as healthy
       reason: HealthChecksPassing
       status: 'True'
       type: Available

Procedure

1. Restore the backed up Red Hat Quay configuration by entering the following command:

   $ oc create -f ./config-bundle.yaml

   Important

   If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

2. Restore the generated keys from the backup by entering the following command:

   $ oc create -f ./managed-secret-keys.yaml

3. Restore the QuayRegistry custom resource by entering the following command:

   $ oc create -f ./quay-registry.yaml

4. Check the status of the Red Hat Quay deployment by entering the following command. Wait for it to be available:

   $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

Procedure

1. Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). For example:

   apiVersion: quay.redhat.com/v1
   kind: QuayRegistry
   metadata:
     name: registry
     namespace: ns
   spec:
     components:
       …
       - kind: horizontalpodautoscaler
         managed: false 

   1

   
       - kind: quay
         managed: true
         overrides: 

   2

   
           replicas: 0
       - kind: clair
         managed: true
         overrides:
           replicas: 0
       - kind: mirror
         managed: true
         overrides:
           replicas: 0
       …

   where:

   managed: false:: Specifies that the component is not managed by the Red Hat Quay Operator.

   overrides:: Specifies the replica count for the component.

2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:

   $ oc get pods -n <quay-namespace>

   Example output:

   registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
   registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
   registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

Procedure

1. Identify your Quay database pod by entering the following command:

   $ oc get pod -l quay-component=postgres -n  <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

   Example output:

   quayregistry-quay-database-59f54bb7-58xs7

2. Upload the backup by copying it from the local environment and into the pod by entering the following command:

   $ oc cp ./backup.sql -n <quay_namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql

3. Open a remote terminal to the database by entering the following command:

   $ oc rsh -n <quay_namespace> registry-quay-database-66969cd859-n2ssm

4. Enter psql by entering the following command:

   bash-4.4$ psql

5. You can list the database by entering the following command:

   postgres=# \l

   Example output

                                                     List of databases
              Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
   ----------------------------+----------------------------+----------+------------+------------+-----------------------
   postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
   quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

6. Drop the existing database by entering the following command:

   postgres=# DROP DATABASE "quayregistry-quay-database";

   Example output

   DROP DATABASE

7. Exit the postgres CLI by entering the following command:

   \q

8. Redirect your PostgreSQL database to your backup database by entering the following command:

   sh-4.4$ psql < /tmp/backup.sql

9. Exit bash by entering the following command:

   sh-4.4$ exit

Procedure

1. Export the AWS_ACCESS_KEY_ID by entering the following command:

   $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)

2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

   $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)

3. Upload all blobs to the bucket by running the following command:

   $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')