Chapter 9. Backing up and restoring Red Hat Quay on an OpenShift Container Platform deployment
Use the content within this section to back up and restore Red Hat Quay on an OpenShift Container Platform deployment.
9.1. Backing up Red Hat Quay
This procedure is exclusively for OpenShift Container Platform and NooBaa deployments.
Prerequisites
- A Red Hat Quay deployment on OpenShift Container Platform.
Procedure
Backup the
QuayRegistrycustom resource by exporting it:$ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml > quay-registry.yaml
Edit the resulting
quayregistry.yamland remove the status section and the following metadata fields:metadata.creationTimestamp metadata.finalizers metadata.generation metadata.resourceVersion metadata.uid
Backup the managed keys secret:
NoteIf you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Quay for the first time. These are stored in a secret called
<quay-registry-name>-quay-registry-managed-secret-keysin the QuayRegistry namespace.$ oc get secret -n <quay-namespace> <quay-registry-name>-quay-registry-managed-secret-keys -o yaml > managed-secret-keys.yaml
Edit the the resulting
managed-secret-keys.yamlfile and remove all owner references. Yourmanaged-secret-keys.yamlfile should look similar to the following:apiVersion: v1 kind: Secret type: Opaque metadata: name: <quayname>-quay-registry-managed-secret-keys namespace: <quay-namespace> data: CONFIG_EDITOR_PW: <redacted> DATABASE_SECRET_KEY: <redacted> DB_ROOT_PW: <redacted> DB_URI: <redacted> SECRET_KEY: <redacted> SECURITY_SCANNER_V4_PSK: <redacted>
All information under the
dataproperty should remain the same.Backup the current Quay configuration:
$ oc get secret -n <quay-namespace> $(oc get quayregistry <quay-registry-name> -n <quay-namespace> -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yamlBackup the
/conf/stack/config.yamlfile mounted inside of the Quay pods:$ oc exec -it quay-pod-name -- cat /conf/stack/config.yaml > quay-config.yaml
Scale down the Quay the Quay Operator:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>Scale down the Quay namespace:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>Wait for the
registry-quay-apppods to disappear. You can check their status by running the following command:$ oc get pods -n <quay-namespace>
Example output:
registry-quay-config-editor-77847fc4f5-nsbbv 1/1 Running 0 9m1s registry-quay-database-66969cd859-n2ssm 1/1 Running 0 6d1h registry-quay-mirror-758fc68ff7-5wxlp 1/1 Running 0 8m29s registry-quay-mirror-758fc68ff7-lbl82 1/1 Running 0 8m29s registry-quay-redis-7cc5f6c977-956g8 1/1 Running 0 5d21h
Identify the Quay PostgreSQL pod name:
$ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'Exampe output:
quayregistry-quay-database-59f54bb7-58xs7
Obtain the Quay database name:
$ oc -n <quay-namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay-namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}' quayregistry-quay-databaseDownload a backup database:
$ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database > backup.sql
Decode and export the
AWS_ACCESS_KEY_ID:$ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)Decode and export the
AWS_SECRET_ACCESS_KEY_ID:$ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)Create a new directory and copy all blobs to it:
$ mkdir blobs
$ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs
Scale up the Quay the Quay Operator:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>Scale up the Quay namespace:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>Check the status of the Operator:
$ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml
Example output:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: example-registry namespace: <quay-namespace> ... spec: components: - kind: quay managed: true ... - kind: clairpostgres managed: true configBundleSecret: init-config-bundle-secret status: configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24 configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org currentVersion: 3.7.0 lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org 0 5d21h
9.2. Restoring Red Hat Quay
This procedure is used to restore Red Hat Quay when the Red Hat Quay Operator manages the database. It should be performed after a backup of your Quay registry has been performed.
Prerequisites
- Red Hat Quay is deployed on OpenShift Container Platform using the Quay Operator.
- Your Red Hat Quay database has been backed up.
Procedure
Restore the backed up Quay configuration and the randomly generated keys:
$ oc create -f ./config-bundle.yaml
$ oc create -f ./managed-secret-keys.yaml
NoteIf you receive the error
Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your exist resource with$ oc delete Secret config-bundle-secret -n <quay-namespace>and recreate it with$ oc create -f ./config-bundle.yaml.Restore the QuayRegistry custom resource:
$ oc create -f ./quay-registry.yaml
Scale down the Quay the Quay Operator:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>Scale down the Quay namespace:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>Identify your Quay database pod:
$ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'Example output:
quayregistry-quay-database-59f54bb7-58xs7
Upload the backup by copying it from the local environment and into the pod:
$ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
Open a remote terminal to the database:
$ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm
Enter psql:
bash-4.4$ psql
You can list the database by running the following command:
postgres=# \l
Example output:
List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ----------------------------+----------------------------+----------+------------+------------+----------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | quayregistry-quay-database | quayregistry-quay-database | UTF8 | en_US.utf8 | en_US.utf8 |Drop the database:
postgres=# DROP DATABASE "quayregistry-quay-database";
Example output:
DROP DATABASE
Exit the postgres CLI to re-enter bash-4.4:
\q
Redirect your PostgreSQL database to your backup database:
sh-4.4$ psql < /tmp/backup.sql
Exit bash:
sh-4.4$ exit
Export the
AWS_ACCESS_KEY_ID:$ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)Export the
AWS_SECRET_ACCESS_KEY:$ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)Upload all blobs to the bucket by running the following command:
$ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') ./blobs s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')Scale up the Quay the Quay Operator:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> |awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>Scale up the Quay namespace:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay-namespace> -l quay-component=quay -o jsonpath='{.items[0].metadata.name}') -n <quay-namespace>Check the status of the Operator and ensure it has come back online:
$ oc get quayregistry -n <quay-namespace> <registry-name> -o yaml
Example output:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: example-registry namespace: quay-enterprise ... spec: components: - kind: quay managed: true ... - kind: clairpostgres managed: true configBundleSecret: init-config-bundle-secret status: configEditorCredentialsSecret: example-registry-quay-config-editor-credentials-fg2gdgtm24 configEditorEndpoint: https://example-registry-quay-config-editor-quay-enterprise.apps.docs.gcp.quaydev.org currentVersion: 3.7.0 lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org 0 5d21h
