Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 15. Backing up and restoring Red Hat Quay managed by the Red Hat Quay Operator

Use the content within this section to back up and restore Red Hat Quay when managed by the Red Hat Quay Operator on OpenShift Container Platform.

15.1. Backing up Red Hat Quay
Copy link

This procedure describes how to create a backup of Red Hat Quay deployed on OpenShift Container Platform using the Red Hat Quay Operator

Prerequisites

  • A healthy Red Hat Quay deployment on OpenShift Container Platform using the Red Hat Quay Operator (status condition Available is set to true)
  • The components quay, postgres and objectstorage are set to managed: true
  • If the component clair is set to managed: true the component clairpostgres is also set to managed: true (starting with Red Hat Quay Operator v3.7 or later)
Note

If your deployment contains partially unmanaged database or storage components and you are using external services for Postgres or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to create a backup of the data. You can refer to the tools described in this guide as a starting point on how to backup your external Postgres database or object storage.

15.1.1. Red Hat Quay configuration backup
Copy link

  1. Backup the QuayRegistry custom resource by exporting it: 

    $ oc get quayregistry <quay-registry-name> -n <quay-namespace> -o yaml > quay-registry.yaml
    Copy to Clipboard Toggle word wrap

  2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields: 

      metadata.creationTimestamp
  metadata.finalizers
  metadata.generation
  metadata.resourceVersion
  metadata.uid
    Copy to Clipboard Toggle word wrap

  3. Backup the managed keys secret:

    Note

    If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Quay for the first time. These are stored in a secret called <quay-registry-name>-quay-registry-managed-secret-keys in the namespace of the QuayRegistry resource. 

    $ oc get secret -n <quay-namespace> <quay-registry-name>-quay-registry-managed-secret-keys -o yaml > managed-secret-keys.yaml
    Copy to Clipboard Toggle word wrap

  4. Edit the the resulting managed-secret-keys.yaml file and remove the entry metadata.ownerReferences. Your managed-secret-keys.yaml file should look similar to the following: 

    apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: <quayname>-quay-registry-managed-secret-keys
  namespace: <quay-namespace>
data:
  CONFIG_EDITOR_PW: <redacted>
  DATABASE_SECRET_KEY: <redacted>
  DB_ROOT_PW: <redacted>
  DB_URI: <redacted>
  SECRET_KEY: <redacted>
  SECURITY_SCANNER_V4_PSK: <redacted>
    Copy to Clipboard Toggle word wrap

    All information under the data property should remain the same.

  5. Backup the current Quay configuration: 

    $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay-registry-name> -n <quay-namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml
    Copy to Clipboard Toggle word wrap

  6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods: 

    $ oc exec -it quay-pod-name -- cat /conf/stack/config.yaml > quay-config.yaml
    Copy to Clipboard Toggle word wrap

15.1.2. Scale down your Red Hat Quay deployment
Copy link

Important

This step is needed to create a consistent backup of the state of your Red Hat Quay deployment. Do not omit this step, including in setups where Postgres databases and/or S3-compatible object storage are provided by external services (unmanaged by the Operator).

  1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your QuayRegistry resource should look similar to the following: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: registry
  namespace: ns
spec:
  components:
    …
    - kind: horizontalpodautoscaler
      managed: false
    1 
    
    - kind: quay
      managed: true
      overrides:
    2 
    
        replicas: 0
    - kind: clair
      managed: true
      overrides:
        replicas: 0
    - kind: mirror
      managed: true
      overrides:
        replicas: 0
    …
    Copy to Clipboard Toggle word wrap
    1
    Disable auto scaling of Quay, Clair and Mirroring workers
    2
    Set the replica count to 0 for components accessing the database and objectstorage

  2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources: 

    $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

  3. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command: 

    $ oc get pods -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

    Example output: 

    $ oc get pod

quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
quayregistry-quay-config-editor-6dfdcfc44f-hlvwm   1/1     Running     0             73s
quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m
    Copy to Clipboard Toggle word wrap

15.1.3. Red Hat Quay managed database backup
Copy link

Note

If your Red Hat Quay deployment is configured with external (unmanged) Postgres database(s), refer to your vendor’s documentation on how to create a consistent backup of these databases.

  1. Identify the Quay PostgreSQL pod name: 

    $ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'
    Copy to Clipboard Toggle word wrap

    Example output: 

    quayregistry-quay-database-59f54bb7-58xs7
    Copy to Clipboard Toggle word wrap

  2. Obtain the Quay database name: 

    $ oc -n <quay-namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay-namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
quayregistry-quay-database
    Copy to Clipboard Toggle word wrap

  3. Download a backup database: 

    $ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql
    Copy to Clipboard Toggle word wrap

The instructions in this section apply to the following configurations:

  • Standalone, multi-cloud object gateway configurations
  • OpenShift Data Foundations storage requires that the Red Hat Quay Operator provisioned an S3 object storage bucket from, through the ObjectStorageBucketClaim API
Note

If your Red Hat Quay deployment is configured with external (unmanged) object storage, refer to your vendor’s documentation on how to create a copy of the content of Quay’s storage bucket.

  1. Decode and export the AWS_ACCESS_KEY_ID: 

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
    Copy to Clipboard Toggle word wrap

  2. Decode and export the AWS_SECRET_ACCESS_KEY_ID: 

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
    Copy to Clipboard Toggle word wrap

  3. Create a new directory and copy all blobs to it: 

    $ mkdir blobs

$ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs
    Copy to Clipboard Toggle word wrap
Note

You can also use rclone or sc3md instead of the AWS command line utility.

15.1.4. Scale the Red Hat Quay deployment back up
Copy link

  1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: registry
  namespace: ns
spec:
  components:
    …
    - kind: horizontalpodautoscaler
      managed: true
    1 
    
    - kind: quay
    2 
    
      managed: true
    - kind: clair
      managed: true
    - kind: mirror
      managed: true
    …
    Copy to Clipboard Toggle word wrap
    1
    Re-enables auto scaling of Quay, Clair and Mirroring workers again (if desired)
    2
    Replica overrides are removed again to scale the Quay components back up

  2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again: 

    $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
    Copy to Clipboard Toggle word wrap

  3. Check the status of the Red Hat Quay deployment: 

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

    Example output: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  ...
  name: registry
  namespace: <quay-namespace>
  ...
spec:
  ...
status:
  - lastTransitionTime: '2022-06-20T05:31:17Z'
    lastUpdateTime: '2022-06-20T17:31:13Z'
    message: All components reporting as healthy
    reason: HealthChecksPassing
    status: 'True'
    type: Available
    Copy to Clipboard Toggle word wrap

15.2. Restoring Red Hat Quay
Copy link

This procedure is used to restore Red Hat Quay when the Red Hat Quay Operator manages the database. It should be performed after a backup of your Red Hat Quay registry has been performed. See Backing up Red Hat Quay for more information.

Prerequisites

  • Red Hat Quay is deployed on OpenShift Container Platform using the Red Hat Quay Operator.
  • A backup of the Red Hat Quay configuration managed by the Red Hat Quay Operator has been created following the instructions in the Backing up Red Hat Quay section
  • Your Red Hat Quay database has been backed up.
  • The object storage bucket used by Red Hat Quay has been backed up.
  • The components quay, postgres and objectstorage are set to managed: true
  • If the component clair is set to managed: true, the component clairpostgres is also set to managed: true (starting with Red Hat Quay Operator v3.7 or later)
  • There is no running Red Hat Quay deployment managed by the Red Hat Quay Operator in the target namespace on your OpenShift Container Platform cluster
Note

If your deployment contains partially unmanaged database or storage components and you are using external services for Postgres or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to restore their data from a backup prior to restore Red Hat Quay

Note

These instructions assume you have followed the process in the Backing up Red Hat Quay guide and create the backup files with the same names.

  1. Restore the backed up Red Hat Quay configuration and the generated keys from the backup: 

    $ oc create -f ./config-bundle.yaml

$ oc create -f ./managed-secret-keys.yaml
    Copy to Clipboard Toggle word wrap
    Important

    If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

  2. Restore the QuayRegistry custom resource: 

    $ oc create -f ./quay-registry.yaml
    Copy to Clipboard Toggle word wrap

  3. Check the status of the Red Hat Quay deployment and wait for it to be available: 

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

15.2.2. Scale down your Red Hat Quay deployment
Copy link

  1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your QuayRegistry resource should look similar to the following: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: registry
  namespace: ns
spec:
  components:
    …
    - kind: horizontalpodautoscaler
      managed: false
    1 
    
    - kind: quay
      managed: true
      overrides:
    2 
    
        replicas: 0
    - kind: clair
      managed: true
      overrides:
        replicas: 0
    - kind: mirror
      managed: true
      overrides:
        replicas: 0
    …
    Copy to Clipboard Toggle word wrap
    1
    Disable auto scaling of Quay, Clair and Mirroring workers
    2
    Set the replica count to 0 for components accessing the database and objectstorage

  2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources: 

    $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>

$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

  3. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Operator) to disappear. You can check their status by running the following command: 

    $ oc get pods -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

    Example output: 

    registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h
    Copy to Clipboard Toggle word wrap

15.2.3. Restore your Red Hat Quay database
Copy link

  1. Identify your Quay database pod: 

    $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'
    Copy to Clipboard Toggle word wrap

    Example output: 

    quayregistry-quay-database-59f54bb7-58xs7
    Copy to Clipboard Toggle word wrap

  2. Upload the backup by copying it from the local environment and into the pod: 

    $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
    Copy to Clipboard Toggle word wrap

  3. Open a remote terminal to the database: 

    $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm
    Copy to Clipboard Toggle word wrap

  4. Enter psql: 

    bash-4.4$ psql
    Copy to Clipboard Toggle word wrap

  5. You can list the database by running the following command: 

    postgres=# \l
    Copy to Clipboard Toggle word wrap

    Example output: 

                                                      List of databases
           Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
----------------------------+----------------------------+----------+------------+------------+-----------------------
postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |
    Copy to Clipboard Toggle word wrap

  6. Drop the database: 

    postgres=# DROP DATABASE "quayregistry-quay-database";
    Copy to Clipboard Toggle word wrap

    Example output: 

    DROP DATABASE
    Copy to Clipboard Toggle word wrap

  7. Exit the postgres CLI to re-enter bash-4.4: 

    \q
    Copy to Clipboard Toggle word wrap

  8. Redirect your PostgreSQL database to your backup database: 

    sh-4.4$ psql < /tmp/backup.sql
    Copy to Clipboard Toggle word wrap

  9. Exit bash: 

    sh-4.4$ exit
    Copy to Clipboard Toggle word wrap

  1. Export the AWS_ACCESS_KEY_ID: 

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
    Copy to Clipboard Toggle word wrap

  2. Export the AWS_SECRET_ACCESS_KEY: 

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
    Copy to Clipboard Toggle word wrap

  3. Upload all blobs to the bucket by running the following command: 

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')
    Copy to Clipboard Toggle word wrap
Note

You can also use rclone or sc3md instead of the AWS command line utility.

15.2.5. Scale up your Red Hat Quay deployment
Copy link

  1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: registry
  namespace: ns
spec:
  components:
    …
    - kind: horizontalpodautoscaler
      managed: true
    1 
    
    - kind: quay
    2 
    
      managed: true
    - kind: clair
      managed: true
    - kind: mirror
      managed: true
    …
    Copy to Clipboard Toggle word wrap
    1
    Re-enables auto scaling of Red Hat Quay, Clair and mirroring workers again (if desired)
    2
    Replica overrides are removed again to scale the Red Hat Quay components back up

  2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again: 

    $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
    Copy to Clipboard Toggle word wrap

  3. Check the status of the Red Hat Quay deployment: 

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
    Copy to Clipboard Toggle word wrap

    Example output: 

    apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  ...
  name: registry
  namespace: <quay-namespace>
  ...
spec:
  ...
status:
  - lastTransitionTime: '2022-06-20T05:31:17Z'
    lastUpdateTime: '2022-06-20T17:31:13Z'
    message: All components reporting as healthy
    reason: HealthChecksPassing
    status: 'True'
    type: Available
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat