Manage Red Hat Quay

1. Copy the certificate file and primary key file to your configuration directory, ensuring they are named ssl.cert and ssl.key respectively:

   $ cp ~/ssl.cert $QUAY/config
   $ cp ~/ssl.key $QUAY/config
   $ cd $QUAY/config

2. Edit the config.yaml file and specify that you want Quay to handle TLS:

   config.yaml

   ...
   SERVER_HOSTNAME: quay-server.example.com
   ...
   PREFERRED_URL_SCHEME: https
   ...

3. Stop the Quay container and restart the registry:

   $ sudo podman rm -f quay
   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
     --name=quay \
     -v $QUAY/config:/conf/stack:Z \
     -v $QUAY/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.7.13

1. Start the Quay container in configuration mode:

   $ sudo podman run --rm -it --name quay_config -p 80:8080 -p 443:8443 registry.redhat.io/quay/quay-rhel8:v3.7.13 config secret

2. In the Server Configuration section, select Red Hat Quay handles TLS for TLS. Upload the certificate file and private key file created earlier, ensuring that the Server Hostname matches the value used when creating the certs. Validate and download the updated configuration.

3. Stop the Quay container and then restart the registry:

   $ sudo podman rm -f quay
   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
   --name=quay \
   -v $QUAY/config:/conf/stack:Z \
   -v $QUAY/storage:/datastorage:Z \
   registry.redhat.io/quay/quay-rhel8:v3.7.13

1. Copy the root CA file to the consolidated system-wide trust store:

   $ sudo cp rootCA.pem /etc/pki/ca-trust/source/anchors/

2. Update the system-wide trust store configuration:

   $ sudo update-ca-trust extract

3. You can use the trust list command to ensure that the Quay server has been configured:

   $ trust list | grep quay
       label: quay-server.example.com

   Now, when you browse to the registry at https://quay-server.example.com, the lock icon shows that the connection is secure:

   

4. To remove the root CA from system-wide trust, delete the file and update the configuration:

   $ sudo rm /etc/pki/ca-trust/source/anchors/rootCA.pem
   $ sudo update-ca-trust extract
   $ trust list | grep quay
   $

1. View certificate to be added to the container

   $ cat storage.crt
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
   [...]
   -----END CERTIFICATE-----

2. Create certs directory and copy certificate there

   $ mkdir -p quay/config/extra_ca_certs
   $ cp storage.crt quay/config/extra_ca_certs/
   $ tree quay/config/
   ├── config.yaml
   ├── extra_ca_certs
   │   ├── storage.crt

3. Obtain the Quay container’s CONTAINER ID with podman ps:

   $ sudo podman ps
   CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS
   5a3e82c4a75f        <registry>/<repo>/quay:v3.7.13 "/sbin/my_init"          24 hours ago        Up 18 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 443/tcp   grave_keller

4. Restart the container with that ID:

   $ sudo podman restart 5a3e82c4a75f

5. Examine the certificate copied into the container namespace:

   $ sudo podman exec -it 5a3e82c4a75f cat /etc/ssl/certs/storage.pem
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV

1. Begin by base64 encoding the contents of the certificate:

   $ cat ca.crt
   -----BEGIN CERTIFICATE-----
   MIIDljCCAn6gAwIBAgIBATANBgkqhkiG9w0BAQsFADA5MRcwFQYDVQQKDA5MQUIu
   TElCQ09SRS5TTzEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE2
   MDExMjA2NTkxMFoXDTM2MDExMjA2NTkxMFowOTEXMBUGA1UECgwOTEFCLkxJQkNP
   UkUuU08xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZI
   [...]
   -----END CERTIFICATE-----
   
   $ cat ca.crt | base64 -w 0
   [...]
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

2. Use the kubectl tool to edit the quay-enterprise-config-secret.

   $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret

3. Add an entry for the cert and paste the full base64 encoded string under the entry:

     custom-cert.crt:
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

4. Finally, recycle all Red Hat Quay pods. Use kubectl delete to remove all Red Hat Quay pods. The Red Hat Quay Deployment will automatically schedule replacement pods with the new certificate data.

1. Deploy a (preferably fault-tolerant) Postgres database server. Note that Clair requires the uuid-ossp extension to be added to its Postgres database. If the user supplied in Clair’s config.yaml has the necessary privileges to create the extension then it will be added automatically by Clair itself. If not, then the extension must be added before starting Clair. If the extension is not present, the following error will be displayed when Clair attempts to start.

   ERROR: Please load the "uuid-ossp" extension. (SQLSTATE 42501)

2. Create a Clair config file in a specific folder, for example, /etc/clairv4/config/config.yaml).

   config.yaml

   introspection_addr: :8089
   http_listen_addr: :8080
   log_level: debug
   indexer:
     connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable
     scanlock_retry: 10
     layer_scan_concurrency: 5
     migrations: true
   matcher:
     connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable
     max_conn_pool: 100
     run: ""
     migrations: true
     indexer_addr: clair-indexer
   notifier:
     connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable
     delivery_interval: 1m
     poll_interval: 5m
     migrations: true
   
   # tracing and metrics
   trace:
     name: "jaeger"
     probability: 1
     jaeger:
       agent_endpoint: "localhost:6831"
       service_name: "clair"
   metrics:
     name: "prometheus"

1. Run Clair via the container image, mounting in the configuration from the file you created.

   $ podman run -p 8080:8080 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /etc/clair4/config:/clair -d registry.redhat.io/quay/clair-rhel8:v3.7.13

2. Follow the remaining instructions from the previous section for configuring Red Hat Quay to use the new Clair V4 endpoint.

1. Log in to your Red Hat Quay cluster and select an organization for which you have configured Clair scanning.

2. Select a repository from that organization that holds some images and select Tags from the left navigation. The following figure shows an example of a repository with two images that have been scanned:

   

3. If vulnerabilities are found, select to under the Security Scan column for the image to see either all vulnerabilities or those that are fixable. The following figure shows information on all vulnerabilities found:

   

1. First ensure that your Clair configuration has disabled automated Updaters from running.

   config.yaml

   matcher:
     disable_updaters: true

2. Export out the latest Updater data to a local archive. This requires the clairctl tool which can be run directly as a binary, or via the Clair container image. Assuming your Clair configuration is in /etc/clairv4/config/config.yaml, to run via the container image:

   $ podman run -it --rm -v /etc/clairv4/config:/cfg:Z -v /path/to/output/directory:/updaters:Z --entrypoint /bin/clairctl registry.redhat.io/quay/clair-rhel8:v3.7.13 --config /cfg/config.yaml export-updaters  /updaters/updaters.gz

   Note that you need to explicitly reference the Clair configuration. This will create the Updater archive in /etc/clairv4/updaters/updaters.gz. If you want to ensure the archive was created without any errors from the source databases, you can supply the --strict flag to clairctl. The archive file should be copied over to a volume that is accessible from the disconnected host running Clair. From the disconnected host, use the same procedure now to import the archive into Clair.

   $ podman run -it --rm -v /etc/clairv4/config:/cfg:Z -v /path/to/output/directory:/updaters:Z --entrypoint /bin/clairctl registry.redhat.io/quay/clair-rhel8:v3.7.13 --config /cfg/config.yaml import-updaters /updaters/updaters.gz

1. Go to Operators → OperatorHub (select Security) to see the available Container Security Operator.
2. Select the Container Security Operator, then select Install to go to the Create Operator Subscription page.
3. Check the settings (all namespaces and automatic approval strategy, by default), and select Subscribe. The Container Security appears after a few moments on the Installed Operators screen.

4. Optionally, you can add custom certificates to the CSO. In this example, create a certificate named quay.crt in the current directory. Then run the following command to add the cert to the CSO (restart the Operator pod for the new certs to take effect):

   $ oc create secret generic container-security-operator-extra-certs --from-file=quay.crt -n openshift-operators

5. Open the OpenShift Dashboard (Home → Dashboards). A link to Image Security appears under the status section, with a listing of the number of vulnerabilities found so far. Select the link to see a Security breakdown, as shown in the following figure:

   

6. You can do one of two things at this point to follow up on any detected vulnerabilities:

   • Select the link to the vulnerability. You are taken to the container registry, Red Hat Quay or other registry where the container came from, where you can see information about the vulnerability. The following figure shows an example of detected vulnerabilities from a Quay.io registry:

     

   • Select the namespaces link to go to the ImageManifestVuln screen, where you can see the name of the selected image and all namespaces where that image is running. The following figure indicates that a particular vulnerable image is running in two namespaces:

     

Procedure

1. Log in to Red Hat Quay through the web UI.
2. Select the organization for which the external application will be configured.
3. On the navigation pane, select Applications.
4. Select Create New Application and enter a name for the new application, for example, openshift.
5. On the OAuth Applications page, select your application, for example, openshift.
6. On the navigation pane, select Generate Token.

7. Select the following fields:

   • Administer Organization
   • Administer Repositories
   • Create Repositories
   • View all visible repositories
   • Read/Write to any accessible repositories
   • Administer User
   • Read User Information
8. Review the assigned permissions.
9. Select Authorize Application and then confirm confirm the authorization by selecting Authorize Application.

10. Save the generated access token.

    Important

    Red Hat Quay does not offer token management. You cannot list tokens, delete tokens, or modify tokens. The generated access token is only shown once and cannot be re-obtained after closing the page.

Procedure

1. Open the Administrator perspective of the web console and navigate to Operators → OperatorHub on the navigation pane.
2. Search for Quay Bridge Operator, click the Quay Bridge Operator title, and then click Install.
3. Select the version to install, for example, stable-3.7, and then click Install.
4. Click View Operator when the installation finishes to go to the Quay Bridge Operator’s Details page. Alternatively, you can click Installed Operators → Red Hat Quay Bridge Operator to go to the Details page.

1. Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.

   

2. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file.

Procedure

1. Create a new organization or choose an existing one. Initially, no quota is configured, as can be seen on the Organization Settings tab:

   

2. Log in to the registry as a superuser and navigate to the Manage Organizations tab on the Super User Admin Panel. Click the Options icon of the organization for which you want to create storage quota limits:

   

3. Click Configure Quota and enter the initial quota, for example, 10 MB. Then click Apply and Close:

   

4. Check that the quota consumed shows 0 of 10 MB on the Manage Organizations tab of the superuser panel:

   

   The consumed quota information is also available directly on the Organization page:

   Initial consumed quota

   

5. To increase the quota to 100MB, navigate to the Manage Organizations tab on the superuser panel. Click the Options icon and select Configure Quota, setting the quota to 100 MB. Click Apply and then Close:

   

6. Push a sample image to the organization from the command line:

   Sample commands

   $ podman pull ubuntu:18.04
   
   $ podman tag docker.io/library/ubuntu:18.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04
   
   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

7. On the superuser panel, the quota consumed per organization is displayed:

   

8. The Organization page shows the total proportion of the quota used by the image:

   Total Quota Consumed for first image

   

9. Pull, tag, and push a second image, for example, nginx:

   Sample commands

   $ podman pull nginx
   
   $ podman tag docker.io/library/nginx example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx
   
   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

10. The Organization page shows the total proportion of the quota used by each repository in that organization:

    Total Quota Consumed for each repository

    

11. Create reject and warning limits:

    From the superuser panel, navigate to the Manage Organizations tab. Click the Options icon for the organization and select Configure Quota. In the Quota Policy section, with the Action type set to Reject, set the Quota Threshold to 80 and click Add Limit:

    

12. To create a warning limit, select Warning as the Action type, set the Quota Threshold to 70 and click Add Limit:

    

13. Click Close on the quota popup. The limits are viewable, but not editable, on the Settings tab of the Organization page:

    

14. Push an image where the reject limit is exceeded:

    Because the reject limit (80%) has been set to below the current repository size (~83%), the next push is rejected automatically.

    Sample image push

    $ podman pull ubuntu:20.04
    
    $ podman tag docker.io/library/ubuntu:20.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04
    
    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04

    Sample output when quota exceeded

    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0002] failed, retrying in 1s ... (1/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0005] failed, retrying in 1s ... (2/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0009] failed, retrying in 1s ... (3/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace

15. When limits are exceeded, notifications are displayed in the UI:

    Quota notifications

    

Procedure

1. Back up the Quay config.yaml of your standalone deployment:

   $ mkdir /tmp/quay-backup
   $ cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup

2. Create a backup of the database that your standalone Quay deployment is using:

   $ pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql

3. Install the AWS CLI if you do not have it already.

4. Create an ~/.aws/ directory:

   $ mkdir ~/.aws/

5. Obtain the access_key and secret_key from the Quay config.yaml of your standalone deployment:

   $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml

   Example output:

   DISTRIBUTED_STORAGE_CONFIG:
       minio-1:
           - RadosGWStorage
           - access_key: ##########
             bucket_name: quay
             hostname: 172.24.10.50
             is_secure: false
             port: "9000"
             secret_key: ##########
             storage_path: /datastorage/registry

6. Store the access_key and secret_key from the Quay config.yaml file in your ~/.aws directory:

   $ touch ~/.aws/credentials

7. Optional: Check that your access_key and secret_key are stored:

   $ cat > ~/.aws/credentials << EOF
   [default]
   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
   EOF

   Example output:

   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG

   Note

   If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can, you can configure these by running aws configure and manually inputting the credentials.

8. In your quay-backup directory, create a bucket_backup directory:

   $ mkdir /tmp/quay-backup/bucket-backup

9. Backup all blobs from the S3 storage:

   $ aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/

   Note

   The PUBLIC_S3_ENDPOINT can be read from the Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG. If the endpoint is insecure, use http instead of https in the endpoint URL.

1. Scale down the Red Hat Quay Operator:

   $ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators

2. Scale down the application and mirror deployments:

   $ oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT

3. Copy the database SQL backup to the Quay PostgreSQL database instance:

   $ oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata

4. Obtain the database password from the Operator-created config.yaml file:

   $ oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret

   Example output:

         "name": "QUAY_CONFIG_SECRET_NAME"

   $ oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

   cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI

   Example output:

   postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME

5. Execute a shell inside of the database pod:

   # oc exec -it quay-postgresql-database-pod -- /bin/bash

6. Enter psql:

   bash-4.4$ psql

7. Drop the database:

   postgres=# DROP DATABASE "example-restore-registry-quay-database";

   Example output:

   DROP DATABASE

8. Create a new database and set the owner as the same name:

   postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database";

   Example output:

   CREATE DATABASE

9. Connect to the database:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

10. Create a pg_trmg extension of your Quay database:

    example-restore-registry-quay-database=# create extension pg_trgm ;

    Example output:

    CREATE EXTENSION

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Set the password for your PostgreSQL deployment:

    bash-4.4$ psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

13. Exit bash mode:

    bash-4.4$ exit

14. Create a new configuration bundle for the Red Hat Quay Operator.

    $ touch config-bundle.yaml

15. In your new config-bundle.yaml, include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml:

    $ cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml

    Note

    You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file.

16. Create a configuration bundle secret inside of your OpenShift cluster:

    $ oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml

17. Scale up the Quay pods:

    $ oc scale --replicas=1 deployment quayregistry-quay-app
    deployment.apps/quayregistry-quay-app scaled

18. Scale up the mirror pods:

    $ oc scale --replicas=1  deployment quayregistry-quay-mirror
    deployment.apps/quayregistry-quay-mirror scaled

19. Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle:

    $ oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}'

    Note

    If Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default.

20. Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file:

    $ touch credentials.yaml

    $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

    $ cat > ~/.aws/credentials << EOF
    [default]
    aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
    aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
    EOF

    Note

    If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can configure these by running aws configure and manually inputting the credentials.

21. Record the NooBaa’s publicly available endpoint:

    $ oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}"

22. Sync the backup data to the NooBaa backend storage:

    $ aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME

23. Scale the Operator back up to 1 pod:

    $ oc scale –replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators

Procedure

1. Create a temporary backup directory, for example, quay-backup:

   $ mkdir /tmp/quay-backup

2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install:

   $ podman run --name quay-app \
      -v /opt/quay-install/config:/conf/stack:Z \
      -v /opt/quay-install/storage:/datastorage:Z \
      {productrepo}/{quayimage}:{productminv}

   Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command:

   $ cd /opt/quay-install

3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command:

   $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

4. Back up the Quay container service by entering the following command:

   $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -
   
     /usr/bin/podman run --name quay-app \
     -v /opt/quay-install/config:/conf/stack:Z \
     -v /opt/quay-install/storage:/datastorage:Z \
     {productrepo}/{quayimage}:{productminv}

5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command:

   $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command:

   $ grep DB_URI /tmp/quay-backup/quay-config.yaml

   Example output:

   $ postgresql://<username>:test123@172.24.10.50/quay

7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command:

   $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command:

   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - S3Storage
       - s3_bucket: <bucket_name>
         storage_path: /registry
         s3_access_key: <s3_access_key>
         s3_secret_key: <s3_secret_key>
         host: <host_name>

9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7:

   $ export AWS_ACCESS_KEY_ID=<access_key>

10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG:

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
    download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

Procedure

1. Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container:

   $ mkdir /opt/new-quay-install

2. Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1:

   $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

3. Change into the new-quay-install directory by entering the following command:

   $ cd /opt/new-quay-install/

4. Extract the contents of your Red Hat Quay directory:

   $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

5. Recall the DB_URI from your backed-up config.yaml file by entering the following command:

   $ grep DB_URI config.yaml

   Example output:

   postgresql://<username>:test123@172.24.10.50/quay

6. Run the following command to enter the PostgreSQL database server:

   $ sudo postgres

7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command:

   $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
   postgres=> CREATE DATABASE example_restore_registry_quay_database;

   Example output:

   CREATE DATABASE

8. Connect to the database by running the following command:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

9. Create a pg_trmg extension of your Quay database by running the following command:

   example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Example output:

   CREATE EXTENSION

10. Exit the postgres CLI by entering the following command:

    \q

11. Import the database backup to your new database by running the following command:

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle:

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13:

    $ export AWS_ACCESS_KEY_ID=<access_key>

14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

15. Create a new s3 bucket by entering the following command:

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

16. Upload all blobs to the new s3 bucket by entering the following command:

    $ aws s3 sync --no-verify-ssl \
    --endpoint-url <example_endpoint_url> 1
    /tmp/quay-backup/blob-backup/. s3://quay/

    1

    The Red Hat Quay registry endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
    ...

17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <new_bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

Procedure

1. Enter the following command to ensure that garbage collection is properly working:

   $ sudo podman logs <container_id>

   Example output:

1. Delete an image tag.

2. Enter the following command to ensure that the tag was deleted:

   $ podman logs quay-app

   Example output: