Manage Red Hat Quay

Procedure

1. Copy the certificate file and primary key file to your configuration directory, ensuring they are named ssl.cert and ssl.key respectively:

   cp ~/ssl.cert ~/ssl.key $QUAY/config

2. Change into the $QUAY/config directory by entering the following command:

   $ cd $QUAY/config

3. Edit the config.yaml file and specify that you want Red Hat Quay to handle TLS/SSL:

   config.yaml

   ...
   SERVER_HOSTNAME: quay-server.example.com
   ...
   PREFERRED_URL_SCHEME: https
   ...

4. Optional: Append the contents of the rootCA.pem file to the end of the ssl.cert file by entering the following command:

   $ cat rootCA.pem >> ssl.cert

5. Stop the Quay container by entering the following command:

   $ sudo podman stop quay

6. Restart the registry by entering the following command:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
     --name=quay \
     -v $QUAY/config:/conf/stack:Z \
     -v $QUAY/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.8.15

Procedure

1. Start the Quay container in configuration mode:

   $ sudo podman run --rm -it --name quay_config -p 80:8080 -p 443:8443 registry.redhat.io/quay/quay-rhel8:v3.8.15 config secret

2. In the Server Configuration section, select Red Hat Quay handles TLS for SSL/TLS. Upload the certificate file and private key file created earlier, ensuring that the Server Hostname matches the value used when the certificates were created.
3. Validate and download the updated configuration.

4. Stop the Quay container and then restart the registry by entering the following command:

   $ sudo podman rm -f quay
   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
   --name=quay \
   -v $QUAY/config:/conf/stack:Z \
   -v $QUAY/storage:/datastorage:Z \
   registry.redhat.io/quay/quay-rhel8:v3.8.15

Procedure

1. Enter the following command to copy the rootCA.pem file to the consolidated system-wide trust store:

   $ sudo cp rootCA.pem /etc/pki/ca-trust/source/anchors/

2. Enter the following command to update the system-wide trust store configuration:

   $ sudo update-ca-trust extract

3. Optional. You can use the trust list command to ensure that the Quay server has been configured:

   $ trust list | grep quay
       label: quay-server.example.com

   Now, when you browse to the registry at https://quay-server.example.com, the lock icon shows that the connection is secure:

   

4. To remove the rootCA.pem file from system-wide trust, delete the file and update the configuration:

   $ sudo rm /etc/pki/ca-trust/source/anchors/rootCA.pem

   $ sudo update-ca-trust extract

   $ trust list | grep quay

1. View certificate to be added to the container

   $ cat storage.crt
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
   [...]
   -----END CERTIFICATE-----

2. Create certs directory and copy certificate there

   $ mkdir -p quay/config/extra_ca_certs
   $ cp storage.crt quay/config/extra_ca_certs/
   $ tree quay/config/
   ├── config.yaml
   ├── extra_ca_certs
   │   ├── storage.crt

3. Obtain the Quay container’s CONTAINER ID with podman ps:

   $ sudo podman ps
   CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS
   5a3e82c4a75f        <registry>/<repo>/quay:v3.8.15 "/sbin/my_init"          24 hours ago        Up 18 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 443/tcp   grave_keller

4. Restart the container with that ID:

   $ sudo podman restart 5a3e82c4a75f

5. Examine the certificate copied into the container namespace:

   $ sudo podman exec -it 5a3e82c4a75f cat /etc/ssl/certs/storage.pem
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV

1. Begin by base64 encoding the contents of the certificate:

   $ cat ca.crt
   -----BEGIN CERTIFICATE-----
   MIIDljCCAn6gAwIBAgIBATANBgkqhkiG9w0BAQsFADA5MRcwFQYDVQQKDA5MQUIu
   TElCQ09SRS5TTzEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTE2
   MDExMjA2NTkxMFoXDTM2MDExMjA2NTkxMFowOTEXMBUGA1UECgwOTEFCLkxJQkNP
   UkUuU08xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZI
   [...]
   -----END CERTIFICATE-----
   
   $ cat ca.crt | base64 -w 0
   [...]
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

2. Use the kubectl tool to edit the quay-enterprise-config-secret.

   $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret

3. Add an entry for the cert and paste the full base64 encoded string under the entry:

     custom-cert.crt:
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

4. Finally, recycle all Red Hat Quay pods. Use kubectl delete to remove all Red Hat Quay pods. The Red Hat Quay Deployment will automatically schedule replacement pods with the new certificate data.

Procedure

1. In your Red Hat Quay installation directory, create a new directory for the Clair database data:

   $ mkdir /home/<user-name>/quay-poc/postgres-clairv4

2. Set the appropriate permissions for the postgres-clairv4 file by entering the following command:

   $ setfacl -m u:26:-wx /home/<user-name>/quay-poc/postgres-clairv4

3. Deploy a Clair Postgres database by entering the following command:

   $ sudo podman run -d --name postgresql-clairv4 \
     -e POSTGRESQL_USER=clairuser \
     -e POSTGRESQL_PASSWORD=clairpass \
     -e POSTGRESQL_DATABASE=clair \
     -e POSTGRESQL_ADMIN_PASSWORD=adminpass \
     -p 5433:5433 \
     -v /home/<user-name>/quay-poc/postgres-clairv4:/var/lib/pgsql/data:Z \
     {postgresimage}

4. Install the Postgres uuid-ossp module for your Clair deployment:

   $ podman exec -it postgresql-clairv4 /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"" | psql -d clair -U postgres'

   Example output

   CREATE EXTENSION

   Note

   Clair requires the uuid-ossp extension to be added to its Postgres database. For users with proper privileges, creating the extension will automatically be added by Clair. If users do not have the proper privileges, the extension must be added before start Clair.

   If the extension is not present, the following error will be displayed when Clair attempts to start: ERROR: Please load the "uuid-ossp" extension. (SQLSTATE 42501).

5. Stop the Quay container if it is running and restart it in configuration mode, loading the existing configuration as a volume:

   $ sudo podman run --rm -it --name quay_config \
     -p 80:8080 -p 443:8443 \
     -v $QUAY/config:/conf/stack:Z \
     {productrepo}/{quayimage}:{productminv} config secret

6. Log in to the configuration tool and click Enable Security Scanning in the Security Scanner section of the UI.
7. Set the HTTP endpoint for Clair using a port that is not already in use on the quay-server system, for example, 8081.

8. Create a pre-shared key (PSK) using the Generate PSK button.

   Security Scanner UI

   

9. Validate and download the config.yaml file for Red Hat Quay, and then stop the Quay container that is running the configuration editor.

10. Extract the new configuration bundle into your Red Hat Quay installation directory, for example:

    $ tar xvf quay-config.tar.gz -d /home/<user-name>/quay-poc/

11. Create a folder for your Clair configuration file, for example:

    $ mkdir /etc/opt/clairv4/config/

12. Change into the Clair configuration folder:

    $ cd /etc/opt/clairv4/config/

13. Create a Clair configuration file, for example:

    http_listen_addr: :8081
    introspection_addr: :8088
    log_level: debug
    indexer:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      scanlock_retry: 10
      layer_scan_concurrency: 5
      migrations: true
    matcher:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      max_conn_pool: 100
      run: ""
      migrations: true
      indexer_addr: clair-indexer
    notifier:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      delivery_interval: 1m
      poll_interval: 5m
      migrations: true
    auth:
      psk:
        key: "MTU5YzA4Y2ZkNzJoMQ=="
        iss: ["quay"]
    # tracing and metrics
    trace:
      name: "jaeger"
      probability: 1
      jaeger:
        agent_endpoint: "localhost:6831"
        service_name: "clair"
    metrics:
      name: "prometheus"

    For more information about Clair’s configuration format, see Clair configuration reference.

14. Start Clair by using the container image, mounting in the configuration from the file you created:

    $ sudo podman run -d --name clairv4 \
    -p 8081:8081 -p 8088:8088 \
    -e CLAIR_CONF=/clair/config.yaml \
    -e CLAIR_MODE=combo \
    -v /etc/opt/clairv4/config:/clair:Z \
    registry.redhat.io/quay/clair-rhel8:v3.8.15

    Note

    Running multiple Clair containers is also possible, but for deployment scenarios beyond a single container the use of a container orchestrator like Kubernetes or OpenShift Container Platform is strongly recommended.

Procedure

1. Pull a sample image by entering the following command:

   $ podman pull ubuntu:20.04

2. Tag the image to your registry by entering the following command:

   $ sudo podman tag docker.io/library/ubuntu:20.04 <quay-server.example.com>/<user-name>/ubuntu:20.04

3. Push the image to your Red Hat Quay registry by entering the following command:

   $ sudo podman push --tls-verify=false quay-server.example.com/quayadmin/ubuntu:20.04

4. Log in to your Red Hat Quay deployment through the UI.
5. Click the repository name, for example, quayadmin/ubuntu.

6. In the navigation pane, click Tags.

   Report summary

   

7. Click the image report, for example, 45 medium, to show a more detailed report:

   Report details

   

Procedure

1. Log in to Red Hat Quay through the web UI.
2. Select the organization for which the external application will be configured.
3. On the navigation pane, select Applications.
4. Select Create New Application and enter a name for the new application, for example, openshift.
5. On the OAuth Applications page, select your application, for example, openshift.
6. On the navigation pane, select Generate Token.

7. Select the following fields:

   • Administer Organization
   • Administer Repositories
   • Create Repositories
   • View all visible repositories
   • Read/Write to any accessible repositories
   • Administer User
   • Read User Information
8. Review the assigned permissions.
9. Select Authorize Application and then confirm confirm the authorization by selecting Authorize Application.

10. Save the generated access token.

    Important

    Red Hat Quay does not offer token management. You cannot list tokens, delete tokens, or modify tokens. The generated access token is only shown once and cannot be re-obtained after closing the page.

Procedure

1. Open the Administrator perspective of the web console and navigate to Operators → OperatorHub on the navigation pane.
2. Search for Quay Bridge Operator, click the Quay Bridge Operator title, and then click Install.
3. Select the version to install, for example, stable-3.7, and then click Install.
4. Click View Operator when the installation finishes to go to the Quay Bridge Operator’s Details page. Alternatively, you can click Installed Operators → Red Hat Quay Bridge Operator to go to the Details page.

1. Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.

   

2. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file.

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to IPv6, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: IPv6
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to IPv6 by entering the following command:

   $ curl <quay_endpoint>/health/instance
   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to dual-stack, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: dual-stack
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to both channels by entering the following command:

   1. For IPv4, enter the following command:

      $ curl --ipv4 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   2. For IPv6, enter the following command:

      $ curl --ipv6 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

Procedure

1. In your deployment’s config.yaml file, add the LDAP_SUPERUSER_FILTER parameter and add the group of users you want configured as super users, for example, root:

   ---
   AUTHENTICATION_TYPE: LDAP
   ---
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,o=<example_organization_unit>,dc=<example_domain_component>,dc=com)
   LDAP_SUPERUSER_FILTER: (<filterField>=<value>)
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com

2. Start, or restart, your Red Hat Quay deployment.

Procedure

1. Create a new organization or choose an existing one. Initially, no quota is configured, as can be seen on the Organization Settings tab:

   

2. Log in to the registry as a superuser and navigate to the Manage Organizations tab on the Super User Admin Panel. Click the Options icon of the organization for which you want to create storage quota limits:

   

3. Click Configure Quota and enter the initial quota, for example, 10 MB. Then click Apply and Close:

   

4. Check that the quota consumed shows 0 of 10 MB on the Manage Organizations tab of the superuser panel:

   

   The consumed quota information is also available directly on the Organization page:

   Initial consumed quota

   

5. To increase the quota to 100MB, navigate to the Manage Organizations tab on the superuser panel. Click the Options icon and select Configure Quota, setting the quota to 100 MB. Click Apply and then Close:

   

6. Push a sample image to the organization from the command line:

   Sample commands

   $ podman pull ubuntu:18.04
   
   $ podman tag docker.io/library/ubuntu:18.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04
   
   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

7. On the superuser panel, the quota consumed per organization is displayed:

   

8. The Organization page shows the total proportion of the quota used by the image:

   Total Quota Consumed for first image

   

9. Pull, tag, and push a second image, for example, nginx:

   Sample commands

   $ podman pull nginx
   
   $ podman tag docker.io/library/nginx example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx
   
   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

10. The Organization page shows the total proportion of the quota used by each repository in that organization:

    Total Quota Consumed for each repository

    

11. Create reject and warning limits:

    From the superuser panel, navigate to the Manage Organizations tab. Click the Options icon for the organization and select Configure Quota. In the Quota Policy section, with the Action type set to Reject, set the Quota Threshold to 80 and click Add Limit:

    

12. To create a warning limit, select Warning as the Action type, set the Quota Threshold to 70 and click Add Limit:

    

13. Click Close on the quota popup. The limits are viewable, but not editable, on the Settings tab of the Organization page:

    

14. Push an image where the reject limit is exceeded:

    Because the reject limit (80%) has been set to below the current repository size (~83%), the next push is rejected automatically.

    Sample image push

    $ podman pull ubuntu:20.04
    
    $ podman tag docker.io/library/ubuntu:20.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04
    
    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04

    Sample output when quota exceeded

    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0002] failed, retrying in 1s ... (1/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0005] failed, retrying in 1s ... (2/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0009] failed, retrying in 1s ... (3/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace

15. When limits are exceeded, notifications are displayed in the UI:

    Quota notifications

    

Procedure

1. Back up the Quay config.yaml of your standalone deployment:

   $ mkdir /tmp/quay-backup
   $ cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup

2. Create a backup of the database that your standalone Quay deployment is using:

   $ pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql

3. Install the AWS CLI if you do not have it already.

4. Create an ~/.aws/ directory:

   $ mkdir ~/.aws/

5. Obtain the access_key and secret_key from the Quay config.yaml of your standalone deployment:

   $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml

   Example output:

   DISTRIBUTED_STORAGE_CONFIG:
       minio-1:
           - RadosGWStorage
           - access_key: ##########
             bucket_name: quay
             hostname: 172.24.10.50
             is_secure: false
             port: "9000"
             secret_key: ##########
             storage_path: /datastorage/registry

6. Store the access_key and secret_key from the Quay config.yaml file in your ~/.aws directory:

   $ touch ~/.aws/credentials

7. Optional: Check that your access_key and secret_key are stored:

   $ cat > ~/.aws/credentials << EOF
   [default]
   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
   EOF

   Example output:

   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG

   Note

   If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can, you can configure these by running aws configure and manually inputting the credentials.

8. In your quay-backup directory, create a bucket_backup directory:

   $ mkdir /tmp/quay-backup/bucket-backup

9. Backup all blobs from the S3 storage:

   $ aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/

   Note

   The PUBLIC_S3_ENDPOINT can be read from the Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG. If the endpoint is insecure, use http instead of https in the endpoint URL.

1. Scale down the Red Hat Quay Operator:

   $ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators

2. Scale down the application and mirror deployments:

   $ oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT

3. Copy the database SQL backup to the Quay PostgreSQL database instance:

   $ oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata

4. Obtain the database password from the Operator-created config.yaml file:

   $ oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret

   Example output:

         "name": "QUAY_CONFIG_SECRET_NAME"

   $ oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

   cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI

   Example output:

   postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME

5. Execute a shell inside of the database pod:

   # oc exec -it quay-postgresql-database-pod -- /bin/bash

6. Enter psql:

   bash-4.4$ psql

7. Drop the database:

   postgres=# DROP DATABASE "example-restore-registry-quay-database";

   Example output:

   DROP DATABASE

8. Create a new database and set the owner as the same name:

   postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database";

   Example output:

   CREATE DATABASE

9. Connect to the database:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

10. Create a pg_trmg extension of your Quay database:

    example-restore-registry-quay-database=# create extension pg_trgm ;

    Example output:

    CREATE EXTENSION

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Set the password for your PostgreSQL deployment:

    bash-4.4$ psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

13. Exit bash mode:

    bash-4.4$ exit

14. Create a new configuration bundle for the Red Hat Quay Operator.

    $ touch config-bundle.yaml

15. In your new config-bundle.yaml, include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml:

    $ cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml

    Note

    You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file.

16. Create a configuration bundle secret inside of your OpenShift cluster:

    $ oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml

17. Scale up the Quay pods:

    $ oc scale --replicas=1 deployment quayregistry-quay-app
    deployment.apps/quayregistry-quay-app scaled

18. Scale up the mirror pods:

    $ oc scale --replicas=1  deployment quayregistry-quay-mirror
    deployment.apps/quayregistry-quay-mirror scaled

19. Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle:

    $ oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}'

    Note

    If Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default.

20. Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file:

    $ touch credentials.yaml

    $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

    $ cat > ~/.aws/credentials << EOF
    [default]
    aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
    aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
    EOF

    Note

    If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can configure these by running aws configure and manually inputting the credentials.

21. Record the NooBaa’s publicly available endpoint:

    $ oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}"

22. Sync the backup data to the NooBaa backend storage:

    $ aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME

23. Scale the Operator back up to 1 pod:

    $ oc scale –replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators

Procedure

1. Create a temporary backup directory, for example, quay-backup:

   $ mkdir /tmp/quay-backup

2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install:

   $ podman run --name quay-app \
      -v /opt/quay-install/config:/conf/stack:Z \
      -v /opt/quay-install/storage:/datastorage:Z \
      registry.redhat.io/quay/quay-rhel8:v3.8.15

   Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command:

   $ cd /opt/quay-install

3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command:

   $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

4. Back up the Quay container service by entering the following command:

   $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -
   
     /usr/bin/podman run --name quay-app \
     -v /opt/quay-install/config:/conf/stack:Z \
     -v /opt/quay-install/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.8.15

5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command:

   $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command:

   $ grep DB_URI /tmp/quay-backup/quay-config.yaml

   Example output:

   $ postgresql://<username>:test123@172.24.10.50/quay

7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command:

   $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command:

   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - S3Storage
       - s3_bucket: <bucket_name>
         storage_path: /registry
         s3_access_key: <s3_access_key>
         s3_secret_key: <s3_secret_key>
         host: <host_name>

9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7:

   $ export AWS_ACCESS_KEY_ID=<access_key>

10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG:

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
    download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

Procedure

1. Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container:

   $ mkdir /opt/new-quay-install

2. Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1:

   $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

3. Change into the new-quay-install directory by entering the following command:

   $ cd /opt/new-quay-install/

4. Extract the contents of your Red Hat Quay directory:

   $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

5. Recall the DB_URI from your backed-up config.yaml file by entering the following command:

   $ grep DB_URI config.yaml

   Example output:

   postgresql://<username>:test123@172.24.10.50/quay

6. Run the following command to enter the PostgreSQL database server:

   $ sudo postgres

7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command:

   $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
   postgres=> CREATE DATABASE example_restore_registry_quay_database;

   Example output:

   CREATE DATABASE

8. Connect to the database by running the following command:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

9. Create a pg_trmg extension of your Quay database by running the following command:

   example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Example output:

   CREATE EXTENSION

10. Exit the postgres CLI by entering the following command:

    \q

11. Import the database backup to your new database by running the following command:

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle:

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13:

    $ export AWS_ACCESS_KEY_ID=<access_key>

14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

15. Create a new s3 bucket by entering the following command:

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

16. Upload all blobs to the new s3 bucket by entering the following command:

    $ aws s3 sync --no-verify-ssl \
    --endpoint-url <example_endpoint_url> 1
    /tmp/quay-backup/blob-backup/. s3://quay/

    1

    The Red Hat Quay registry endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
    ...

17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <new_bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

Procedure

1. Enter the following command to ensure that garbage collection is properly working:

   $ sudo podman logs <container_id>

   Example output:

   gcworker stdout | 2022-11-14 18:46:52,458 [63] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2022-11-14 18:47:22 UTC)" executed successfully

2. Delete an image tag.

3. Enter the following command to ensure that the tag was deleted:

   $ podman logs quay-app

   Example output:

   gunicorn-web stdout | 2022-11-14 19:23:44,574 [233] [INFO] [gunicorn.access] 192.168.0.38 - - [14/Nov/2022:19:23:44 +0000] "DELETE /api/v1/repository/quayadmin/busybox/tag/test HTTP/1.0" 204 0 "http://quay-server.example.com/repository/quayadmin/busybox?tab=tags" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"

Procedure

1. On your web browser, navigate to https://{quay-ip-endpoint}/health/instance.

2. You are taken to the health instance page, which returns information like the following:

   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   For Red Hat Quay, "status_code": 200 means that the instance is health. Conversely, if you receive "status_code": 503, there is an issue with your deployment.