Manage Red Hat Quay

Procedure

1. Generate the root CA key by entering the following command:

   $ openssl genrsa -out rootCA.key 2048

2. Generate the root CA certificate by entering the following command:

   $ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

3. Enter the information that will be incorporated into your certificate request, including the server hostname, for example:

   Country Name (2 letter code) [XX]:IE
   State or Province Name (full name) []:GALWAY
   Locality Name (eg, city) [Default City]:GALWAY
   Organization Name (eg, company) [Default Company Ltd]:QUAY
   Organizational Unit Name (eg, section) []:DOCS
   Common Name (eg, your name or your server's hostname) []:quay-server.example.com

4. Generate the server key by entering the following command:

   $ openssl genrsa -out ssl.key 2048

5. Generate a signing request by entering the following command:

   $ openssl req -new -key ssl.key -out ssl.csr

6. Enter the information that will be incorporated into your certificate request, including the server hostname, for example:

   Country Name (2 letter code) [XX]:IE
   State or Province Name (full name) []:GALWAY
   Locality Name (eg, city) [Default City]:GALWAY
   Organization Name (eg, company) [Default Company Ltd]:QUAY
   Organizational Unit Name (eg, section) []:DOCS
   Common Name (eg, your name or your server's hostname) []:quay-server.example.com
   Email Address []:

7. Create a configuration file openssl.cnf, specifying the server hostname, for example:

   Example openssl.cnf file

   [req]
   req_extensions = v3_req
   distinguished_name = req_distinguished_name
   [req_distinguished_name]
   [ v3_req ]
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   subjectAltName = @alt_names
   [alt_names]
   DNS.1 = <quay-server.example.com>
   IP.1 = 192.168.1.112

8. Use the configuration file to generate the certificate ssl.cert:

   $ openssl x509 -req -in ssl.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out ssl.cert -days 356 -extensions v3_req -extfile openssl.cnf

9. Confirm your created certificates and files by entering the following command:

   $ ls /path/to/certificates

   Example output

   rootCA.key ssl-bundle.cert ssl.key custom-ssl-config-bundle-secret.yaml rootCA.pem ssl.cert
   openssl.cnf rootCA.srl  ssl.csr

Procedure

1. Copy the certificate file and primary key file to your configuration directory, ensuring they are named ssl.cert and ssl.key respectively:

   cp ~/ssl.cert ~/ssl.key /path/to/configuration_directory

2. Navigate to the configuration directory by entering the following command:

   $ cd /path/to/configuration_directory

3. Edit the config.yaml file and specify that you want Red Hat Quay to handle SSL/TLS:

   Example config.yaml file

   # ...
   SERVER_HOSTNAME: <quay-server.example.com>
   ...
   PREFERRED_URL_SCHEME: https
   # ...

4. Optional: Append the contents of the rootCA.pem file to the end of the ssl.cert file by entering the following command:

   $ cat rootCA.pem >> ssl.cert

5. Stop the Quay container by entering the following command:

   $ sudo podman stop <quay_container_name>

6. Restart the registry by entering the following command:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
     --name=quay \
     -v $QUAY/config:/conf/stack:Z \
     -v $QUAY/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.13.1

Procedure

1. Start the Quay container in configuration mode:

   $ sudo podman run --rm -it --name quay_config -p 80:8080 -p 443:8443 registry.redhat.io/quay/quay-rhel8:v3.13.1 config secret

2. In the Server Configuration section, select Red Hat Quay handles TLS for SSL/TLS. Upload the certificate file and private key file created earlier, ensuring that the Server Hostname matches the value used when the certificates were created.
3. Validate and download the updated configuration.

4. Stop the Quay container and then restart the registry by entering the following command:

   $ sudo podman rm -f quay
   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
   --name=quay \
   -v $QUAY/config:/conf/stack:Z \
   -v $QUAY/storage:/datastorage:Z \
   registry.redhat.io/quay/quay-rhel8:v3.13.1

Procedure

1. Enter the following command to attempt to log in to the Red Hat Quay registry with SSL/TLS enabled:

   $ sudo podman login quay-server.example.com

   Example output

   Error: error authenticating creds for "quay-server.example.com": error pinging docker registry quay-server.example.com: Get "https://quay-server.example.com/v2/": x509: certificate signed by unknown authority

2. Because Podman does not trust self-signed certificates, you must use the --tls-verify=false option:

   $ sudo podman login --tls-verify=false quay-server.example.com

   Example output

   Login Succeeded!

   In a subsequent section, you will configure Podman to trust the root Certificate Authority.

Procedure

1. Navigate to your Red Hat Quay registry endpoint, for example, https://quay-server.example.com. If configured correctly, the browser warns of the potential risk:

   

2. Proceed to the log in screen. The browser notifies you that the connection is not secure. For example:

   

   In the following section, you will configure Podman to trust the root Certificate Authority.

Procedure

1. Copy the root CA file to one of /etc/containers/certs.d/ or /etc/docker/certs.d/. Use the exact path determined by the server hostname, and name the file ca.crt:

   $ sudo cp rootCA.pem /etc/containers/certs.d/quay-server.example.com/ca.crt

2. Verify that you no longer need to use the --tls-verify=false option when logging in to your Red Hat Quay registry:

   $ sudo podman login quay-server.example.com

   Example output

   Login Succeeded!

Procedure

1. Enter the following command to copy the rootCA.pem file to the consolidated system-wide trust store:

   $ sudo cp rootCA.pem /etc/pki/ca-trust/source/anchors/

2. Enter the following command to update the system-wide trust store configuration:

   $ sudo update-ca-trust extract

3. Optional. You can use the trust list command to ensure that the Quay server has been configured:

   $ trust list | grep quay
       label: quay-server.example.com

   Now, when you browse to the registry at https://quay-server.example.com, the lock icon shows that the connection is secure:

   

4. To remove the rootCA.pem file from system-wide trust, delete the file and update the configuration:

   $ sudo rm /etc/pki/ca-trust/source/anchors/rootCA.pem

   $ sudo update-ca-trust extract

   $ trust list | grep quay

Procedure

1. Base64 encode the contents of an SSL/TLS certificate by entering the following command:

   $ cat ca.crt | base64 -w 0

   Example output

   ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

2. Enter the following kubectl command to edit the quay-enterprise-config-secret file:

   $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret

3. Add an entry for the certificate and paste the full base64 encoded stringer under the entry. For example:

     custom-cert.crt:
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

4. Use the kubectl delete command to remove all Red Hat Quay pods. For example:

   $ kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6  quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms

   Afterwards, the Red Hat Quay deployment automatically schedules replace pods with the new certificate data.

Procedure

1. Obtain an Elasticsearch account.

2. Update your Red Hat Quay config.yaml file to include the following information:

   # ...
   LOGS_MODEL: elasticsearch 1
   LOGS_MODEL_CONFIG:
       producer: elasticsearch 2
       elasticsearch_config:
           host: http://<host.elasticsearch.example>:<port> 3
           port: 9200 4
           access_key: <access_key> 5
           secret_key: <secret_key> 6
           use_ssl: True 7
           index_prefix: <logentry> 8
           aws_region: <us-east-1> 9
   # ...

   1

   The method for handling log data.

   2

   Choose either Elasticsearch or Kinesis to direct logs to an intermediate Kinesis stream on AWS. You need to set up your own pipeline to send logs from Kinesis to Elasticsearch, for example, Logstash.

   3

   The hostname or IP address of the system providing the Elasticsearch service.

   4

   The port number providing the Elasticsearch service on the host you just entered. Note that the port must be accessible from all systems running the Red Hat Quay registry. The default is TCP port 9200.

   5

   The access key needed to gain access to the Elasticsearch service, if required.

   6

   The secret key needed to gain access to the Elasticsearch service, if required.

   7

   Whether to use SSL/TLS for Elasticsearch. Defaults to True.

   8

   Choose a prefix to attach to log entries.

   9

   If you are running on AWS, set the AWS region (otherwise, leave it blank).

3. Optional. If you are using Kinesis as your logs producer, you must include the following fields in your config.yaml file:

       kinesis_stream_config:
           stream_name: <kinesis_stream_name> 1
           access_key: <aws_access_key> 2
           secret_key: <aws_secret_key> 3
           aws_region: <aws_region> 4

   1

   The name of the Kinesis stream.

   2

   The name of the AWS access key needed to gain access to the Kinesis stream, if required.

   3

   The name of the AWS secret key needed to gain access to the Kinesis stream, if required.

   4

   The Amazon Web Services (AWS) region.

4. Save your config.yaml file and restart your Red Hat Quay deployment.

Procedure

1. In your Red Hat Quay installation directory, create a new directory for the Clair database data:

   $ mkdir /home/<user-name>/quay-poc/postgres-clairv4

2. Set the appropriate permissions for the postgres-clairv4 file by entering the following command:

   $ setfacl -m u:26:-wx /home/<user-name>/quay-poc/postgres-clairv4

3. Deploy a Clair PostgreSQL database by entering the following command:

   $ sudo podman run -d --name postgresql-clairv4 \
     -e POSTGRESQL_USER=clairuser \
     -e POSTGRESQL_PASSWORD=clairpass \
     -e POSTGRESQL_DATABASE=clair \
     -e POSTGRESQL_ADMIN_PASSWORD=adminpass \
     -p 5433:5432 \
     -v /home/<user-name>/quay-poc/postgres-clairv4:/var/lib/pgsql/data:Z \
     registry.redhat.io/rhel8/postgresql-15

4. Install the PostgreSQL uuid-ossp module for your Clair deployment:

   $ sudo podman exec -it postgresql-clairv4 /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"" | psql -d clair -U postgres'

   Example output

   CREATE EXTENSION

   Note

   Clair requires the uuid-ossp extension to be added to its PostgreSQL database. For users with proper privileges, creating the extension will automatically be added by Clair. If users do not have the proper privileges, the extension must be added before start Clair.

   If the extension is not present, the following error will be displayed when Clair attempts to start: ERROR: Please load the "uuid-ossp" extension. (SQLSTATE 42501).

5. Stop the Quay container if it is running and restart it in configuration mode, loading the existing configuration as a volume:

   $ sudo podman run --rm -it --name quay_config \
     -p 80:8080 -p 443:8443 \
     -v $QUAY/config:/conf/stack:Z \
     registry.redhat.io/quay/quay-rhel8:v3.13.1 config secret

6. Log in to the configuration tool and click Enable Security Scanning in the Security Scanner section of the UI.
7. Set the HTTP endpoint for Clair using a port that is not already in use on the quay-server system, for example, 8081.

8. Create a pre-shared key (PSK) using the Generate PSK button.

   Security Scanner UI

   

9. Validate and download the config.yaml file for Red Hat Quay, and then stop the Quay container that is running the configuration editor.

10. Extract the new configuration bundle into your Red Hat Quay installation directory, for example:

    $ tar xvf quay-config.tar.gz -d /home/<user-name>/quay-poc/

11. Create a folder for your Clair configuration file, for example:

    $ mkdir /etc/opt/clairv4/config/

12. Change into the Clair configuration folder:

    $ cd /etc/opt/clairv4/config/

13. Create a Clair configuration file, for example:

    http_listen_addr: :8081
    introspection_addr: :8088
    log_level: debug
    indexer:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      scanlock_retry: 10
      layer_scan_concurrency: 5
      migrations: true
    matcher:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      max_conn_pool: 100
      migrations: true
      indexer_addr: clair-indexer
    notifier:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      delivery_interval: 1m
      poll_interval: 5m
      migrations: true
    auth:
      psk:
        key: "MTU5YzA4Y2ZkNzJoMQ=="
        iss: ["quay"]
    # tracing and metrics
    trace:
      name: "jaeger"
      probability: 1
      jaeger:
        agent:
          endpoint: "localhost:6831"
        service_name: "clair"
    metrics:
      name: "prometheus"

    For more information about Clair’s configuration format, see Clair configuration reference.

14. Start Clair by using the container image, mounting in the configuration from the file you created:

    $ sudo podman run -d --name clairv4 \
    -p 8081:8081 -p 8088:8088 \
    -e CLAIR_CONF=/clair/config.yaml \
    -e CLAIR_MODE=combo \
    -v /etc/opt/clairv4/config:/clair:Z \
    registry.redhat.io/quay/clair-rhel8:v3.13.1

    Note

    Running multiple Clair containers is also possible, but for deployment scenarios beyond a single container the use of a container orchestrator like Kubernetes or OpenShift Container Platform is strongly recommended.

Procedure

1. Pull a sample image by entering the following command:

   $ podman pull ubuntu:20.04

2. Tag the image to your registry by entering the following command:

   $ sudo podman tag docker.io/library/ubuntu:20.04 <quay-server.example.com>/<user-name>/ubuntu:20.04

3. Push the image to your Red Hat Quay registry by entering the following command:

   $ sudo podman push --tls-verify=false quay-server.example.com/quayadmin/ubuntu:20.04

4. Log in to your Red Hat Quay deployment through the UI.
5. Click the repository name, for example, quayadmin/ubuntu.

6. In the navigation pane, click Tags.

   Report summary

   

7. Click the image report, for example, 45 medium, to show a more detailed report:

   Report details

   

   Note

   In some cases, Clair shows duplicate reports on images, for example, ubi8/nodejs-12 or ubi8/nodejs-16. This occurs because vulnerabilities with same name are for different packages. This behavior is expected with Clair vulnerability reporting and will not be addressed as a bug.

1. Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.

   

2. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file.

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to IPv6, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: IPv6
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to IPv6 by entering the following command:

   $ curl <quay_endpoint>/health/instance
   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to dual-stack, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: dual-stack
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to both channels by entering the following command:

   1. For IPv4, enter the following command:

      $ curl --ipv4 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   2. For IPv6, enter the following command:

      $ curl --ipv6 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

1. Update your config.yaml file directly to include the following relevant information:

   # ...
   AUTHENTICATION_TYPE: LDAP 1
   # ...
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com 2
   LDAP_ADMIN_PASSWD: ABC123 3
   LDAP_ALLOW_INSECURE_FALLBACK: false 4
   LDAP_BASE_DN: 5
     - dc=example
     - dc=com
   LDAP_EMAIL_ATTR: mail 6
   LDAP_UID_ATTR: uid 7
   LDAP_URI: ldap://<example_url>.com 8
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,dc=<domain_name>,dc=com) 9
   LDAP_USER_RDN: 10
     - ou=people
   LDAP_SECONDARY_USER_RDNS: 11
       - ou=<example_organization_unit_one>
       - ou=<example_organization_unit_two>
       - ou=<example_organization_unit_three>
       - ou=<example_organization_unit_four>
   # ...

   1

   Required. Must be set to LDAP.

   2

   Required. The admin DN for LDAP authentication.

   3

   Required. The admin password for LDAP authentication.

   4

   Required. Whether to allow SSL/TLS insecure fallback for LDAP authentication.

   5

   Required. The base DN for LDAP authentication.

   6

   Required. The email attribute for LDAP authentication.

   7

   Required. The UID attribute for LDAP authentication.

   8

   Required. The LDAP URI.

   9

   Required. The user filter for LDAP authentication.

   10

   Required. The user RDN for LDAP authentication.

   11

   Optional. Secondary User Relative DNs if there are multiple Organizational Units where user objects are located.

2. After you have added all required LDAP fields, save the changes and restart your Red Hat Quay deployment.

Procedure

1. In your deployment’s config.yaml file, add the LDAP_RESTRICTED_USER_FILTER parameter and specify the group of restricted users, for example, members:

   # ...
   AUTHENTICATION_TYPE: LDAP
   # ...
   FEATURE_RESTRICTED_USERS: true 1
   # ...
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,o=<example_organization_unit>,dc=<example_domain_component>,dc=com)
   LDAP_RESTRICTED_USER_FILTER: (<filterField>=<value>) 2
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   # ...

   1

   Must be set to true when configuring an LDAP restricted user.

   2

   Configures specified users as restricted users.

2. Start, or restart, your Red Hat Quay deployment.

Procedure

1. In your deployment’s config.yaml file, add the LDAP_SUPERUSER_FILTER parameter and add the group of users you want configured as super users, for example, root:

   # ...
   AUTHENTICATION_TYPE: LDAP
   # ...
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,o=<example_organization_unit>,dc=<example_domain_component>,dc=com)
   LDAP_SUPERUSER_FILTER: (<filterField>=<value>) 1
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   # ...

   1

   Configures specified users as superusers.

2. Start, or restart, your Red Hat Quay deployment.

1. Use the following reference and update your config.yaml file with your desired OIDC provider’s credentials:

   AUTHENTICATION_TYPE: OIDC
   # ...
   AZURE_LOGIN_CONFIG: 1
       CLIENT_ID: <client_id> 2
       CLIENT_SECRET: <client_secret> 3
       OIDC_SERVER: <oidc_server_address_> 4
       SERVICE_NAME: Microsoft Entra ID 5
       VERIFIED_EMAIL_CLAIM_NAME: <verified_email> 6
   # ...

   1

   The parent key that holds the OIDC configuration settings. In this example, the parent key used is AZURE_LOGIN_CONFIG, however, the string AZURE can be replaced with any arbitrary string based on your specific needs, for example ABC123.However, the following strings are not accepted: GOOGLE, GITHUB. These strings are reserved for their respective identity platforms and require a specific config.yaml entry contingent upon when platform you are using.

   2

   The client ID of the application that is being registered with the identity provider.

   3

   The client secret of the application that is being registered with the identity provider.

   4

   The address of the OIDC server that is being used for authentication. In this example, you must use sts.windows.net as the issuer identifier. Using https://login.microsoftonline.com results in the following error: Could not create provider for AzureAD. Error: oidc: issuer did not match the issuer returned by provider, expected "https://login.microsoftonline.com/73f2e714-xxxx-xxxx-xxxx-dffe1df8a5d5" got "https://sts.windows.net/73f2e714-xxxx-xxxx-xxxx-dffe1df8a5d5/".

   5

   The name of the service that is being authenticated.

   6

   The name of the claim that is used to verify the email address of the user.

2. Proper configuration of Microsoft Entra ID results three redirects with the following format:

   • https://QUAY_HOSTNAME/oauth2/<name_of_service>/callback
   • https://QUAY_HOSTNAME/oauth2/<name_of_service>/callback/attach
   • https://QUAY_HOSTNAME/oauth2/<name_of_service>/callback/cli
3. Restart your Red Hat Quay deployment.

Procedure

1. On the Red Hat Single Sign-On UI:

   1. Click Clients and then the name of the application or service that can request authentication of a user.

   2. On the Settings page of your client, ensure that the following options are set or enabled:

      • Client ID
      • Valid redirect URI
      • Client authentication
      • Authorization
      • Standard flow

      • Direct access grants

        Note

        Settings can differ depending on your setup.

   3. On the Credentials page, store the Client Secret for future use.
   4. On the Users page, click Add user and enter a username, for example, service-account-quaydev. Then, click Create.
   5. Click the name of of the user, for example service-account-quaydev on the Users page.
   6. Click the Credentials tab → Set password → and provide a password for the user. If warranted, you can make this password temporary by selecting the Temporary option.

   7. Click the Realm settings tab → OpenID Endpoint Configuration. Store the /protocol/openid-connect/token endpoint. For example:

      http://localhost:8080/realms/master/protocol/openid-connect/token

2. On a web browser, navigate to the following URL:

   http://<keycloak_url>/realms/<realm_name>/protocol/openid-connect/auth?response_type=code&client_id=<client_id>

3. When prompted, log in with the service-account-quaydev user and the temporary password you set. Complete the login by providing the required information and setting a permanent password if necessary.

4. You are redirected to the URI address provided for your client. For example:

   https://localhost:3000/cb?session_state=5c9bce22-6b85-4654-b716-e9bbb3e755bc&iss=http%3A%2F%2Flocalhost%3A8080%2Frealms%2Fmaster&code=ea5b76eb-47a5-4e5d-8f71-0892178250db.5c9bce22-6b85-4654-b716-e9bbb3e755bc.cdffafbc-20fb-42b9-b254-866017057f43

   Take note of the code provided in the address. For example:

   code=ea5b76eb-47a5-4e5d-8f71-0892178250db.5c9bce22-6b85-4654-b716-e9bbb3e755bc.cdffafbc-20fb-42b9-b254-866017057f43

   Note

   This is a temporary code that can only be used one time. If necessary, you can refresh the page or revisit the URL to obtain another code.

5. On your terminal, use the following curl -X POST command to generate a temporary OAuth2 access token:

   $ curl -X POST "http://localhost:8080/realms/master/protocol/openid-connect/token" 1
   -H "Content-Type: application/x-www-form-urlencoded" \
   -d "client_id=quaydev" 2
   -d "client_secret=g8gPsBLxVrLo2PjmZkYBdKvcB9C7fmBz" 3
   -d "grant_type=authorization_code"
   -d "code=ea5b76eb-47a5-4e5d-8f71-0892178250db.5c9bce22-6b85-4654-b716-e9bbb3e755bc.cdffafbc-20fb-42b9-b254-866017057f43" 4

   1

   The protocol/openid-connect/token endpoint found on the Realm settings page of the Red Hat Single Sign-On UI.

   2

   The Client ID used for this procedure.

   3

   The Client Secret for the Client ID.

   4

   The code returned from the redirect URI.

   Example output

   {"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJTVmExVHZ6eDd2cHVmc1dkZmc1SHdua1ZDcVlOM01DN1N5T016R0QwVGhVIn0...",
   "expires_in":60,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJiNTBlZTVkMS05OTc1LTQwMzUtYjNkNy1lMWQ5ZTJmMjg0MTEifQ.oBDx6B3pUkXQO8m-M3hYE7v-w25ak6y70CQd5J8f5EuldhvTwpWrC1K7yOglvs09dQxtq8ont12rKIoCIi4WXw","token_type":"Bearer","not-before-policy":0,"session_state":"5c9bce22-6b85-4654-b716-e9bbb3e755bc","scope":"profile email"}

6. Store the access_token from the previously step, as it will be exchanged for a Red Hat Quay robot account token in the following procedure.

Procedure

1. On the Red Hat Single Sign-On main page:

   1. Select the appropriate realm that is authenticated for use with Red Hat Quay. Store the issuer URL, for example, https://keycloak-auth-realm.quayadmin.org/realms/quayrealm.
   2. Click Users → the name of the user to be linked with the robot account for authentication. You must use the same user account that you used when generating the OAuth2 access token.

   3. On the Details page, store the ID of the user, for example, 449e14f8-9eb5-4d59-a63e-b7a77c75f770.

      Note

      The information collected in this step will vary depending on your OIDC provider. For example, with Red Hat Single Sign-On, the ID of a user is used as the Subject to set up the robot account federation in a subsequent step. For a different OIDC provider, like Microsoft Entra ID, this information is stored as the Subject.

2. On your Red Hat Quay registry:

   1. Navigate to Organizations and click the name of your organization, for example, fed_test.
   2. Click Robot Accounts.
   3. Click the menu kebab → Set robot federation.
   4. Click the + symbol.

   5. In the popup window, include the following information:

      • Issuer URL: https://keycloak-auth-realm.quayadmin.org/realms/quayrealm. For Red Hat Single Sign-On, this is the the URL of your Red Hat Single Sign-On realm. This might vary depending on your OIDC provider.
      • Subject: 449e14f8-9eb5-4d59-a63e-b7a77c75f770. For Red Hat Single Sign-On, the Subject is the ID of your Red Hat Single Sign-On user. This varies depending on your OIDC provider. For example, if you are using Microsoft Entra ID, the Subject will be the Subject or your Entra ID user.
   6. Click Save.

Procedure

1. Save the following Python script in a .py file, for example, robot_fed_token_auth.py

   import requests
   import os
   
   TOKEN=os.environ.get('TOKEN')
   robot_user = "fed-test+robot1"
   
   def get_quay_robot_token(fed_token):
       URL = "https://<quay-server.example.com>/oauth2/federation/robot/token"
       response = requests.get(URL, auth=(robot_user,fed_token)) 1
       print(response)
       print(response.text)
   
   if __name__ == "__main__":
       get_quay_robot_token(TOKEN)

   1

   If your Red Hat Quay deployment is using custom SSL/TLS certificates, the response must be response = requests.get(URL,auth=(robot_user,fed_token),verify=False), which includes the verify=False flag.

2. Export the OAuth2 access token as TOKEN. For example:

   $ export TOKEN = eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJTVmExVHZ6eDd2cHVmc1dkZmc1SHdua1ZDcVlOM01DN1N5T016R0QwVGhVIn0...

3. Run the robot_fed_token_auth.py script by entering the following command:

   $ python3 robot_fed_token_auth.py

   Example output

   <Response [200]>
   {"token": "291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJlbWFpbF92ZXJpZ..."}

   Important

   This token expires after one hour. After one hour, a new token must be generated.

4. Export the robot account access token as QUAY_TOKEN. For example:

   $ export QUAY_TOKEN=291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJlbWFpbF92ZXJpZ

Procedure

1. Log in to your Red Hat Quay registry using the fest_test+robot1 robot account and the QUAY_TOKEN access token. For example:

   $ podman login <quay-server.example.com> -u fed_test+robot1 -p $QUAY_TOKEN

2. Pull an image from a Red Hat Quay repository for which the robot account has the proper permissions. For example:

   $ podman pull <quay-server.example.com/<repository_name>/<image_name>>

   Example output

   Getting image source signatures
   Copying blob 900e6061671b done
   Copying config 8135583d97 done
   Writing manifest to image destination
   Storing signatures
   8135583d97feb82398909c9c97607159e6db2c4ca2c885c0b8f590ee0f9fe90d
   0.57user 0.11system 0:00.99elapsed 68%CPU (0avgtext+0avgdata 78716maxresident)k
   800inputs+15424outputs (18major+6528minor)pagefaults 0swaps

3. Attempt to pull an image from a Red Hat Quay repository for which the robot account does not have the proper permissions. For example:

   $ podman pull <quay-server.example.com/<different_repository_name>/<image_name>>

   Example output

   Error: initializing source docker://quay-server.example.com/example_repository/busybox:latest: reading manifest in quay-server.example.com/example_repository/busybox: unauthorized: access to the requested resource is not authorized

   After one hour, the credentials for this robot account are set to expire. Afterwards, you must generate a new access token for this robot account.

Procedure

1. Log in to the Amazon Web Services (AWS) console and navigate to the Identity and Access Management (IAM) console.
2. In the navigation pane, under Access management click Users.

3. Click Create User and enter the following information:

   1. Enter a valid username, for example, quay-user.
   2. For Permissions options, click Add user to group.
4. On the review and create page, click Create user. You are redirected to the Users page.
5. Click the username, for example, quay-user.
6. Copy the ARN of the user, for example, arn:aws:iam::123492922789:user/quay-user.
7. On the same page, click the Security credentials tab.
8. Navigate to Access keys.
9. Click Create access key.
10. On the Access key best practices & alternatives page, click Command Line Interface (CLI), then, check the confirmation box. Then click Next.
11. Optional. On the Set description tag - optional page, enter a description.
12. Click Create access key.

13. Copy and store the access key and the secret access key.

    Important

    This is the only time that the secret access key can be viewed or downloaded. You cannot recover it later. However, you can create a new access key any time.

14. Click Done.

Procedure

1. If you are not already, navigate to the IAM dashboard by clicking Dashboard.
2. In the navigation pane, click Roles under Access management.

3. Click Create role.

   • Click Custom Trust Policy, which shows an editable JSON policy. By default, it shows the following information:

     {
     	"Version": "2012-10-17",
     	"Statement": [
     		{
     			"Sid": "Statement1",
     			"Effect": "Allow",
     			"Principal": {},
     			"Action": "sts:AssumeRole"
     		}
     	]
     }

4. Under the Principal configuration field, add your AWS ARN information. For example:

   {
       "Version": "2012-10-17",
       "Statement": [
      	 {
      		 "Sid": "Statement1",
      		 "Effect": "Allow",
      		 "Principal": {
      		 	"AWS": "arn:aws:iam::123492922789:user/quay-user"
      		 },
      		 "Action": "sts:AssumeRole"
      	 }
       ]
   }

5. Click Next.
6. On the Add permissions page, type AmazonS3FullAccess in the search box. Check the box to add that policy to the S3 role, then click Next.

7. On the Name, review, and create page, enter the following information:

   1. Enter a role name, for example, example-role.
   2. Optional. Add a description.
8. Click the Create role button. You are navigated to the Roles page. Under Role name, the newly created S3 should be available.

Procedure

1. Update your config.yaml file for Red Hat Quay to include the following information:

   # ...
   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - STSS3Storage
       - sts_role_arn: <role_arn> 1
         s3_bucket: <s3_bucket_name> 2
         storage_path: <storage_path> 3
         s3_region: <region> 4
         sts_user_access_key: <s3_user_access_key> 5
         sts_user_secret_key: <s3_user_secret_key> 6
   # ...

   1

   The unique Amazon Resource Name (ARN) required when configuring AWS STS

   2

   The name of your s3 bucket.

   3

   The storage path for data. Usually /datastorage.

   4

   Optional. The Amazon Web Services region. Defaults to us-east-1.

   5

   The generated AWS S3 user access key required when configuring AWS STS.

   6

   The generated AWS S3 user secret key required when configuring AWS STS.

2. Restart your Red Hat Quay deployment.

Verification

1. Tag a sample image, for example, busybox, that will be pushed to the repository. For example:

   $ podman tag docker.io/library/busybox <quay-server.example.com>/<organization_name>/busybox:test

2. Push the sample image by running the following command:

   $ podman push <quay-server.example.com>/<organization_name>/busybox:test

3. Verify that the push was successful by navigating to the Organization that you pushed the image to in your Red Hat Quay registry → Tags.
4. Navigate to the Amazon Web Services (AWS) console and locate your s3 bucket.
5. Click the name of your s3 bucket.
6. On the Objects page, click datastorage/.

7. On the datastorage/ page, the following resources should seen:

   • sha256/

   • uploads/

     These resources indicate that the push was successful, and that AWS STS is properly configured.

Procedure

1. Pull a sample image, for example, ubuntu:18.04, by entering the following command:

   $ podman pull ubuntu:18.04

2. Tag the same image twice by entering the following command:

   $ podman tag docker.io/library/ubuntu:18.04 quay-server.example.com/quota-test/ubuntu:tag1

   $ podman tag docker.io/library/ubuntu:18.04 quay-server.example.com/quota-test/ubuntu:tag2

3. Push the sample image to your organization by entering the following commands:

   $ podman push --tls-verify=false quay-server.example.com/quota-test/ubuntu:tag1

   $ podman push --tls-verify=false quay-server.example.com/quota-test/ubuntu:tag2

4. On the Red Hat Quay UI, navigate to Organization and click the Repository Name, for example, quota-test/ubuntu. Then, click Tags. There should be two repository tags, tag1 and tag2, each with the same manifest. For example:

   

   However, by clicking on the Organization link, we can see that the Total Quota Consumed is 27.94 MB, meaning that the Ubuntu image has only been accounted for once:

   

   If you delete one of the Ubuntu tags, the Total Quota Consumed remains the same.

   Note

   If you have configured the Red Hat Quay time machine to be longer than 0 seconds, subtraction will not happen until those tags pass the time machine window. If you want to expedite permanent deletion, see Permanently deleting an image tag in Red Hat Quay 3.9.

Procedure

1. Create a new organization or choose an existing one. Initially, no quota is configured, as can be seen on the Organization Settings tab:

   

2. Log in to the registry as a superuser and navigate to the Manage Organizations tab on the Super User Admin Panel. Click the Options icon of the organization for which you want to create storage quota limits:

   

3. Click Configure Quota and enter the initial quota, for example, 10 MB. Then click Apply and Close:

   

4. Check that the quota consumed shows 0 of 10 MB on the Manage Organizations tab of the superuser panel:

   

   The consumed quota information is also available directly on the Organization page:

   Initial consumed quota

   

5. To increase the quota to 100MB, navigate to the Manage Organizations tab on the superuser panel. Click the Options icon and select Configure Quota, setting the quota to 100 MB. Click Apply and then Close:

   

6. Pull a sample image by entering the following command:

   $ podman pull ubuntu:18.04

7. Tag the sample image by entering the following command:

   $ podman tag docker.io/library/ubuntu:18.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

8. Push the sample image to the organization by entering the following command:

   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

9. On the superuser panel, the quota consumed per organization is displayed:

   

10. The Organization page shows the total proportion of the quota used by the image:

    Total Quota Consumed for first image

    

11. Pull a second sample image by entering the following command:

    $ podman pull nginx

12. Tag the second image by entering the following command:

    $ podman tag docker.io/library/nginx example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

13. Push the second image to the organization by entering the following command:

    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

14. The Organization page shows the total proportion of the quota used by each repository in that organization:

    Total Quota Consumed for each repository

    

15. Create reject and warning limits:

    From the superuser panel, navigate to the Manage Organizations tab. Click the Options icon for the organization and select Configure Quota. In the Quota Policy section, with the Action type set to Reject, set the Quota Threshold to 80 and click Add Limit:

    

16. To create a warning limit, select Warning as the Action type, set the Quota Threshold to 70 and click Add Limit:

    

17. Click Close on the quota popup. The limits are viewable, but not editable, on the Settings tab of the Organization page:

    

18. Push an image where the reject limit is exceeded:

    Because the reject limit (80%) has been set to below the current repository size (~83%), the next pushed image is rejected automatically.

    Sample image push

    $ podman pull ubuntu:20.04
    
    $ podman tag docker.io/library/ubuntu:20.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04
    
    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04

    Sample output when quota exceeded

    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0002] failed, retrying in 1s ... (1/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0005] failed, retrying in 1s ... (2/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0009] failed, retrying in 1s ... (3/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace

19. When limits are exceeded, notifications are displayed in the UI:

    Quota notifications

    

Procedure

1. On the Red Hat Quay UI, click your username → Super User Admin Panel.
2. In the navigation pane, click Manage Organizations.
3. Click Calculate, next to Total Registry Size: 0.00 KB, Updated: Never , Calculation required. Then, click Ok.

4. After a few minutes, depending on the size of your registry, refresh the page. Now, the Total Registry Size should be calculated. For example:

   

Procedure

1. Create a temporary backup directory, for example, quay-backup:

   $ mkdir /tmp/quay-backup

2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install:

   $ podman run --name quay-app \
      -v /opt/quay-install/config:/conf/stack:Z \
      -v /opt/quay-install/storage:/datastorage:Z \
      registry.redhat.io/quay/quay-rhel8:v3.13.1

   Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command:

   $ cd /opt/quay-install

3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command:

   $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

4. Back up the Quay container service by entering the following command:

   $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -
   
     /usr/bin/podman run --name quay-app \
     -v /opt/quay-install/config:/conf/stack:Z \
     -v /opt/quay-install/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.13.1

5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command:

   $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command:

   $ grep DB_URI /tmp/quay-backup/quay-config.yaml

   Example output:

   $ postgresql://<username>:test123@172.24.10.50/quay

7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command:

   $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command:

   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - S3Storage
       - s3_bucket: <bucket_name>
         storage_path: /registry
         s3_access_key: <s3_access_key>
         s3_secret_key: <s3_secret_key>
         host: <host_name>
         s3_region: <region>

9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7:

   $ export AWS_ACCESS_KEY_ID=<access_key>

10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG:

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
    download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

Procedure

1. Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container:

   $ mkdir /opt/new-quay-install

2. Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1:

   $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

3. Change into the new-quay-install directory by entering the following command:

   $ cd /opt/new-quay-install/

4. Extract the contents of your Red Hat Quay directory:

   $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

5. Recall the DB_URI from your backed-up config.yaml file by entering the following command:

   $ grep DB_URI config.yaml

   Example output:

   postgresql://<username>:test123@172.24.10.50/quay

6. Run the following command to enter the PostgreSQL database server:

   $ sudo postgres

7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command:

   $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
   postgres=> CREATE DATABASE example_restore_registry_quay_database;

   Example output:

   CREATE DATABASE

8. Connect to the database by running the following command:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

9. Create a pg_trmg extension of your Quay database by running the following command:

   example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Example output:

   CREATE EXTENSION

10. Exit the postgres CLI by entering the following command:

    \q

11. Import the database backup to your new database by running the following command:

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle:

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_region: <region>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13:

    $ export AWS_ACCESS_KEY_ID=<access_key>

14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

15. Create a new s3 bucket by entering the following command:

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

16. Upload all blobs to the new s3 bucket by entering the following command:

    $ aws s3 sync --no-verify-ssl \
    --endpoint-url <example_endpoint_url> 1
    /tmp/quay-backup/blob-backup/. s3://quay/

    1

    The Red Hat Quay registry endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
    ...

17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <new_bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          s3_region: <region>
          host: <host_name>

Procedure

1. Back up the config.yaml of your standalone Red Hat Quay deployment:

   $ mkdir /tmp/quay-backup
   $ cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup

2. Create a backup of the database that your standalone Red Hat Quay deployment is using:

   $ pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql

3. Install the AWS CLI if you do not have it already.

4. Create an ~/.aws/ directory:

   $ mkdir ~/.aws/

5. Obtain the access_key and secret_key from the config.yaml of your standalone deployment:

   $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml

   Example output:

   DISTRIBUTED_STORAGE_CONFIG:
       minio-1:
           - RadosGWStorage
           - access_key: ##########
             bucket_name: quay
             hostname: 172.24.10.50
             is_secure: false
             port: "9000"
             secret_key: ##########
             storage_path: /datastorage/registry

6. Store the access_key and secret_key from the config.yaml file in your ~/.aws directory:

   $ touch ~/.aws/credentials

7. Optional: Check that your access_key and secret_key are stored:

   $ cat > ~/.aws/credentials << EOF
   [default]
   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
   EOF

   Example output:

   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG

   Note

   If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can, you can configure these by running aws configure and manually inputting the credentials.

8. In your quay-backup directory, create a bucket_backup directory:

   $ mkdir /tmp/quay-backup/bucket-backup

9. Backup all blobs from the S3 storage:

   $ aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/

   Note

   The PUBLIC_S3_ENDPOINT can be read from the Red Hat Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG. If the endpoint is insecure, use http instead of https in the endpoint URL.

1. Scale down the Red Hat Quay Operator:

   $ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators

2. Scale down the application and mirror deployments:

   $ oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT

3. Copy the database SQL backup to the Quay PostgreSQL database instance:

   $ oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata

4. Obtain the database password from the Operator-created config.yaml file:

   $ oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret

   Example output:

         "name": "QUAY_CONFIG_SECRET_NAME"

   $ oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

   cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI

   Example output:

   postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME

5. Execute a shell inside of the database pod:

   # oc exec -it quay-postgresql-database-pod -- /bin/bash

6. Enter psql:

   bash-4.4$ psql

7. Drop the database:

   postgres=# DROP DATABASE "example-restore-registry-quay-database";

   Example output:

   DROP DATABASE

8. Create a new database and set the owner as the same name:

   postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database";

   Example output:

   CREATE DATABASE

9. Connect to the database:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

10. Create a pg_trmg extension of your Quay database:

    example-restore-registry-quay-database=# create extension pg_trgm ;

    Example output:

    CREATE EXTENSION

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Set the password for your PostgreSQL deployment:

    bash-4.4$ psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

13. Exit bash mode:

    bash-4.4$ exit

14. Create a new configuration bundle for the Red Hat Quay Operator.

    $ touch config-bundle.yaml

15. In your new config-bundle.yaml, include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml:

    $ cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml

    Note

    You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file.

16. Create a configuration bundle secret inside of your OpenShift cluster:

    $ oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml

17. Scale up the Quay pods:

    $ oc scale --replicas=1 deployment quayregistry-quay-app
    deployment.apps/quayregistry-quay-app scaled

18. Scale up the mirror pods:

    $ oc scale --replicas=1  deployment quayregistry-quay-mirror
    deployment.apps/quayregistry-quay-mirror scaled

19. Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle:

    $ oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}'

    Note

    If Red Hat Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default.

20. Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file:

    $ touch credentials.yaml

    $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

    $ cat > ~/.aws/credentials << EOF
    [default]
    aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
    aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
    EOF

    Note

    If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can configure these by running aws configure and manually inputting the credentials.

21. Record the NooBaa’s publicly available endpoint:

    $ oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}"

22. Sync the backup data to the NooBaa backend storage:

    $ aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME

23. Scale the Operator back up to 1 pod:

    $ oc scale –replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators

Procedure

1. Add the IGNORE_UNKNOWN_MEDIATYPES configuration field to your Red Hat Quay config.yaml file:

   IGNORE_UNKNOWN_MEDIATYPES: true

   With this field enabled, your Red Hat Quay deployment accepts unknown and unrecognized artifact types.

Procedure

1. Enter the following command to ensure that garbage collection is properly working:

   $ sudo podman logs <container_id>

   Example output:

   gcworker stdout | 2022-11-14 18:46:52,458 [63] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2022-11-14 18:47:22 UTC)" executed successfully

2. Delete an image tag.

3. Enter the following command to ensure that the tag was deleted:

   $ podman logs quay-app

   Example output:

   gunicorn-web stdout | 2022-11-14 19:23:44,574 [233] [INFO] [gunicorn.access] 192.168.0.38 - - [14/Nov/2022:19:23:44 +0000] "DELETE /api/v1/repository/quayadmin/busybox/tag/test HTTP/1.0" 204 0 "http://quay-server.example.com/repository/quayadmin/busybox?tab=tags" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"

Procedure

1. Log in to your deployment.

2. In the navigation pane of your deployment, you are given the option to toggle between Current UI and New UI. Click the toggle button to set it to new UI, and then click Use Beta Environment, for example: