Manage Red Hat Quay

Procedure

1. Copy the certificate file and primary key file to your configuration directory, ensuring they are named ssl.cert and ssl.key respectively:

   cp ~/ssl.cert ~/ssl.key $QUAY/config

2. Change into the $QUAY/config directory by entering the following command:

   $ cd $QUAY/config

3. Edit the config.yaml file and specify that you want Red Hat Quay to handle TLS/SSL:

   config.yaml

   ...
   SERVER_HOSTNAME: quay-server.example.com
   ...
   PREFERRED_URL_SCHEME: https
   ...

4. Optional: Append the contents of the rootCA.pem file to the end of the ssl.cert file by entering the following command:

   $ cat rootCA.pem >> ssl.cert

5. Stop the Quay container by entering the following command:

   $ sudo podman stop quay

6. Restart the registry by entering the following command:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
     --name=quay \
     -v $QUAY/config:/conf/stack:Z \
     -v $QUAY/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.9.7

Procedure

1. Start the Quay container in configuration mode:

   $ sudo podman run --rm -it --name quay_config -p 80:8080 -p 443:8443 registry.redhat.io/quay/quay-rhel8:v3.9.7 config secret

2. In the Server Configuration section, select Red Hat Quay handles TLS for SSL/TLS. Upload the certificate file and private key file created earlier, ensuring that the Server Hostname matches the value used when the certificates were created.
3. Validate and download the updated configuration.

4. Stop the Quay container and then restart the registry by entering the following command:

   $ sudo podman rm -f quay
   $ sudo podman run -d --rm -p 80:8080 -p 443:8443 \
   --name=quay \
   -v $QUAY/config:/conf/stack:Z \
   -v $QUAY/storage:/datastorage:Z \
   registry.redhat.io/quay/quay-rhel8:v3.9.7

Procedure

1. Enter the following command to copy the rootCA.pem file to the consolidated system-wide trust store:

   $ sudo cp rootCA.pem /etc/pki/ca-trust/source/anchors/

2. Enter the following command to update the system-wide trust store configuration:

   $ sudo update-ca-trust extract

3. Optional. You can use the trust list command to ensure that the Quay server has been configured:

   $ trust list | grep quay
       label: quay-server.example.com

   Now, when you browse to the registry at https://quay-server.example.com, the lock icon shows that the connection is secure:

   

4. To remove the rootCA.pem file from system-wide trust, delete the file and update the configuration:

   $ sudo rm /etc/pki/ca-trust/source/anchors/rootCA.pem

   $ sudo update-ca-trust extract

   $ trust list | grep quay

1. View certificate to be added to the container

   $ cat storage.crt
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV
   [...]
   -----END CERTIFICATE-----

2. Create certs directory and copy certificate there

   $ mkdir -p quay/config/extra_ca_certs
   $ cp storage.crt quay/config/extra_ca_certs/
   $ tree quay/config/
   ├── config.yaml
   ├── extra_ca_certs
   │   ├── storage.crt

3. Obtain the Quay container’s CONTAINER ID with podman ps:

   $ sudo podman ps
   CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS
   5a3e82c4a75f        <registry>/<repo>/quay:v3.9.7 "/sbin/my_init"          24 hours ago        Up 18 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 443/tcp   grave_keller

4. Restart the container with that ID:

   $ sudo podman restart 5a3e82c4a75f

5. Examine the certificate copied into the container namespace:

   $ sudo podman exec -it 5a3e82c4a75f cat /etc/ssl/certs/storage.pem
   -----BEGIN CERTIFICATE-----
   MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV

Procedure

1. Base64 encode the contents of an SSL/TLS certificate by entering the following command:

   $ cat ca.crt | base64 -w 0

   Example output

   ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

2. Enter the following kubectl command to edit the quay-enterprise-config-secret file:

   $ kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret

3. Add an entry for the certificate and paste the full base64 encoded stringer under the entry. For example:

     custom-cert.crt:
   c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

4. Use the kubectl delete command to remove all Red Hat Quay pods. For example:

   $ kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6 quayregistry-quay-config-editor-6dfdcfc44f-hlvwm quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms

   Afterwards, the Red Hat Quay deployment automatically schedules replace pods with the new certificate data.

Procedure

1. Obtain an Elasticsearch account.
2. Open the Red Hat Quay Config Tool (either during or after Red Hat Quay deployment).

3. Scroll to the Action Log Storage Configuration setting and select Elasticsearch. The following figure shows the Elasticsearch settings that appear:

   

4. Fill in the following information for your Elasticsearch instance:

   • Elasticsearch hostname: The hostname or IP address of the system providing the Elasticsearch service.
   • Elasticsearch port: The port number providing the Elasticsearch service on the host you just entered. Note that the port must be accessible from all systems running the Red Hat Quay registry. The default is TCP port 9200.
   • Elasticsearch access key: The access key needed to gain access to the Elastic search service, if required.
   • Elasticsearch secret key: The secret key needed to gain access to the Elastic search service, if required.
   • AWS region: If you are running on AWS, set the AWS region (otherwise, leave it blank).
   • Index prefix: Choose a prefix to attach to log entries.

   • Logs Producer: Choose either Elasticsearch (default) or Kinesis to direct logs to an intermediate Kinesis stream on AWS. You need to set up your own pipeline to send logs from Kinesis to Elasticsearch (for example, Logstash). The following figure shows additional fields you would need to fill in for Kinesis:

     

5. If you chose Elasticsearch as the Logs Producer, no further configuration is needed. If you chose Kinesis, fill in the following:

   • Stream name: The name of the Kinesis stream.
   • AWS access key: The name of the AWS access key needed to gain access to the Kinesis stream, if required.
   • AWS secret key: The name of the AWS secret key needed to gain access to the Kinesis stream, if required.
   • AWS region: The AWS region.
6. When you are done, save the configuration. The configuration tool checks your settings. If there is a problem connecting to the Elasticsearch or Kinesis services, you will see an error and have the opportunity to continue editing. Otherwise, logging will begin to be directed to your Elasticsearch configuration after the cluster restarts with the new configuration.

Procedure

1. In your Red Hat Quay installation directory, create a new directory for the Clair database data:

   $ mkdir /home/<user-name>/quay-poc/postgres-clairv4

2. Set the appropriate permissions for the postgres-clairv4 file by entering the following command:

   $ setfacl -m u:26:-wx /home/<user-name>/quay-poc/postgres-clairv4

3. Deploy a Clair Postgres database by entering the following command:

   $ sudo podman run -d --name postgresql-clairv4 \
     -e POSTGRESQL_USER=clairuser \
     -e POSTGRESQL_PASSWORD=clairpass \
     -e POSTGRESQL_DATABASE=clair \
     -e POSTGRESQL_ADMIN_PASSWORD=adminpass \
     -p 5433:5433 \
     -v /home/<user-name>/quay-poc/postgres-clairv4:/var/lib/pgsql/data:Z \
     registry.redhat.io/rhel8/postgresql-13:1-109

4. Install the Postgres uuid-ossp module for your Clair deployment:

   $ podman exec -it postgresql-clairv4 /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"" | psql -d clair -U postgres'

   Example output

   CREATE EXTENSION

   Note

   Clair requires the uuid-ossp extension to be added to its Postgres database. For users with proper privileges, creating the extension will automatically be added by Clair. If users do not have the proper privileges, the extension must be added before start Clair.

   If the extension is not present, the following error will be displayed when Clair attempts to start: ERROR: Please load the "uuid-ossp" extension. (SQLSTATE 42501).

5. Stop the Quay container if it is running and restart it in configuration mode, loading the existing configuration as a volume:

   $ sudo podman run --rm -it --name quay_config \
     -p 80:8080 -p 443:8443 \
     -v $QUAY/config:/conf/stack:Z \
     registry.redhat.io/quay/quay-rhel8:{productminv} config secret

6. Log in to the configuration tool and click Enable Security Scanning in the Security Scanner section of the UI.
7. Set the HTTP endpoint for Clair using a port that is not already in use on the quay-server system, for example, 8081.

8. Create a pre-shared key (PSK) using the Generate PSK button.

   Security Scanner UI

   

9. Validate and download the config.yaml file for Red Hat Quay, and then stop the Quay container that is running the configuration editor.

10. Extract the new configuration bundle into your Red Hat Quay installation directory, for example:

    $ tar xvf quay-config.tar.gz -d /home/<user-name>/quay-poc/

11. Create a folder for your Clair configuration file, for example:

    $ mkdir /etc/opt/clairv4/config/

12. Change into the Clair configuration folder:

    $ cd /etc/opt/clairv4/config/

13. Create a Clair configuration file, for example:

    http_listen_addr: :8081
    introspection_addr: :8088
    log_level: debug
    indexer:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      scanlock_retry: 10
      layer_scan_concurrency: 5
      migrations: true
    matcher:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      max_conn_pool: 100
      migrations: true
      indexer_addr: clair-indexer
    notifier:
      connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
      delivery_interval: 1m
      poll_interval: 5m
      migrations: true
    auth:
      psk:
        key: "MTU5YzA4Y2ZkNzJoMQ=="
        iss: ["quay"]
    # tracing and metrics
    trace:
      name: "jaeger"
      probability: 1
      jaeger:
        agent:
          endpoint: "localhost:6831"
        service_name: "clair"
    metrics:
      name: "prometheus"

    For more information about Clair’s configuration format, see Clair configuration reference.

14. Start Clair by using the container image, mounting in the configuration from the file you created:

    $ sudo podman run -d --name clairv4 \
    -p 8081:8081 -p 8088:8088 \
    -e CLAIR_CONF=/clair/config.yaml \
    -e CLAIR_MODE=combo \
    -v /etc/opt/clairv4/config:/clair:Z \
    registry.redhat.io/quay/clair-rhel8:v3.9.7

    Note

    Running multiple Clair containers is also possible, but for deployment scenarios beyond a single container the use of a container orchestrator like Kubernetes or OpenShift Container Platform is strongly recommended.

Procedure

1. Pull a sample image by entering the following command:

   $ podman pull ubuntu:20.04

2. Tag the image to your registry by entering the following command:

   $ sudo podman tag docker.io/library/ubuntu:20.04 <quay-server.example.com>/<user-name>/ubuntu:20.04

3. Push the image to your Red Hat Quay registry by entering the following command:

   $ sudo podman push --tls-verify=false quay-server.example.com/quayadmin/ubuntu:20.04

4. Log in to your Red Hat Quay deployment through the UI.
5. Click the repository name, for example, quayadmin/ubuntu.

6. In the navigation pane, click Tags.

   Report summary

   

7. Click the image report, for example, 45 medium, to show a more detailed report:

   Report details

   

   Note

   In some cases, Clair shows duplicate reports on images, for example, ubi8/nodejs-12 or ubi8/nodejs-16. This occurs because vulnerabilities with same name are for different packages. This behavior is expected with Clair vulnerability reporting and will not be addressed as a bug.

1. Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.

   

2. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file.

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to IPv6, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: IPv6
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to IPv6 by entering the following command:

   $ curl <quay_endpoint>/health/instance
   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_LISTEN_IP_VERSION parameter and set it to dual-stack, for example:

   ---
   FEATURE_GOOGLE_LOGIN: false
   FEATURE_INVITE_ONLY_USER_CREATION: false
   FEATURE_LISTEN_IP_VERSION: dual-stack
   FEATURE_MAILING: false
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: false
   ---

2. Start, or restart, your Red Hat Quay deployment.

3. Check that your deployment is listening to both channels by entering the following command:

   1. For IPv4, enter the following command:

      $ curl --ipv4 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   2. For IPv6, enter the following command:

      $ curl --ipv6 <quay_endpoint>
      {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

Procedure

1. You can use the Red Hat Quay config tool to configure LDAP.

   1. Using the Red Hat Quay config tool, locate the Authentication section. Select LDAP from the dropdown menu, and update the LDAP configuration fields as required.

      

   2. Optional. On the Team synchronization box, and click Enable Team Syncrhonization Support. With team synchronization enabled, Red Hat Quay administrators who are also superusers can set teams to have their membership synchronized with a backing group in LDAP.

      

   3. For Resynchronization duration enter 60m. This option sets the resynchronization duration at which a team must be re-synchronized. This field must be set similar to the following examples: 30m, 1h, 1d.

   4. Optional. For Self-service team syncing setup, you can click Allow non-superusers to enable and manage team syncing to allow superusers the ability to enable and manage team syncing under the organizations that they are administrators for.

      

   5. Locate the LDAP URI box and provide a full LDAP URI, including the ldap:// or ldaps:// prefix, for example, ldap://117.17.8.101.

      

   6. Under Base DN, provide a name which forms the base path for looking up all LDAP records, for example, o=<organization_id>,dc=<example_domain_component>,dc=com.

      

   7. Under User Relative DN, provide a list of Distinguished Name path(s), which form the secondary base path(s) for looking up all user LDAP records relative to the Base DN defined above. For example, uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com. This path, or these paths, is tried if the user is not found through the primary relative DN.

      

      Note

      User Relative DN is relative to Base DN, for example, ou=Users and not ou=Users,dc=<example_domain_component>,dc=com.

   8. Optional. Provide Secondary User Relative DNs if there are multiple Organizational Units where user objects are located. You can type in the Organizational Units and click Add to add multiple RDNs. For example, ou=Users,ou=NYC and ou=Users,ou=SFO.

      The User Relative DN searches with subtree scope. For example, if your organization has Organization Units NYC and SFO under the Users OU (that is, ou=SFO,ou=Users and ou=NYC,ou=Users), Red Hat Quay can authenticate users from both the NYC and SFO Organizational Units if the User Relative DN is set to Users (ou=Users).

   9. Optional. Fill in the Additional User Filter Expression field for all user lookup queries if desired. Distinguished Names used in the filter must be full based. The Base DN is not added automatically added to this field, and you must wrap the text in parentheses, for example, (memberOf=cn=developers,ou=groups,dc=<example_domain_component>,dc=com).

      

   10. Fill in the Administrator DN field for the Red Hat Quay administrator account. This account must be able to login and view the records for all users accounts. For example: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com.

       

   11. Fill in the Administrator DN Password field. This is the password for the administrator distinguished name.

       Important

       The password for this field is stored in plaintext inside of the config.yaml file. Setting up a dedicated account of using a password hash is highly recommended.

   12. Optional. Fill in the UID Attribute field. This is the name of the property field in the LDAP user records that stores your user’s username. Most commonly, uid is entered for this field. This field can be used to log into your Red Hat Quay deployment.

       

   13. Optional. Fill in the Mail Attribute field. This is the name of the property field in your LDAP user records that stores your user’s e-mail addresses. Most commonly, mail is entered for this field. This field can be used to log into your Red Hat Quay deployment.

       

       Note
       • The username to log in must exist in the User Relative DN.
       • If you are using Microsoft Active Directory to setup your LDAP deployment, you must use sAMAccountName for your UID attribute.

   14. Optional. You can add a custom SSL/TLS certificate by clicking Choose File under the Custom TLS Certificate optionl. Additionally, you can enable fallbacks to insecure, non-TLS connections by checking the Allow fallback to non-TLS connections box.

       

       If you upload an SSl/TLS certificate, you must provide an ldaps:// prefix, for example, LDAP_URI: ldaps://ldap_provider.example.org.

2. Alternatively, you can update your config.yaml file directly to include all relevant information. For example:

   ---
   AUTHENTICATION_TYPE: LDAP
   ---
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,dc=<domain_name>,dc=com)
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com

3. After you have added all required LDAP fields, click the Save Configuration Changes button to validate the configuration. All validation must succeed before proceeding. Additional configuration can be performed by selecting the Continue Editing button.

Procedure

1. In your deployment’s config.yaml file, add the LDAP_RESTRICTED_USER_FILTER parameter and specify the group of restricted users, for example, members:

   ---
   AUTHENTICATION_TYPE: LDAP
   ---
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,o=<example_organization_unit>,dc=<example_domain_component>,dc=com)
   LDAP_RESTRICTED_USER_FILTER: (<filterField>=<value>)
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com

2. Start, or restart, your Red Hat Quay deployment.

Procedure

1. In your deployment’s config.yaml file, add the LDAP_SUPERUSER_FILTER parameter and add the group of users you want configured as super users, for example, root:

   ---
   AUTHENTICATION_TYPE: LDAP
   ---
   LDAP_ADMIN_DN: uid=<name>,ou=Users,o=<organization_id>,dc=<example_domain_component>,dc=com
   LDAP_ADMIN_PASSWD: ABC123
   LDAP_ALLOW_INSECURE_FALLBACK: false
   LDAP_BASE_DN:
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com
   LDAP_EMAIL_ATTR: mail
   LDAP_UID_ATTR: uid
   LDAP_URI: ldap://<example_url>.com
   LDAP_USER_FILTER: (memberof=cn=developers,ou=Users,o=<example_organization_unit>,dc=<example_domain_component>,dc=com)
   LDAP_SUPERUSER_FILTER: (<filterField>=<value>)
   LDAP_USER_RDN:
       - ou=<example_organization_unit>
       - o=<organization_id>
       - dc=<example_domain_component>
       - dc=com

2. Start, or restart, your Red Hat Quay deployment.

Procedure

1. Pull a sample image, for example, ubuntu:18.04, by entering the following command:

   $ podman pull ubuntu:18.04

2. Tag the same image twice by entering the following command:

   $ podman tag docker.io/library/ubuntu:18.04 quay-server.example.com/quota-test/ubuntu:tag1

   $ podman tag docker.io/library/ubuntu:18.04 quay-server.example.com/quota-test/ubuntu:tag2

3. Push the sample image to your organization by entering the following commands:

   $ podman push --tls-verify=false quay-server.example.com/quota-test/ubuntu:tag1

   $ podman push --tls-verify=false quay-server.example.com/quota-test/ubuntu:tag2

4. On the Red Hat Quay UI, navigate to Organization and click the Repository Name, for example, quota-test/ubuntu. Then, click Tags. There should be two repository tags, tag1 and tag2, each with the same manifest. For example:

   

   However, by clicking on the Organization link, we can see that the Total Quota Consumed is 27.94 MB, meaning that the Ubuntu image has only been accounted for once:

   

   If you delete one of the Ubuntu tags, the Total Quota Consumed remains the same.

   Note

   If you have configured the Red Hat Quay time machine to be longer than 0 seconds, subtraction will not happen until those tags pass the time machine window. If you want to expedite permanent deletion, see Permanently deleting an image tag in Red Hat Quay 3.9.

Procedure

1. Create a new organization or choose an existing one. Initially, no quota is configured, as can be seen on the Organization Settings tab:

   

2. Log in to the registry as a superuser and navigate to the Manage Organizations tab on the Super User Admin Panel. Click the Options icon of the organization for which you want to create storage quota limits:

   

3. Click Configure Quota and enter the initial quota, for example, 10 MB. Then click Apply and Close:

   

4. Check that the quota consumed shows 0 of 10 MB on the Manage Organizations tab of the superuser panel:

   

   The consumed quota information is also available directly on the Organization page:

   Initial consumed quota

   

5. To increase the quota to 100MB, navigate to the Manage Organizations tab on the superuser panel. Click the Options icon and select Configure Quota, setting the quota to 100 MB. Click Apply and then Close:

   

6. Pull a sample image by entering the following command:

   $ podman pull ubuntu:18.04

7. Tag the sample image by entering the following command:

   $ podman tag docker.io/library/ubuntu:18.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

8. Push the sample image to the organization by entering the following command:

   $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:18.04

9. On the superuser panel, the quota consumed per organization is displayed:

   

10. The Organization page shows the total proportion of the quota used by the image:

    Total Quota Consumed for first image

    

11. Pull a second sample image by entering the following command:

    $ podman pull nginx

12. Tag the second image by entering the following command:

    $ podman tag docker.io/library/nginx example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

13. Push the second image to the organization by entering the following command:

    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/nginx

14. The Organization page shows the total proportion of the quota used by each repository in that organization:

    Total Quota Consumed for each repository

    

15. Create reject and warning limits:

    From the superuser panel, navigate to the Manage Organizations tab. Click the Options icon for the organization and select Configure Quota. In the Quota Policy section, with the Action type set to Reject, set the Quota Threshold to 80 and click Add Limit:

    

16. To create a warning limit, select Warning as the Action type, set the Quota Threshold to 70 and click Add Limit:

    

17. Click Close on the quota popup. The limits are viewable, but not editable, on the Settings tab of the Organization page:

    

18. Push an image where the reject limit is exceeded:

    Because the reject limit (80%) has been set to below the current repository size (~83%), the next pushed image is rejected automatically.

    Sample image push

    $ podman pull ubuntu:20.04
    
    $ podman tag docker.io/library/ubuntu:20.04 example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04
    
    $ podman push --tls-verify=false example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org/testorg/ubuntu:20.04

    Sample output when quota exceeded

    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0002] failed, retrying in 1s ... (1/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0005] failed, retrying in 1s ... (2/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    WARN[0009] failed, retrying in 1s ... (3/3). Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace
    Getting image source signatures
    Copying blob d4dfaa212623 [--------------------------------------] 8.0b / 3.5KiB
    Copying blob cba97cc5811c [--------------------------------------] 8.0b / 15.0KiB
    Copying blob 0c78fac124da [--------------------------------------] 8.0b / 71.8MiB
    Error: Error writing blob: Error initiating layer upload to /v2/testorg/ubuntu/blobs/uploads/ in example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org: denied: Quota has been exceeded on namespace

19. When limits are exceeded, notifications are displayed in the UI:

    Quota notifications

    

Procedure

1. On the Red Hat Quay UI, click your username → Super User Admin Panel.
2. In the navigation pane, click Manage Organizations.
3. Click Calculate, next to Total Registry Size: 0.00 KB, Updated: Never , Calculation required. Then, click Ok.

4. After a few minutes, depending on the size of your registry, refresh the page. Now, the Total Registry Size should be calculated. For example:

   

Procedure

1. Create a temporary backup directory, for example, quay-backup:

   $ mkdir /tmp/quay-backup

2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install:

   $ podman run --name quay-app \
      -v /opt/quay-install/config:/conf/stack:Z \
      -v /opt/quay-install/storage:/datastorage:Z \
      registry.redhat.io/quay/quay-rhel8:v3.9.7

   Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command:

   $ cd /opt/quay-install

3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command:

   $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

4. Back up the Quay container service by entering the following command:

   $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -
   
     /usr/bin/podman run --name quay-app \
     -v /opt/quay-install/config:/conf/stack:Z \
     -v /opt/quay-install/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.9.7

5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command:

   $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command:

   $ grep DB_URI /tmp/quay-backup/quay-config.yaml

   Example output:

   $ postgresql://<username>:test123@172.24.10.50/quay

7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command:

   $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command:

   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - S3Storage
       - s3_bucket: <bucket_name>
         storage_path: /registry
         s3_access_key: <s3_access_key>
         s3_secret_key: <s3_secret_key>
         host: <host_name>

9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7:

   $ export AWS_ACCESS_KEY_ID=<access_key>

10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG:

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
    download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

Procedure

1. Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container:

   $ mkdir /opt/new-quay-install

2. Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1:

   $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

3. Change into the new-quay-install directory by entering the following command:

   $ cd /opt/new-quay-install/

4. Extract the contents of your Red Hat Quay directory:

   $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

5. Recall the DB_URI from your backed-up config.yaml file by entering the following command:

   $ grep DB_URI config.yaml

   Example output:

   postgresql://<username>:test123@172.24.10.50/quay

6. Run the following command to enter the PostgreSQL database server:

   $ sudo postgres

7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command:

   $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
   postgres=> CREATE DATABASE example_restore_registry_quay_database;

   Example output:

   CREATE DATABASE

8. Connect to the database by running the following command:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

9. Create a pg_trmg extension of your Quay database by running the following command:

   example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Example output:

   CREATE EXTENSION

10. Exit the postgres CLI by entering the following command:

    \q

11. Import the database backup to your new database by running the following command:

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle:

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13:

    $ export AWS_ACCESS_KEY_ID=<access_key>

14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

15. Create a new s3 bucket by entering the following command:

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

16. Upload all blobs to the new s3 bucket by entering the following command:

    $ aws s3 sync --no-verify-ssl \
    --endpoint-url <example_endpoint_url> 1
    /tmp/quay-backup/blob-backup/. s3://quay/

    1

    The Red Hat Quay registry endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
    ...

17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <new_bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

Procedure

1. Back up the config.yaml of your standalone Red Hat Quay deployment:

   $ mkdir /tmp/quay-backup
   $ cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup

2. Create a backup of the database that your standalone Red Hat Quay deployment is using:

   $ pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql

3. Install the AWS CLI if you do not have it already.

4. Create an ~/.aws/ directory:

   $ mkdir ~/.aws/

5. Obtain the access_key and secret_key from the config.yaml of your standalone deployment:

   $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml

   Example output:

   DISTRIBUTED_STORAGE_CONFIG:
       minio-1:
           - RadosGWStorage
           - access_key: ##########
             bucket_name: quay
             hostname: 172.24.10.50
             is_secure: false
             port: "9000"
             secret_key: ##########
             storage_path: /datastorage/registry

6. Store the access_key and secret_key from the config.yaml file in your ~/.aws directory:

   $ touch ~/.aws/credentials

7. Optional: Check that your access_key and secret_key are stored:

   $ cat > ~/.aws/credentials << EOF
   [default]
   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
   EOF

   Example output:

   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG

   Note

   If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can, you can configure these by running aws configure and manually inputting the credentials.

8. In your quay-backup directory, create a bucket_backup directory:

   $ mkdir /tmp/quay-backup/bucket-backup

9. Backup all blobs from the S3 storage:

   $ aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/

   Note

   The PUBLIC_S3_ENDPOINT can be read from the Red Hat Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG. If the endpoint is insecure, use http instead of https in the endpoint URL.

1. Scale down the Red Hat Quay Operator:

   $ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators

2. Scale down the application and mirror deployments:

   $ oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT

3. Copy the database SQL backup to the Quay PostgreSQL database instance:

   $ oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata

4. Obtain the database password from the Operator-created config.yaml file:

   $ oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret

   Example output:

         "name": "QUAY_CONFIG_SECRET_NAME"

   $ oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

   cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI

   Example output:

   postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME

5. Execute a shell inside of the database pod:

   # oc exec -it quay-postgresql-database-pod -- /bin/bash

6. Enter psql:

   bash-4.4$ psql

7. Drop the database:

   postgres=# DROP DATABASE "example-restore-registry-quay-database";

   Example output:

   DROP DATABASE

8. Create a new database and set the owner as the same name:

   postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database";

   Example output:

   CREATE DATABASE

9. Connect to the database:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

10. Create a pg_trmg extension of your Quay database:

    example-restore-registry-quay-database=# create extension pg_trgm ;

    Example output:

    CREATE EXTENSION

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Set the password for your PostgreSQL deployment:

    bash-4.4$ psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

13. Exit bash mode:

    bash-4.4$ exit

14. Create a new configuration bundle for the Red Hat Quay Operator.

    $ touch config-bundle.yaml

15. In your new config-bundle.yaml, include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml:

    $ cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml

    Note

    You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file.

16. Create a configuration bundle secret inside of your OpenShift cluster:

    $ oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml

17. Scale up the Quay pods:

    $ oc scale --replicas=1 deployment quayregistry-quay-app
    deployment.apps/quayregistry-quay-app scaled

18. Scale up the mirror pods:

    $ oc scale --replicas=1  deployment quayregistry-quay-mirror
    deployment.apps/quayregistry-quay-mirror scaled

19. Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle:

    $ oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}'

    Note

    If Red Hat Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default.

20. Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file:

    $ touch credentials.yaml

    $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

    $ cat > ~/.aws/credentials << EOF
    [default]
    aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
    aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
    EOF

    Note

    If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can configure these by running aws configure and manually inputting the credentials.

21. Record the NooBaa’s publicly available endpoint:

    $ oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}"

22. Sync the backup data to the NooBaa backend storage:

    $ aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME

23. Scale the Operator back up to 1 pod:

    $ oc scale –replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators

Procedure

1. Add the IGNORE_UNKNOWN_MEDIATYPES configuration field to your Red Hat Quay config.yaml file:

   IGNORE_UNKNOWN_MEDIATYPES: true

   With this field enabled, your Red Hat Quay deployment accepts unknown and unrecognized artifact types.

Procedure

1. Enter the following command to ensure that garbage collection is properly working:

   $ sudo podman logs <container_id>

   Example output:

   gcworker stdout | 2022-11-14 18:46:52,458 [63] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2022-11-14 18:47:22 UTC)" executed successfully

2. Delete an image tag.

3. Enter the following command to ensure that the tag was deleted:

   $ podman logs quay-app

   Example output:

   gunicorn-web stdout | 2022-11-14 19:23:44,574 [233] [INFO] [gunicorn.access] 192.168.0.38 - - [14/Nov/2022:19:23:44 +0000] "DELETE /api/v1/repository/quayadmin/busybox/tag/test HTTP/1.0" 204 0 "http://quay-server.example.com/repository/quayadmin/busybox?tab=tags" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"

Procedure

1. In your deployment’s config.yaml file, add the FEATURE_UI_V2 parameter and set it to true, for example:

   ---
   FEATURE_TEAM_SYNCING: false
   FEATURE_UI_V2: true
   FEATURE_USER_CREATION: true
   ---

2. Log in to your Red Hat Quay deployment.

3. In the navigation pane of your Red Hat Quay deployment, you are given the option to toggle between Current UI and New UI. Click the toggle button to set it to new UI, and then click Use Beta Environment, for example:

   

1. In the navigation pane of your Red Hat Quay deployment, you are given the option to toggle between Current UI and New UI. Click the toggle button to set it to Current UI.

   

Procedure

1. On your web browser, navigate to https://{quay-ip-endpoint}/health/instance.

2. You are taken to the health instance page, which returns information like the following:

   {"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}

   For Red Hat Quay, "status_code": 200 means that the instance is health. Conversely, if you receive "status_code": 503, there is an issue with your deployment.