Troubleshooting Red Hat Quay

Procedure

1. Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command:

   $ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'

   Copy to Clipboard Toggle word wrap

   Example output

   $2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm

   Copy to Clipboard Toggle word wrap

2. Enter the following command to show the container ID of your Red Hat Quay container registry:

   $ sudo podman ps -a

   Copy to Clipboard Toggle word wrap

   Example output

   CONTAINER ID  IMAGE                                       COMMAND         CREATED         STATUS             PORTS                   NAMES
   70560beda7aa  registry.redhat.io/rhel8/redis-5:1         run-redis       2 hours ago     Up 2 hours ago     0.0.0.0:6379->6379/tcp                       redis
   8012f4491d10  registry.redhat.io/quay/quay-rhel8:v3.8.2  registry        3 minutes ago   Up 8 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay
   8b35b493ac05  registry.redhat.io/rhel8/postgresql-10:1   run-postgresql  39 seconds ago  Up 39 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay

   Copy to Clipboard Toggle word wrap

3. Execute an interactive shell for the postgresql container image by entering the following command:

   $ sudo podman exec -it 8b35b493ac05 /bin/bash

   Copy to Clipboard Toggle word wrap

4. Re-enter the quay PostgreSQL database server, specifying the database, username, and host address:

   bash-4.4$ psql -d quay -U quayuser -h 192.168.1.28 -W

   Copy to Clipboard Toggle word wrap

5. Update the password_hash of the superuser admin who lost their password:

   quay=> UPDATE public.user SET password_hash = '$2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm' where username = 'quayadmin';

   Copy to Clipboard Toggle word wrap

   Example output

   UPDATE 1

   Copy to Clipboard Toggle word wrap

6. Enter the following to command to ensure that the password_hash has been updated:

   quay=> select * from public.user;

   Copy to Clipboard Toggle word wrap

   Example output

   id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
   ----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
   -------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
   ---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
   1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492

   Copy to Clipboard Toggle word wrap

7. Log in to your Red Hat Quay deployment using the new password:

   $ sudo podman login -u quayadmin -p newpass1234 http://quay-server.example.com --tls-verify=false

   Copy to Clipboard Toggle word wrap

   Example output

   Login Succeeded!

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your Red Hat Quay deployment.
2. On the OpenShift Container Platform UI, navigate to Workloads → Secrets.
3. Select the namespace for your Red Hat Quay deployment, for example, Project quay.
4. Locate and store the PostgreSQL database credentials.

5. Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command:

   $ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'

   Copy to Clipboard Toggle word wrap

   Example output

   $2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y

   Copy to Clipboard Toggle word wrap

6. On the CLI, log in to the database, for example:

   $ oc rsh quayuser-quay-quay-database-669c8998f-v9qsl

   Copy to Clipboard Toggle word wrap

7. Enter the following command to open a connection to the quay PostgreSQL database server, specifying the database, username, and host address:

   sh-4.4$ psql -U quayuser-quay-quay-database -d quayuser-quay-quay-database -W

   Copy to Clipboard Toggle word wrap

8. Enter the following command to connect to the default database for the current user:

   quay=> \c

   Copy to Clipboard Toggle word wrap

9. Update the password_hash of the superuser admin who lost their password:

   quay=> UPDATE public.user SET password_hash = '$2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y' where username = 'quayadmin';

   Copy to Clipboard Toggle word wrap

10. Enter the following to command to ensure that the password_hash has been updated:

    quay=> select * from public.user;

    Copy to Clipboard Toggle word wrap

    Example output

    id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
    ----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
    -------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
    ---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
    1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492

    Copy to Clipboard Toggle word wrap

11. Navigate to your Red Hat Quay UI on OpenShift Container Platform and log in using the new credentials.

Procedure

1. Exec into the Red Hat Quay pod or container. For more information, see "Interacting with the Red Hat Quay database".

2. Enter the following command to show all users for external authentication:

   quay=# select * from federatedlogin;

   Copy to Clipboard Toggle word wrap

   Example output

   id | user_id | service_id |                service_ident                |               metadata_json
   ----+---------+------------+---------------------------------------------+-------------------------------------------
   1 |       1 |          3 | testuser0                                   | {}
   2 |       1 |          8 | PK7Zpg2Yu2AnfUKG15hKNXqOXirqUog6G-oE7OgzSWc | {"service_username": "live.com#testuser0"}
   3 |       2 |          3 | testuser1                                    | {}
   4 |       2 |          4 | 110875797246250333431                       | {"service_username": "testuser1"}
   5 |       3 |          3 | testuser2                                    | {}
   6 |       3 |          1 | 26310880                                    | {"service_username": "testuser2"}
   (6 rows)

   Copy to Clipboard Toggle word wrap

3. Verify that the users are inserted into the user table:

   quay=# select username, email from "user";

   Copy to Clipboard Toggle word wrap

   Example output

   username  |        email
   -----------+----------------------
   testuser0 | testuser0@outlook.com
   testuser1  | testuser1@gmail.com
   testuser2  | testuser2@redhat.com
   (3 rows)

   Copy to Clipboard Toggle word wrap

Procedure

1. Exec into the Red Hat Quay database.

   1. Enter the following commands to exec into the Red Hat Quay database pod on OpenShift Container Platform:

      $ oc exec -it <quay_database_pod> -- psql

      Copy to Clipboard Toggle word wrap

   2. Enter the following command to exec into the Red Hat Quay database on a standalone deployment:

      $ sudo podman exec -it <quay_container_name> /bin/bash

      Copy to Clipboard Toggle word wrap

2. Enter the PostgreSQL shell.

   Warning

   Interacting with the PostgreSQL database is potentially destructive. It is highly recommended that you perform the following procedure with the help of a Red Hat Quay Support Specialist.

   1. If you are using the Red Hat Quay Operator, enter the following command to enter the PostgreSQL shell:

      $ oc rsh <quay_pod_name> psql -U your_username -d your_database_name

      Copy to Clipboard Toggle word wrap

   2. If you are on a standalone Red Hat Quay deployment, enter the following command to enter the PostgreSQL shell:

      bash-4.4$ psql -U your_username -d your_database_name

      Copy to Clipboard Toggle word wrap

Procedure

1. If your container or pod is in a crashloopbackoff state, you can enter the following commands.

   1. Enter the following command to scale down the Red Hat Quay Operator:

      $ oc scale deployment/quay-operator.v3.8.z  --replicas=0

      Copy to Clipboard Toggle word wrap

      Example output

      deployment.apps/quay-operator.v3.8.z scaled

      Copy to Clipboard Toggle word wrap

   2. Enter the following command to scale down the Red Hat Quay database:

      $ oc scale deployment/<quay_database> --replicas=0

      Copy to Clipboard Toggle word wrap

      Example output

      deployment.apps/<quay_database> scaled

      Copy to Clipboard Toggle word wrap

   3. Enter the following command to edit the Red Hat Quay database:

      Warning

      Interacting with the PostgreSQL database is potentially destructive. It is highly recommended that you perform the following procedure with the help of a Red Hat Quay Support Specialist.

      $ oc edit deployment <quay_database>

      Copy to Clipboard Toggle word wrap

      ...
       template:
         metadata:
           creationTimestamp: null
           labels:
             quay-component: <quay_database>
             quay-operator/quayregistry: quay-operator.v3.8.z
         spec:
           containers:
           - env:
             - name: POSTGRESQL_USER
               value: postgres
             - name: POSTGRESQL_DATABASE
               value: postgres
             - name: POSTGRESQL_PASSWORD
               value: postgres
             - name: POSTGRESQL_ADMIN_PASSWORD
               value: postgres
             - name: POSTGRESQL_MAX_CONNECTIONS
               value: "1000"
             image: registry.redhat.io/rhel8/postgresql-10@sha256:a52ad402458ec8ef3f275972c6ebed05ad64398f884404b9bb8e3010c5c95291
             imagePullPolicy: IfNotPresent
             name: postgres
             command: ["/bin/bash", "-c", "sleep 86400"] 

      1

      
      ...

      Copy to Clipboard Toggle word wrap

      • Add the command: ["/bin/bash", "-c", "sleep 86400"] line in the same indentation.

        Example output

        deployment.apps/<quay_database> edited

        Copy to Clipboard Toggle word wrap

   4. Execute the following command inside of your <quay_database>:

      $ oc exec -it <quay_database> -- cat /var/lib/pgsql/data/userdata/postgresql/logs/* /path/to/desired_directory_on_host

      Copy to Clipboard Toggle word wrap

Procedure

1. Check the connectivity between Red Hat Quay and the database pod.

   1. If you are using the Red Hat Quay Operator on OpenShift Container Platform, enter the following command:

      $ oc exec -it _quay_pod_name_ -- curl -v telnet://<database_pod_name>:5432

      Copy to Clipboard Toggle word wrap

   2. If you are using a standalone deployment of Red Hat Quay, enter the following command:

      $ podman exec -it <quay_container_name >curl -v telnet://<database_container_name>:5432

      Copy to Clipboard Toggle word wrap

Procedure

1. Obtain a list of running containers.

2. Monitor disk usage of your Red Hat Quay deployment.

   1. If you are using the Red Hat Quay Operator on OpenShift Container Platform, enter the following command:

      $ oc exec -it <quay_database_pod_name> -- df -ah

      Copy to Clipboard Toggle word wrap

   2. If you are using a standalone deployment of Red Hat Quay, enter the following command:

      $ podman exec -it <quay_database_conatiner_name> df -ah

      Copy to Clipboard Toggle word wrap

3. Monitor other resource usage.

   1. Enter the following command to check resource allocation on a Red Hat Quay Operator deployment:

      $ oc adm top pods

      Copy to Clipboard Toggle word wrap

   2. Enter the following command to check the status of a specific pod on a standalone deployment of Red Hat Quay:

      $ podman pod stats <pod_name>

      Copy to Clipboard Toggle word wrap

   3. Enter the following command to check the status of a specific container on a standalone deployment of Red Hat Quay:

      $ podman stats <container_name>

      Copy to Clipboard Toggle word wrap

      The following information is returned:

      • CPU %. The percentage of CPU usage by the container since the last measurement. This value represents the container’s share of the available CPU resources.
      • MEM USAGE / LIMIT. The current memory usage of the container followed by its memory limit. The values are displayed in the format current_usage / memory_limit. For example, 300.4MiB / 7.795GiB indicates that the container is currently using 300.4 megabytes of memory out of a limit of 7.795 gigabytes.
      • MEM %. The percentage of memory usage by the container in relation to its memory limit.
      • NET I/O. The network I/O (input/output) statistics of the container. It displays the amount of data transmitted and received by the container over the network. The values are displayed in the format: transmitted_bytes / received_bytes.
      • BLOCK I/O. The block I/O (input/output) statistics of the container. It represents the amount of data read from and written to the block devices (for example, disks) used by the container. The values are displayed in the format read_bytes / written_bytes.

Procedure

1. Enter the following command to ensure that your data is replicated in all backend buckets:

   $ aws --profile quay_prod_s3 --endpoint=http://10.0.x.x:port s3 ls ocp-quay --recursive --human-readable --summarize

   Copy to Clipboard Toggle word wrap

   Example output

   Total Objects: 17996
   Total Size: 514.4 GiB

   Copy to Clipboard Toggle word wrap