Red Hat Summit Red Hat SummitSuporteConsoleDesenvolvedoresComeçar um testeContato

Este conteúdo não está disponível no idioma selecionado.

Chapter 6. Troubleshooting Red Hat Quay components

To troubleshoot specific {product-title} components and resolve component-related issues, you can use the procedures in this document. You can also run in debug mode, obtain logs and config, perform health checks, then search the Red Hat Knowledgebase or file a support ticket.

By using the following procedures, you are able to troubleshoot common component issues. Afterwards, you can search for solutions on the Red Hat Knowledgebase or file a support ticket with the Red Hat Support team.

6.1. Troubleshooting the Red Hat Quay database
Copiar o link

To troubleshoot the {product-title} database and resolve connectivity, configuration, or resource issues, you can check deployment type, pod or container status, logs, connectivity, and configuration. You can also examine resource allocation and interact with the PostgreSQL database.

  • Image Metadata. The database stores metadata associated with container images, such as image names, versions, creation timestamps, and the user or organization that owns the image. This information allows for easy identification and organization of container images within the registry.
  • Image Tags. Red Hat Quay allows users to assign tags to container images, enabling convenient labeling and versioning. The PostgreSQL database maintains the mapping between image tags and their corresponding image manifests, allowing users to retrieve specific versions of container images based on the provided tags.
  • Image Layers. Container images are composed of multiple layers, which are stored as individual objects. The database records information about these layers, including their order, checksums, and sizes. This data is crucial for efficient storage and retrieval of container images.
  • User and Organization Data. Red Hat Quay supports user and organization management, allowing users to authenticate and manage access to container images. The PostgreSQL database stores user and organization information, including usernames, email addresses, authentication tokens, and access permissions.
  • Repository Information. Red Hat Quay organizes container images into repositories, which act as logical units for grouping related images. The database maintains repository data, including names, descriptions, visibility settings, and access control information, enabling users to manage and share their repositories effectively.
  • Event Logs. Red Hat Quay tracks various events and activities related to image management and repository operations. These event logs, including image pushes, pulls, deletions, and repository modifications, are stored in the PostgreSQL database, providing an audit trail and allowing administrators to monitor and analyze system activities.

The content in this section covers the following procedures:

  • Checking the type of deployment: Determine if the database is deployed as a container on a virtual machine or as a pod on OpenShift Container Platform.
  • Checking the container or pod status: Verify the status of the database pod or container using specific commands based on the deployment type.
  • Examining the database container or pod logs: Access and examine the logs of the database pod or container, including commands for different deployment types.
  • Checking the connectivity between Red Hat Quay and the database pod: Check the connectivity between Red Hat Quay and the database pod using relevant commands.
  • Checking the database configuration: Check the database configuration at various levels (OpenShift Container Platform or PostgreSQL level) based on the deployment type.
  • Checking resource allocation: Monitor resource allocation for the Red Hat Quay deployment, including disk usage and other resource usage.
  • Interacting with the Red Hat Quay database: Learn how to interact with the PostgreSQL database, including commands to access and query databases.

6.1.1. Troubleshooting Red Hat Quay database issues
Copiar o link

To troubleshoot Red Hat Quay database issues and resolve common errors like database connection refused or authentication failed, you can check the database logs, verify credentials, and ensure the database is running.

6.1.2. Resetting superuser passwords on Red Hat Quay standalone deployments
Copiar o link

To reset the superuser password on your Red Hat Quay standalone deployment, you can generate a bcrypt-hashed password with Python, exec into the PostgreSQL container with podman exec, and update the user table with the new hashed password.

Prerequisites

  • You have created a Red Hat Quay superuser.
  • You have installed Python 3.9.
  • You have installed the pip package manager for Python.
  • You have installed the bcrypt package for pip.

Procedure

  1. Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command: 

    $ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'
    Copy to Clipboard Toggle word wrap

    Example output

    $2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm
    Copy to Clipboard Toggle word wrap

  2. Enter the following command to show the container ID of your Red Hat Quay container registry: 

    $ sudo podman ps -a
    Copy to Clipboard Toggle word wrap

    Example output

    CONTAINER ID  IMAGE                                       COMMAND         CREATED         STATUS             PORTS                   NAMES
70560beda7aa  registry.redhat.io/rhel8/redis-5:1         run-redis       2 hours ago     Up 2 hours ago     0.0.0.0:6379->6379/tcp                       redis
8012f4491d10  registry.redhat.io/quay/quay-rhel8:v3.8.2  registry        3 minutes ago   Up 8 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay
8b35b493ac05  registry.redhat.io/rhel8/postgresql-10:1   run-postgresql  39 seconds ago  Up 39 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay
    Copy to Clipboard Toggle word wrap

  3. Execute an interactive shell for the postgresql container image by entering the following command: 

    $ sudo podman exec -it 8b35b493ac05 /bin/bash
    Copy to Clipboard Toggle word wrap

  4. Re-enter the quay PostgreSQL database server, specifying the database, username, and host address: 

    bash-4.4$ psql -d quay -U quayuser -h 192.168.1.28 -W
    Copy to Clipboard Toggle word wrap

  5. Update the password_hash of the superuser admin who lost their password: 

    quay=> UPDATE public.user SET password_hash = '$2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm' where username = 'quayadmin';
    Copy to Clipboard Toggle word wrap

    Example output

    UPDATE 1
    Copy to Clipboard Toggle word wrap

  6. Enter the following to command to ensure that the password_hash has been updated: 

    quay=> select * from public.user;
    Copy to Clipboard Toggle word wrap

    Example output

    id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
-------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$T8pkgtOoys3G5ut7FV1She6vXlYgU.6TeoGmbbAVQtN8X8ch4knKm | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492
    Copy to Clipboard Toggle word wrap

  7. Log in to your Red Hat Quay deployment using the new password: 

    $ sudo podman login -u quayadmin -p newpass1234 http://quay-server.example.com --tls-verify=false
    Copy to Clipboard Toggle word wrap

    Example output

    Login Succeeded!
    Copy to Clipboard Toggle word wrap

6.1.3. Resetting superuser passwords on the Red Hat Quay Operator
Copiar o link

To reset the superuser password on your Red Hat Quay Operator deployment, you can generate a bcrypt-hashed password with Python, log in to the PostgreSQL database with oc rsh and psql, and update the user table with the new hashed password.

Prerequisites

  • You have created a Red Hat Quay superuser.
  • You have installed Python 3.9.
  • You have installed the pip package manager for Python.
  • You have installed the bcrypt package for pip.

Procedure

  1. Log in to your Red Hat Quay deployment.
  2. On the OpenShift Container Platform UI, navigate to Workloads Secrets.
  3. Select the namespace for your Red Hat Quay deployment, for example, Project quay.
  4. Locate and store the PostgreSQL database credentials.

  5. Generate a secure, hashed password using the bcrypt package in Python 3.9 by entering the following command: 

    $ python3.9 -c 'import bcrypt; print(bcrypt.hashpw(b"newpass1234", bcrypt.gensalt(12)).decode("utf-8"))'
    Copy to Clipboard Toggle word wrap

    Example output

    $2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y
    Copy to Clipboard Toggle word wrap

  6. On the CLI, log in to the database, for example: 

    $ oc rsh quayuser-quay-quay-database-669c8998f-v9qsl
    Copy to Clipboard Toggle word wrap

  7. Enter the following command to open a connection to the quay PostgreSQL database server, specifying the database, username, and host address: 

    sh-4.4$ psql -U quayuser-quay-quay-database -d quayuser-quay-quay-database -W
    Copy to Clipboard Toggle word wrap

  8. Enter the following command to connect to the default database for the current user: 

    quay=> \c
    Copy to Clipboard Toggle word wrap

  9. Update the password_hash of the superuser admin who lost their password: 

    quay=> UPDATE public.user SET password_hash = '$2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y' where username = 'quayadmin';
    Copy to Clipboard Toggle word wrap

  10. Enter the following to command to ensure that the password_hash has been updated: 

    quay=> select * from public.user;
    Copy to Clipboard Toggle word wrap

    Example output

    id | uuid | username | password_hash | email | verified | stripe_id | organization | robot | invoice_email | invalid_login_attempts | last_invalid_login |removed_tag_expiration_s | enabled | invoice_email_address | company | family_name | given_name | location | maximum_queued_builds_count | creation_date | last_accessed
----+--------------------------------------+-----------+--------------------------------------------------------------+-----------------------+---
-------+-----------+--------------+-------+---------------+------------------------+----------------------------+--------------------------+------
---+-----------------------+---------+-------------+------------+----------+-----------------------------+----------------------------+-----------
1 | 73f04ef6-19ba-41d3-b14d-f2f1eed94a4a | quayadmin | $2b$12$zoilcTG6XQeAoVuDuIZH0..UpvQEZcKh3V6puksQJaUQupHgJ4.4y | quayadmin@example.com | t | | f | f | f | 0 | 2023-02-23 07:54:39.116485 | 1209600 | t | | | | | | | 2023-02-23 07:54:39.116492
    Copy to Clipboard Toggle word wrap

  11. Navigate to your Red Hat Quay UI on OpenShift Container Platform and log in using the new credentials.

6.2. Troubleshooting Red Hat Quay authentication
Copiar o link

Authentication and authorization secure access to Red Hat Quay and safeguard container images, verify identities, and enforce access controls.

The following authentication methods are supported by Red Hat Quay:

  • Username and password. Users can authentication by providing their username and password, which are validated against the user database configured in Red Hat Quay. This traditional method requires users to enter their credentials to gain access.
  • OAuth. Red Hat Quay supports OAuth authentication, which allows users to authenticate using their credentials from third party services like Google, GitHub, or Keycloak. OAuth enables a seamless and federated login experience, eliminating the need for separate account creation and simplifying user management.
  • OIDC. OpenID Connect enables single sign-on (SSO) capabilities and integration with enterprise identity providers. With OpenID Connect, users can authenticate using their existing organizational credentials, providing a unified authentication experience across various systems and applications.
  • Token-based authentication. Users can obtain unique tokens that grant access to specific resources within Red Hat Quay. Tokens can be obtained through various means, such as OAuth or by generating API tokens within the Red Hat Quay user interface. Token-based authentication is often used for automated or programmatic access to the registry.
  • External identity provider. Red Hat Quay can integrate with external identity providers, such as LDAP or AzureAD, for authentication purposes. This integration allows organizations to use their existing identity management infrastructure, enabling centralized user authentication and reducing the need for separate user databases.

To troubleshoot Red Hat Quay authentication and authorization issues for specific users, you can exec into the Red Hat Quay pod or container and query the federatedlogin and user tables to verify user records.

Procedure

  1. Exec into the Red Hat Quay pod or container. For more information, see "Interacting with the Red Hat Quay database".

  2. Enter the following command to show all users for external authentication: 

    quay=# select * from federatedlogin;
    Copy to Clipboard Toggle word wrap

    Example output

    id | user_id | service_id |                service_ident                |               metadata_json
----+---------+------------+---------------------------------------------+-------------------------------------------
1 |       1 |          3 | testuser0                                   | {}
2 |       1 |          8 | PK7Zpg2Yu2AnfUKG15hKNXqOXirqUog6G-oE7OgzSWc | {"service_username": "live.com#testuser0"}
3 |       2 |          3 | testuser1                                    | {}
4 |       2 |          4 | 110875797246250333431                       | {"service_username": "testuser1"}
5 |       3 |          3 | testuser2                                    | {}
6 |       3 |          1 | 26310880                                    | {"service_username": "testuser2"}
(6 rows)
    Copy to Clipboard Toggle word wrap

  3. Verify that the users are inserted into the user table: 

    quay=# select username, email from "user";
    Copy to Clipboard Toggle word wrap

    Example output

    username  |        email
-----------+----------------------
testuser0 | testuser0@outlook.com
testuser1  | testuser1@gmail.com
testuser2  | testuser2@redhat.com
(3 rows)
    Copy to Clipboard Toggle word wrap

6.2.2. Interacting with the Red Hat Quay database
Copiar o link

To interact with the Red Hat Quay PostgreSQL database and troubleshoot authorization or authentication issues, you can exec into the database using oc exec for Red Hat Quay on OpenShift Container Platform or podman exec for standalone, then enter the PostgreSQL shell with psql.

Warning

Interacting with the PostgreSQL database is potentially destructive. It is highly recommended that you perform the following procedure with the help of a Red Hat Quay Support Specialist.

Note

Interacting with the PostgreSQL database can also be used to troubleshoot authorization and authentication issues.

Procedure

  1. Exec into the Red Hat Quay database.

    1. Enter the following commands to exec into the Red Hat Quay database pod on OpenShift Container Platform: 

      $ oc exec -it <quay_database_pod> -- psql
      Copy to Clipboard Toggle word wrap

    2. Enter the following command to exec into the Red Hat Quay database on a standalone deployment: 

      $ sudo podman exec -it <quay_container_name> /bin/bash
      Copy to Clipboard Toggle word wrap

  2. Enter the PostgreSQL shell.

    Warning

    Interacting with the PostgreSQL database is potentially destructive. It is highly recommended that you perform the following procedure with the help of a Red Hat Quay Support Specialist.

    1. If you are using the Red Hat Quay Operator, enter the following command to enter the PostgreSQL shell: 

      $ oc rsh <quay_pod_name> psql -U your_username -d your_database_name
      Copy to Clipboard Toggle word wrap

    2. If you are on a standalone Red Hat Quay deployment, enter the following command to enter the PostgreSQL shell: 

      bash-4.4$ psql -U your_username -d your_database_name
      Copy to Clipboard Toggle word wrap

6.2.3. Troubleshooting crashloopbackoff states
Copiar o link

To troubleshoot crashloopbackoff states for your Red Hat Quay deployment and restore pods or containers, you can scale down the Quay Operator and database, then edit the database deployment as needed.

Procedure

  1. If your container or pod is in a crashloopbackoff state, you can enter the following commands.

    1. Enter the following command to scale down the Red Hat Quay Operator: 

      $ oc scale deployment/quay-operator.v3.8.z  --replicas=0
      Copy to Clipboard Toggle word wrap

      Example output

      deployment.apps/quay-operator.v3.8.z scaled
      Copy to Clipboard Toggle word wrap

    2. Enter the following command to scale down the Red Hat Quay database: 

      $ oc scale deployment/<quay_database> --replicas=0
      Copy to Clipboard Toggle word wrap

      Example output

      deployment.apps/<quay_database> scaled
      Copy to Clipboard Toggle word wrap

    3. Enter the following command to edit the Red Hat Quay database:

      Warning

      Interacting with the PostgreSQL database is potentially destructive. It is highly recommended that you perform the following procedure with the help of a Red Hat Quay Support Specialist. 

      $ oc edit deployment <quay_database>
      Copy to Clipboard Toggle word wrap 
      ...
 template:
   metadata:
     creationTimestamp: null
     labels:
       quay-component: <quay_database>
       quay-operator/quayregistry: quay-operator.v3.8.z
   spec:
     containers:
     - env:
       - name: POSTGRESQL_USER
         value: postgres
       - name: POSTGRESQL_DATABASE
         value: postgres
       - name: POSTGRESQL_PASSWORD
         value: postgres
       - name: POSTGRESQL_ADMIN_PASSWORD
         value: postgres
       - name: POSTGRESQL_MAX_CONNECTIONS
         value: "1000"
       image: registry.redhat.io/rhel8/postgresql-10@sha256:a52ad402458ec8ef3f275972c6ebed05ad64398f884404b9bb8e3010c5c95291
       imagePullPolicy: IfNotPresent
       name: postgres
       command: ["/bin/bash", "-c", "sleep 86400"]
      1 
      
...
      Copy to Clipboard Toggle word wrap

      • Add the command: ["/bin/bash", "-c", "sleep 86400"] line in the same indentation.

        Example output

        deployment.apps/<quay_database> edited
        Copy to Clipboard Toggle word wrap

    4. Execute the following command inside of your <quay_database>: 

      $ oc exec -it <quay_database> -- cat /var/lib/pgsql/data/userdata/postgresql/logs/* /path/to/desired_directory_on_host
      Copy to Clipboard Toggle word wrap

6.2.4. Checking the connectivity between Red Hat Quay and the database pod
Copiar o link

To check the connectivity between your Red Hat Quay instance and the database pod and troubleshoot connection issues, you can use oc exec for Operator deployments or podman exec for standalone deployments.

Procedure

  1. Check the connectivity between Red Hat Quay and the database pod.

    1. If you are using the Red Hat Quay Operator on OpenShift Container Platform, enter the following command: 

      $ oc exec -it _quay_pod_name_ -- curl -v telnet://<database_pod_name>:5432
      Copy to Clipboard Toggle word wrap

    2. If you are using a standalone deployment of Red Hat Quay, enter the following command: 

      $ podman exec -it <quay_container_name >curl -v telnet://<database_container_name>:5432
      Copy to Clipboard Toggle word wrap

6.2.5. Checking resource allocation
Copiar o link

To check resource allocation for your Red Hat Quay deployment and monitor disk, CPU, and memory usage, you can use oc exec or podman exec for disk usage and oc adm top pods or podman stats for other resources.

Procedure

  1. Obtain a list of running containers.

  2. Monitor disk usage of your Red Hat Quay deployment.

    1. If you are using the Red Hat Quay Operator on OpenShift Container Platform, enter the following command: 

      $ oc exec -it <quay_database_pod_name> -- df -ah
      Copy to Clipboard Toggle word wrap

    2. If you are using a standalone deployment of Red Hat Quay, enter the following command: 

      $ podman exec -it <quay_database_conatiner_name> df -ah
      Copy to Clipboard Toggle word wrap

  3. Monitor other resource usage.

    1. Enter the following command to check resource allocation on a Red Hat Quay Operator deployment: 

      $ oc adm top pods
      Copy to Clipboard Toggle word wrap

    2. Enter the following command to check the status of a specific pod on a standalone deployment of Red Hat Quay: 

      $ podman pod stats <pod_name>
      Copy to Clipboard Toggle word wrap

    3. Enter the following command to check the status of a specific container on a standalone deployment of Red Hat Quay: 

      $ podman stats <container_name>
      Copy to Clipboard Toggle word wrap

      The following information is returned:

      • CPU %. The percentage of CPU usage by the container since the last measurement. This value represents the container’s share of the available CPU resources.
      • MEM USAGE / LIMIT. The current memory usage of the container followed by its memory limit. The values are displayed in the format current_usage / memory_limit. For example, 300.4MiB / 7.795GiB indicates that the container is currently using 300.4 megabytes of memory out of a limit of 7.795 gigabytes.
      • MEM %. The percentage of memory usage by the container in relation to its memory limit.
      • NET I/O. The network I/O (input/output) statistics of the container. It displays the amount of data transmitted and received by the container over the network. The values are displayed in the format: transmitted_bytes / received_bytes.
      • BLOCK I/O. The block I/O (input/output) statistics of the container. It represents the amount of data read from and written to the block devices (for example, disks) used by the container. The values are displayed in the format read_bytes / written_bytes.

6.3. Troubleshooting Red Hat Quay object storage
Copiar o link

To troubleshoot Red Hat Quay object storage and resolve issues with container image storage, you can use the procedures in this section. Object storage manages data as discrete units called objects, each with a unique identifier and metadata.

Object storage is a type of data storage architecture that manages data as discrete units called objects. Unlike traditional file systems that organize data into hierarchical directories and files, object storage treats data as independent entities with unique identifiers. Each object contains the data itself, along with metadata that describes the object and enables efficient retrieval.

Red Hat Quay uses object storage as the underlying storage mechanism for storing and managing container images. It stores container images as individual objects. Each container image is treated as an object, with its own unique identifier and associated metadata.

6.3.1. Troubleshooting Red Hat Quay object storage issues
Copiar o link

To troubleshoot Red Hat Quay object storage issues, you can check the QuayRegistry CR and config.yaml file, verify supported storage and network connectivity, enable debug mode, and test storage access outside Quay.

Procedure

  • Enter the following command to see what object storage is used: 

    $ oc get quayregistry quay-registry-name -o yaml
    Copy to Clipboard Toggle word wrap
  • Ensure that the object storage you are using is officially supported by Red Hat Quay by checking the tested integrations page.
  • Enable debug mode. For more information, see "Running Red Hat Quay in debug mode".
  • Check your object storage configuration in your config.yaml file. Ensure that it is accurate and matches the settings provided by your object storage provider. You can check information like access credentials, endpoint URLs, bucket and container names, and other relevant configuration parameters.
  • Ensure that Red Hat Quay has network connectivity to the object storage endpoint. Check the network configurations to ensure that there are no restrictions blocking the communication between Red Hat Quay and the object storage endpoint.

  • If FEATURE_STORAGE_PROXY is enabled in your config.yaml file, check to see if its download URL is accessible. This can be found in the Red Hat Quay debug logs. For example: 

    $ curl -vvv "https://QUAY_HOSTNAME/_storage_proxy/dhaWZKRjlyO......Kuhc=/https/quay.hostname.com/quay-test/datastorage/registry/sha256/0e/0e1d17a1687fa270ba4f52a85c0f0e7958e13d3ded5123c3851a8031a9e55681?AWSAccessKeyId=xxxx&Signature=xxxxxx4%3D&Expires=1676066703"
    Copy to Clipboard Toggle word wrap
  • Try access the object storage service outside of Red Hat Quay to determine if the issue is specific to your deployment, or the underlying object storage. You can use command line tools like aws, gsutil, or s3cmd provided by the object storage provider to perform basic operations like listing buckets, containers, or uploading and downloading objects. This might help you isolate the problem.
Red Hat logoGithubredditYoutubeTwitter

Aprender

Experimente, compre e venda

Comunidades

Sobre a documentação da Red Hat

Ajudamos os usuários da Red Hat a inovar e atingir seus objetivos com nossos produtos e serviços com conteúdo em que podem confiar. Explore nossas atualizações recentes.

Tornando o open source mais inclusivo

A Red Hat está comprometida em substituir a linguagem problemática em nosso código, documentação e propriedades da web. Para mais detalhes veja o Blog da Red Hat.

Sobre a Red Hat

Fornecemos soluções robustas que facilitam o trabalho das empresas em plataformas e ambientes, desde o data center principal até a borda da rede.

Theme

© 2026 Red HatVoltar ao topo