Red Hat SummitChapter 13. Prometheus and Grafana metrics under Red Hat Quay
Red Hat Quay exports a Prometheus- and Grafana-compatible endpoint on each instance to allow for easy monitoring and alerting.
13.1. Exposing the Prometheus endpoint
13.1.1. Standalone Red Hat Quay
When using podman run to start the Quay container, expose the metrics port 9091:
sudo podman run -d --rm -p 80:8080 -p 443:8443 -p 9091:9091\ --name=quay \ -v $QUAY/config:/conf/stack:Z \ -v $QUAY/storage:/datastorage:Z \ registry.redhat.io/quay/quay-rhel8:v3.6.8
$ sudo podman run -d --rm -p 80:8080 -p 443:8443 -p 9091:9091\
--name=quay \
-v $QUAY/config:/conf/stack:Z \
-v $QUAY/storage:/datastorage:Z \
registry.redhat.io/quay/quay-rhel8:v3.6.8The metrics will now be available:
curl quay.example.com:9091/metrics
$ curl quay.example.com:9091/metricsSee Monitoring Quay with Prometheus and Grafana for details on configuring Prometheus and Grafana to monitor Quay repository counts.
13.1.2. Red Hat Quay Operator
Determine the cluster IP for the quay-metrics service:
Connect to your cluster and access the metrics using the cluster IP and port for the quay-metrics service:
13.1.3. Setting up Prometheus to consume metrics
Prometheus needs a way to access all Red Hat Quay instances running in a cluster. In the typical setup, this is done by listing all the Red Hat Quay instances in a single named DNS entry, which is then given to Prometheus.
13.1.4. DNS configuration under Kubernetes
A simple Kubernetes service can be configured to provide the DNS entry for Prometheus.
13.1.5. DNS configuration for a manual cluster
SkyDNS is a simple solution for managing this DNS record when not using Kubernetes. SkyDNS can run on an etcd cluster. Entries for each Red Hat Quay instance in the cluster can be added and removed in the etcd store. SkyDNS will regularly read them from there and update the list of Quay instances in the DNS record accordingly.
13.2. Introduction to metrics
Red Hat Quay provides metrics to help monitor the registry, including metrics for general registry usage, uploads, downloads, garbage collection, and authentication.
13.2.1. General registry statistics
General registry statistics can indicate how large the registry has grown.
| Metric name | Description |
|---|---|
| quay_user_rows | Number of users in the database |
| quay_robot_rows | Number of robot accounts in the database |
| quay_org_rows | Number of organizations in the database |
| quay_repository_rows | Number of repositories in the database |
| quay_security_scanning_unscanned_images_remaining_total | Number of images that are not scanned by the latest security scanner |
Sample metrics output
13.2.2. Queue items
The queue items metrics provide information on the multiple queues used by Quay for managing work.
| Metric name | Description |
|---|---|
| quay_queue_items_available | Number of items in a specific queue |
| quay_queue_items_locked | Number of items that are running |
| quay_queue_items_available_unlocked | Number of items that are waiting to be processed |
Metric labels
queue_name: The name of the queue. One of:
- exportactionlogs: Queued requests to export action logs. These logs are then processed and put in storage. A link is then sent to the requester via email.
- namespacegc: Queued namespaces to be garbage collected
- notification: Queue for repository notifications to be sent out
- repositorygc: Queued repositories to be garbage collected
- secscanv4: Notification queue specific for Clair V4
- dockerfilebuild: Queue for Quay docker builds
- imagestoragereplication: Queued blob to be replicated across multiple storages
- chunk_cleanup: Queued blob segments that needs to be deleted. This is only used by some storage implementations, for example, Swift.
For example, the queue labelled repositorygc contains the repositories marked for deletion by the repository garbage collection worker. For metrics with a queue_name label of repositorygc:
- quay_queue_items_locked is the number of repositories currently being deleted.
- quay_queue_items_available_unlocked is the number of repositories waiting to get processed by the worker.
Sample metrics output
13.2.3. Garbage collection metrics
These metrics show you how many resources have been removed from garbage collection (gc). They show many times the gc workers have run and how many namespaces, repositories, and blobs were removed.
| Metric name | Description |
|---|---|
| quay_gc_iterations_total | Number of iterations by the GCWorker |
| quay_gc_namespaces_purged_total | Number of namespaces purged by the NamespaceGCWorker |
| quay_gc_repos_purged_total | Number of repositories purged by the RepositoryGCWorker or NamespaceGCWorker |
| quay_gc_storage_blobs_deleted_total | Number of storage blobs deleted |
Sample metrics output
13.2.3.1. Multipart uploads metrics
The multipart uploads metrics show the number of blobs uploads to storage (S3, Rados, GoogleCloudStorage, RHOCS). These can help identify issues when Quay is unable to correctly upload blobs to storage.
| Metric name | Description |
|---|---|
| quay_multipart_uploads_started_total | Number of multipart uploads to Quay storage that completed |
| quay_multipart_uploads_completed_total | Number of multipart uploads to Quay storage that started |
Sample metrics output
13.2.4. Image push / pull metrics
A number of metrics are available related to pushing and pulling images.
13.2.4.1. Image pulls total
| Metric name | Description |
|---|---|
| quay_registry_image_pulls_total | The number of images downloaded from the registry. |
Metric labels
- protocol: the registry protocol used (should always be v2)
- ref: ref used to pull - tag, manifest
- status: http return code of the request
13.2.4.2. Image bytes pulled
| Metric name | Description |
|---|---|
| quay_registry_image_pulled_estimated_bytes_total | The number of bytes downloaded from the registry |
Metric labels
- protocol: the registry protocol used (should always be v2)
13.2.4.3. Image pushes total
| Metric name | Description |
|---|---|
| quay_registry_image_pushes_total | The number of images uploaded from the registry. |
Metric labels
- protocol: the registry protocol used (should always be v2)
- pstatus: http return code of the request
- pmedia_type: the uploaded manifest type
13.2.4.4. Image bytes pushed
| Metric name | Description |
|---|---|
| quay_registry_image_pushed_bytes_total | The number of bytes uploaded to the registry |
Sample metrics output
HELP quay_registry_image_pushed_bytes_total number of bytes pushed to the registry TYPE quay_registry_image_pushed_bytes_total counter
# HELP quay_registry_image_pushed_bytes_total number of bytes pushed to the registry
# TYPE quay_registry_image_pushed_bytes_total counter
quay_registry_image_pushed_bytes_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="221",process_name="registry:application"} 0
...13.2.5. Authentication metrics
The authentication metrics provide the number of authentication requests, labeled by type and whether it succeeded or not. For example, this metric could be used to monitor failed basic authentication requests.
| Metric name | Description |
|---|---|
| quay_authentication_attempts_total | Number of authentication attempts across the registry and API |
Metric labels
auth_kind: The type of auth used, including:
- basic
- oauth
- credentials
- success: true or false
Sample metrics output
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}