13장. Prometheus and Grafana metrics under Red Hat Quay
Red Hat Quay exports a Prometheus- and Grafana-compatible endpoint on each instance to allow for easy monitoring and alerting.
13.1. Exposing the Prometheus endpoint 링크 복사링크가 클립보드에 복사되었습니다!
13.1.1. Standalone Red Hat Quay 링크 복사링크가 클립보드에 복사되었습니다!
When using podman run to start the Quay container, expose the metrics port 9091:
$ sudo podman run -d --rm -p 80:8080 -p 443:8443 -p 9091:9091\
--name=quay \
-v $QUAY/config:/conf/stack:Z \
-v $QUAY/storage:/datastorage:Z \
registry.redhat.io/quay/quay-rhel9:v3.17.1
The metrics will now be available:
$ curl quay.example.com:9091/metrics
See Monitoring Quay with Prometheus and Grafana for details on configuring Prometheus and Grafana to monitor Quay repository counts.
13.1.2. Red Hat Quay Operator 링크 복사링크가 클립보드에 복사되었습니다!
Determine the cluster IP for the quay-metrics service:
$ oc get services -n quay-enterprise
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-registry-clair-app ClusterIP 172.30.61.161 <none> 80/TCP,8089/TCP 18h
example-registry-clair-postgres ClusterIP 172.30.122.136 <none> 5432/TCP 18h
example-registry-quay-app ClusterIP 172.30.72.79 <none> 443/TCP,80/TCP,8081/TCP,55443/TCP 18h
example-registry-quay-config-editor ClusterIP 172.30.185.61 <none> 80/TCP 18h
example-registry-quay-database ClusterIP 172.30.114.192 <none> 5432/TCP 18h
example-registry-quay-metrics ClusterIP 172.30.37.76 <none> 9091/TCP 18h
example-registry-quay-redis ClusterIP 172.30.157.248 <none> 6379/TCP 18h
Connect to your cluster and access the metrics using the cluster IP and port for the quay-metrics service:
$ oc debug node/master-0
sh-4.4# curl 172.30.37.76:9091/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.0447e-05
go_gc_duration_seconds{quantile="0.25"} 6.2203e-05
...
13.1.3. Setting up Prometheus to consume metrics 링크 복사링크가 클립보드에 복사되었습니다!
Prometheus needs a way to access all Red Hat Quay instances running in a cluster. In the typical setup, this is done by listing all the Red Hat Quay instances in a single named DNS entry, which is then given to Prometheus.
13.1.4. DNS configuration under Kubernetes 링크 복사링크가 클립보드에 복사되었습니다!
A simple Kubernetes service can be configured to provide the DNS entry for Prometheus.
13.1.5. DNS configuration for a manual cluster 링크 복사링크가 클립보드에 복사되었습니다!
SkyDNS is a simple solution for managing this DNS record when not using Kubernetes. SkyDNS can run on an etcd cluster. Entries for each Red Hat Quay instance in the cluster can be added and removed in the etcd store. SkyDNS will regularly read them from there and update the list of Quay instances in the DNS record accordingly.