Chapter 7. Quay Operator features
7.1. Console monitoring and alerting Copy linkLink copied to clipboard!
Red Hat Quay 3.6 provides support for monitoring Quay instances that were deployed using the Operator, from inside the OpenShift console. The new monitoring features include a Grafana dashboard, access to individual metrics, and alerting to notify for frequently restarting Quay pods.
To enable the monitoring features, the Operator must be installed in "all namespaces" mode.
7.1.1. Dashboard Copy linkLink copied to clipboard!
In the OpenShift console, navigate to Monitoring
The dashboard shows various statistics including:
- The number of Organizations, Repositories, Users and Robot accounts
- CPU Usage and Max Memory Usage
- Rates of Image Pulls and Pushes, and Authentication requests
- API request rate
- Latencies
7.1.2. Metrics Copy linkLink copied to clipboard!
You can see the underlying metrics behind the Quay dashboard, by accessing Monitoring quay_ to see the list of metrics available:
Select a sample metric, for example, quay_org_rows:
This metric shows the number of organizations in the registry, and it is directly surfaced in the dashboard as well.
7.1.3. Alerting Copy linkLink copied to clipboard!
An alert is raised if the Quay pods restart too often. The alert can be configured by accessing the Alerting rules tab from Monitoring
Select the QuayPodFrequentlyRestarting rule detail to configure the alert:
7.2. Manually updating the vulnerability databases for Clair in an air-gapped OpenShift cluster Copy linkLink copied to clipboard!
Clair utilizes packages called updaters that encapsulate the logic of fetching and parsing different vulnerability databases. Clair supports running updaters in a different environment and importing the results. This is aimed at supporting installations that disallow the Clair cluster from talking to the Internet directly.
To manually update the vulnerability databases for Clair in an air-gapped OpenShift cluster, use the following steps:
-
Obtain the
clairctlprogram - Retrieve the Clair config
-
Use
clairctlto export the updaters bundle from a Clair instance that has access to the internet - Update the Clair config in the air-gapped OpenShift cluster to allow access to the Clair database
- Transfer the updaters bundle from the system with internet access, to make it available inside the air-gapped environment
-
Use
clairctlto import the updaters bundle into the Clair instance for the air-gapped OpenShift cluster
7.2.1. Obtaining clairctl Copy linkLink copied to clipboard!
To obtain the clairctl program from a Clair deployment in an OpenShift cluster, use the oc cp command, for example:
$ oc -n quay-enterprise cp example-registry-clair-app-64dd48f866-6ptgw:/usr/bin/clairctl ./clairctl
$ chmod u+x ./clairctl
For a standalone Clair deployment, use the podman cp command, for example:
$ sudo podman cp clairv4:/usr/bin/clairctl ./clairctl
$ chmod u+x ./clairctl
7.2.2. Retrieving the Clair config Copy linkLink copied to clipboard!
7.2.2.1. Clair on OpenShift config Copy linkLink copied to clipboard!
To retrieve the configuration file for a Clair instance deployed using the OpenShift Operator, retrieve and decode the config secret using the appropriate namespace, and save it to file, for example:
$ kubectl get secret -n quay-enterprise example-registry-clair-config-secret -o "jsonpath={$.data['config\.yaml']}" | base64 -d > clair-config.yaml
An excerpt from a Clair configuration file is shown below:
clair-config.yaml
http_listen_addr: :8080
introspection_addr: ""
log_level: info
indexer:
connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
scanlock_retry: 10
layer_scan_concurrency: 5
migrations: true
scanner:
package: {}
dist: {}
repo: {}
airgap: false
matcher:
connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
max_conn_pool: 100
indexer_addr: ""
migrations: true
period: null
disable_updaters: false
notifier:
connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
migrations: true
indexer_addr: ""
matcher_addr: ""
poll_interval: 5m
delivery_interval: 1m
...
7.2.2.2. Standalone Clair config Copy linkLink copied to clipboard!
For standalone Clair deployments, the config file is the one specified in CLAIR_CONF environment variable in the podman run command, for example:
sudo podman run -d --rm --name clairv4 \
-p 8081:8081 -p 8089:8089 \
-e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo \
-v /etc/clairv4/config:/clair:Z \
registry.redhat.io/quay/clair-rhel8:v3.6.8
7.2.3. Exporting the updaters bundle Copy linkLink copied to clipboard!
From a Clair instance that has access to the internet, use clairctl with the appropriate configuration file to export the updaters bundle:
$ ./clairctl --config ./config.yaml export-updaters updates.gz
7.2.4. Configuring access to the Clair database in the air-gapped OpenShift cluster Copy linkLink copied to clipboard!
Use
kubectlto determine the Clair database service:$ kubectl get svc -n quay-enterprise NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-registry-clair-app ClusterIP 172.30.224.93 <none> 80/TCP,8089/TCP 4d21h example-registry-clair-postgres ClusterIP 172.30.246.88 <none> 5432/TCP 4d21h ...Forward the Clair database port so that it is accessible from the local machine, for example:
$ kubectl port-forward -n quay-enterprise service/example-registry-clair-postgres 5432:5432Update the Clair configuration file, replacing the value of the
hostin the multipleconnstringfields withlocalhost, for example:clair-config.yaml
... connstring: host=localhost port=5432 dbname=postgres user=postgres password=postgres sslmode=disable ...
As an alternative to using kubectl port-forward, you can use kubefwd instead. With this method, there is no need to modify the connstring field in the Clair configuration file to use localhost.
7.2.5. Importing the updaters bundle into the air-gapped environment Copy linkLink copied to clipboard!
After transferring the updaters bundle to the air-gapped environment, use clairctl to import the bundle into the Clair database deployed by the OpenShift Operator:
$ ./clairctl --config ./clair-config.yaml import-updaters updates.gz
7.3. FIPS readiness and compliance Copy linkLink copied to clipboard!
FIPS (the Federal Information Processing Standard developed by the National Institute of Standards and Technology, NIST) is regarded as the gold standard for securing and encrypting sensitive data, particularly in heavily regulated areas such as banking, healthcare and the public sector. Red Hat Enterprise Linux and Red Hat OpenShift Container Platform support this standard by providing a FIPS mode in which the system would only allow usage of certain, FIPS-validated cryptographic modules, like openssl. This ensures FIPS compliance.
Red Hat Quay supports running on RHEL and OCP in FIPS mode in production since version 3.5. Furthermore, Red Hat Quay itself also commits to exclusively using cryptography libraries that are validated or are in the process of being validated by NIST. Red Hat Quay 3.5 has pending FIPS 140-2 validation based on the RHEL 8.3 cryptography libraries. As soon as that validation is finalized, Red Hat Quay will be officially FIPS compliant.