Chapter 7. Quay Operator features

7.1. Console monitoring and alerting

Red Hat Quay 3.6 provides support for monitoring Quay instances that were deployed using the Operator, from inside the OpenShift console. The new monitoring features include a Grafana dashboard, access to individual metrics, and alerting to notify for frequently restarting Quay pods.

Note

To enable the monitoring features, the Operator must be installed in "all namespaces" mode.

7.1.1. Dashboard

In the OpenShift console, navigate to Monitoring Dashboards and search for the dashboard of your desired Quay registry instance:

Choose Quay dashboard

The dashboard shows various statistics including:

The number of Organizations, Repositories, Users and Robot accounts
CPU Usage and Max Memory Usage
Rates of Image Pulls and Pushes, and Authentication requests
API request rate
Latencies

Console dashboard

7.1.2. Metrics

You can see the underlying metrics behind the Quay dashboard, by accessing Monitoring Metrics in the UI. In the Expression field, enter the text quay_ to see the list of metrics available:

Quay metrics

Select a sample metric, for example, quay_org_rows:

Number of Quay organizations

This metric shows the number of organizations in the registry, and it is directly surfaced in the dashboard as well.

7.1.3. Alerting

An alert is raised if the Quay pods restart too often. The alert can be configured by accessing the Alerting rules tab from Monitoring Alerting in the consol UI and searching for the Quay-specific alert:

Alerting rules

Select the QuayPodFrequentlyRestarting rule detail to configure the alert:

Alerting rule details

7.2. Manually updating the vulnerability databases for Clair in an air-gapped OpenShift cluster

Clair utilizes packages called updaters that encapsulate the logic of fetching and parsing different vulnerability databases. Clair supports running updaters in a different environment and importing the results. This is aimed at supporting installations that disallow the Clair cluster from talking to the Internet directly.

To manually update the vulnerability databases for Clair in an air-gapped OpenShift cluster, use the following steps:

Obtain the clairctl program
Retrieve the Clair config
Use clairctl to export the updaters bundle from a Clair instance that has access to the internet
Update the Clair config in the air-gapped OpenShift cluster to allow access to the Clair database
Transfer the updaters bundle from the system with internet access, to make it available inside the air-gapped environment
Use clairctl to import the updaters bundle into the Clair instance for the air-gapped OpenShift cluster

7.2.1. Obtaining clairctl

To obtain the clairctl program from a Clair deployment in an OpenShift cluster, use the oc cp command, for example:

$ oc -n quay-enterprise cp example-registry-clair-app-64dd48f866-6ptgw:/usr/bin/clairctl ./clairctl
$ chmod u+x ./clairctl

For a standalone Clair deployment, use the podman cp command, for example:

$ sudo podman cp clairv4:/usr/bin/clairctl ./clairctl
$ chmod u+x ./clairctl

7.2.2. Retrieving the Clair config

7.2.2.1. Clair on OpenShift config

To retrieve the configuration file for a Clair instance deployed using the OpenShift Operator, retrieve and decode the config secret using the appropriate namespace, and save it to file, for example:

$ kubectl get secret -n quay-enterprise example-registry-clair-config-secret  -o "jsonpath={$.data['config\.yaml']}" | base64 -d > clair-config.yaml

An excerpt from a Clair configuration file is shown below:

clair-config.yaml

http_listen_addr: :8080
introspection_addr: ""
log_level: info
indexer:
    connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
    scanlock_retry: 10
    layer_scan_concurrency: 5
    migrations: true
    scanner:
        package: {}
        dist: {}
        repo: {}
    airgap: false
matcher:
    connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
    max_conn_pool: 100
    indexer_addr: ""
    migrations: true
    period: null
    disable_updaters: false
notifier:
    connstring: host=example-registry-clair-postgres port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
    migrations: true
    indexer_addr: ""
    matcher_addr: ""
    poll_interval: 5m
    delivery_interval: 1m
    ...

7.2.2.2. Standalone Clair config

For standalone Clair deployments, the config file is the one specified in CLAIR_CONF environment variable in the podman run command, for example:

sudo podman run -d --rm --name clairv4 \
  -p 8081:8081 -p 8089:8089 \
  -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo \
  -v /etc/clairv4/config:/clair:Z \
  registry.redhat.io/quay/clair-rhel8:v3.6.8

7.2.3. Exporting the updaters bundle

From a Clair instance that has access to the internet, use clairctl with the appropriate configuration file to export the updaters bundle:

$ ./clairctl --config ./config.yaml export-updaters updates.gz

7.2.4. Configuring access to the Clair database in the air-gapped OpenShift cluster

Use kubectl to determine the Clair database service:

$ kubectl get svc -n quay-enterprise

NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                             AGE
example-registry-clair-app            ClusterIP      172.30.224.93    <none>        80/TCP,8089/TCP                     4d21h
example-registry-clair-postgres       ClusterIP      172.30.246.88    <none>        5432/TCP                            4d21h
...

Forward the Clair database port so that it is accessible from the local machine, for example:
```
$ kubectl port-forward -n quay-enterprise service/example-registry-clair-postgres 5432:5432
```
Update the Clair configuration file, replacing the value of the host in the multiple connstring fields with localhost, for example:
clair-config.yaml
```
    ...
    connstring: host=localhost port=5432 dbname=postgres user=postgres password=postgres sslmode=disable
    ...
```

Note

As an alternative to using kubectl port-forward, you can use kubefwd instead. With this method, there is no need to modify the connstring field in the Clair configuration file to use localhost.

7.2.5. Importing the updaters bundle into the air-gapped environment

After transferring the updaters bundle to the air-gapped environment, use clairctl to import the bundle into the Clair database deployed by the OpenShift Operator:

$ ./clairctl --config ./clair-config.yaml import-updaters updates.gz

7.3. FIPS readiness and compliance

FIPS (the Federal Information Processing Standard developed by the National Institute of Standards and Technology, NIST) is regarded as the gold standard for securing and encrypting sensitive data, particularly in heavily regulated areas such as banking, healthcare and the public sector. Red Hat Enterprise Linux and Red Hat OpenShift Container Platform support this standard by providing a FIPS mode in which the system would only allow usage of certain, FIPS-validated cryptographic modules, like openssl. This ensures FIPS compliance.

Red Hat Quay supports running on RHEL and OCP in FIPS mode in production since version 3.5. Furthermore, Red Hat Quay itself also commits to exclusively using cryptography libraries that are validated or are in the process of being validated by NIST. Red Hat Quay 3.5 has pending FIPS 140-2 validation based on the RHEL 8.3 cryptography libraries. As soon as that validation is finalized, Red Hat Quay will be officially FIPS compliant.