Chapter 13. OCI Support and Red Hat Quay
Container registries such as Red Hat Quay were originally designed to support container images in the Docker image format. To promote the use of additional runtimes apart from Docker, the Open Container Initiative (OCI) was created to provide a standardization surrounding container runtimes and image formats. Most container registries support the OCI standardization as it is based on the Docker image manifest V2, Schema 2 format.
In addition to container images, a variety of artifacts have emerged that support not just individual applications, but the Kubernetes platform as a whole. These range from Open Policy Agent (OPA) policies for security and governance to Helm charts and Operators to aid in application deployment.
Red Hat Quay is a private container registry that not only stores container images, but supports an entire ecosystem of tooling to aid in the management of containers. Support for OCI based artifacts in Red Hat Quay 3.6 has extended from solely Helm to include cosign and ztsd compression schemes by default. As such, FEATURE_HELM_OCI_SUPPORT
has been deprecated.
When Red Hat Quay 3.6 is deployed using the OpenShift Operator, support for Helm and OCI artifacts is now enabled by default under the FEATURE_GENERAL_OCI_SUPPORT
configuration. If you need to explicitly enable the feature, for example, if it has previously been disabled or if you have upgraded from a version where it is not enabled by default, see the section Explicitly enabling OCI and Helm support.
13.1. Helm and OCI prerequisites
Trusted certificates: Communication between the Helm client and Quay is facilitated over HTTPS and as of Helm 3.5, support is only available for registries communicating over HTTPS with trusted certificates. In addition, the operating system must trust the certificates exposed by the registry. Support in future Helm releases will allow for communicating with remote registries insecurely. With that in mind, ensure that your operating system has been configured to trust the certificates used by Quay, for example:
$ sudo cp rootCA.pem /etc/pki/ca-trust/source/anchors/ $ sudo update-ca-trust extract
- Generally available: As of Helm 3.8, OCI registry support for charts is now generally available.
Install Helm client: Download your desired version from the Helm releases page. Unpack it and move the helm binary to its desired destination:
$ tar -zxvf helm-v3.8.2-linux-amd64.tar.gz $ mv linux-amd64/helm /usr/local/bin/helm
-
Create organization in Quay: Create a new organization for storing the Helm charts, using the Quay registry UI. For example, create an organization named
helm
.
13.2. Helm charts with Red Hat Quay
Helm, as a graduated project of the Cloud Native Computing Foundation (CNCF), has become the de facto package manager for Kubernetes as it simplifies how applications are packaged and deployed. Helm uses a packaging format called Charts which contain the Kubernetes resources representing an application. Charts can be made available for general distribution and consumption in repositories. A Helm repository is an HTTP server that serves an index.yaml
metadata file and optionally a set of packaged charts. Beginning with Helm version 3, support was made available for distributing charts in OCI registries as an alternative to a traditional repository.
13.2.1. Using Helm charts with Red Hat Quay
Use the following example to download and push an etherpad chart from the Red Hat Community of Practice (CoP) repository.
Procedure
Add a chart repository:
$ helm repo add redhat-cop https://redhat-cop.github.io/helm-charts
Update the information of available charts locally from the chart repository:
$ helm repo update
Download a chart from a repository:
$ helm pull redhat-cop/etherpad --version=0.0.4 --untar
Package the chart into a chart archive:
$ helm package ./etherpad
Example output
Successfully packaged chart and saved it to: /home/user/linux-amd64/etherpad-0.0.4.tgz
Log in to your Quay repository using
helm registry login
:$ helm registry login quay370.apps.quayperf370.perfscale.devcluster.openshift.com
Push the chart to your Quay repository using the
helm push
command:$ helm push etherpad-0.0.4.tgz oci://quay370.apps.quayperf370.perfscale.devcluster.openshift.com
Example output:
Pushed: quay370.apps.quayperf370.perfscale.devcluster.openshift.com/etherpad:0.0.4 Digest: sha256:a6667ff2a0e2bd7aa4813db9ac854b5124ff1c458d170b70c2d2375325f2451b
Ensure that the push worked by deleting the local copy, and then pulling the chart from the repository:
$ rm -rf etherpad-0.0.4.tgz
$ helm pull oci://quay370.apps.quayperf370.perfscale.devcluster.openshift.com/etherpad --version 0.0.4
Example output:
Pulled: quay370.apps.quayperf370.perfscale.devcluster.openshift.com/etherpad:0.0.4 Digest: sha256:4f627399685880daf30cf77b6026dc129034d68c7676c7e07020b70cf7130902
13.3. OCI and Helm configuration fields
Support for Helm is now supported under the FEATURE_GENERAL_OCI_SUPPORT
property. If you need to explicitly enable the feature, for example, if it has previously been disabled or if you have upgraded from a version where it is not enabled by default, you need to add two properties in the Quay configuration to enable the use of OCI artifacts:
FEATURE_GENERAL_OCI_SUPPORT: true FEATURE_HELM_OCI_SUPPORT: true
Field | Type | Description |
---|---|---|
FEATURE_GENERAL_OCI_SUPPORT | Boolean |
Enable support for OCI artifacts |
FEATURE_HELM_OCI_SUPPORT | Boolean |
Enable support for Helm artifacts |
As of Red Hat Quay 3.6, FEATURE_HELM_OCI_SUPPORT
has been deprecated and will be removed in a future version of Red Hat Quay. In Red Hat Quay 3.6, Helm artifacts are supported by default and included under the FEATURE_GENERAL_OCI_SUPPORT
property. Users are no longer required to update their config.yaml files to enable support.
13.4. Cosign OCI support with Red Hat Quay
Cosign is a tool that can be used to sign and verify container images. It uses the ECDSA-P256 signature algorithm and Red Hat’s Simple Signing payload format to create public keys that are stored in PKIX files. Private keys are stored as encrypted PEM files.
Cosign currently supports the following:
- Hardware and KMS Signing
- Bring-your-own PKI
- OIDC PKI
- Built-in binary transparency and timestamping service
13.5. Using cosign with quay
If you have Go 1.16+, you can directly install cosign with the following command:
$ go install github.com/sigstore/cosign/cmd/cosign@v1.0.0 go: downloading github.com/sigstore/cosign v1.0.0 go: downloading github.com/peterbourgon/ff/v3 v3.1.0 ...
Next, generate a keypair:
$ cosign generate-key-pair Enter password for private key: Enter again: Private key written to cosign.key Public key written to cosign.pub
Sign the keypair with the following command:
$ cosign sign -key cosign.key quay-server.example.com/user1/busybox:test Enter password for private key: Pushing signature to: quay-server.example.com/user1/busybox:sha256-ff13b8f6f289b92ec2913fa57c5dd0a874c3a7f8f149aabee50e3d01546473e3.sig
Some users may experience the following error:
error: signing quay-server.example.com/user1/busybox:test: getting remote image: GET https://quay-server.example.com/v2/user1/busybox/manifests/test: UNAUTHORIZED: access to the requested resource is not authorized; map[]
Because cosign relies on ~/.docker/config.json for authorization, you might need to execute the following command:
$ podman login --authfile ~/.docker/config.json quay-server.example.com Username: Password: Login Succeeded!
You can see the updated authorization configuration using the following command:
$ cat ~/.docker/config.json { "auths": { "quay-server.example.com": { "auth": "cXVheWFkbWluOnBhc3N3b3Jk" } }
13.6. Adding other OCI media types to Quay
Helm, cosign, and ztsd compression scheme artifacts are built into Red Hat Quay 3.6 by default. For any other OCI media type that is not supported by default, you can add them to the ALLOWED_OCI_ARTIFACT_TYPES
configuration in Quay’s config.yaml using the following format:
ALLOWED_OCI_ARTIFACT_TYPES: <oci config type 1>: - <oci layer type 1> - <oci layer type 2> <oci config type 2>: - <oci layer type 3> - <oci layer type 4> ...
For example, you can add Singularity (SIF) support by adding the following to your config.yaml:
... ALLOWED_OCI_ARTIFACT_TYPES: application/vnd.oci.image.config.v1+json: - application/vnd.dev.cosign.simplesigning.v1+json application/vnd.cncf.helm.config.v1+json: - application/tar+gzip application/vnd.sylabs.sif.config.v1+json: - application/vnd.sylabs.sif.layer.v1+tar ...
When adding OCI media types that are not configured by default, users will also need to manually add support for cosign and Helm if desired. The ztsd compression scheme is supported by default, so users will not need to add that OCI media type to their config.yaml to enable support.
13.7. Disabling OCI artifacts in Quay
If you want to disable OCI artifact support, you can set FEATURE_GENERAL_OCI_SUPPORT
to False
in your config.yaml:
... FEATURE_GENERAL_OCI_SUPPORT = False ...