Deploying the Red Hat Quay Operator on OpenShift Container Platform
Deploying the Red Hat Quay Operator on OpenShift Container Platform
Abstract
Preface
Red Hat Quay is an enterprise-quality container registry. Use Red Hat Quay to build and store container images, then make them available to deploy across your enterprise.
The Red Hat Quay Operator provides a simple method to deploy and manage Red Hat Quay on an OpenShift cluster.
With the release of Red Hat Quay 3.4.0, the Red Hat Quay Operator was re-written to offer an enhanced experience and to add more support for Day 2 operations. As a result, the Red Hat Quay Operator is now simpler to use and is more opinionated. The key difference from versions prior to Red Hat Quay 3.4.0 include the following:
-
The
QuayEcosystem
custom resource has been replaced with theQuayRegistry
custom resource. The default installation options produces a fully supported Red Hat Quay environment, with all managed dependencies, such as database, caches, object storage, and so on, supported for production use.
NoteSome components might not be highly available.
- A new validation library for Red Hat Quay’s configuration.
Object storage can now be managed by the Red Hat Quay Operator using the
ObjectBucketClaim
Kubernetes APINoteRed Hat OpenShift Data Foundation can be used to provide a supported implementation of this API on OpenShift Container Platform.
- Customization of the container images used by deployed pods for testing and development scenarios.
Chapter 1. Introduction to the Red Hat Quay Operator
Use the content in this chapter to execute the following:
- Install Red Hat Quay on OpenShift Container Platform using the Red Hat Quay Operator
- Configure managed, or unmanaged, object storage
- Configure unmanaged components, such as the database, Redis, routes, TLS, and so on
- Deploy the Red Hat Quay registry on OpenShift Container Platform using the Red Hat Quay Operator
- Use advanced features supported by Red Hat Quay
- Upgrade the Red Hat Quay registry by using the Red Hat Quay Operator
1.1. Red Hat Quay Operator components
Red Hat Quay has many dependencies. These dependencies include a database, object storage, Redis, and others. The Red Hat Quay Operator manages an opinionated deployment of Red Hat Quay and its dependencies on Kubernetes. These dependencies are treated as components and are configured through the QuayRegistry
API.
In the QuayRegistry
custom resource, the spec.components
field configures components. Each component contains two fields: kind
(the name of the component), and managed
(a boolean that addresses whether the component lifecycle is handled by the Red Hat Quay Operator).
By default, all components are managed and auto-filled upon reconciliation for visibility:
Example QuayRegistry
resource
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: configBundleSecret: config-bundle-secret components: - kind: quay managed: true - kind: postgres managed: true - kind: clair managed: true - kind: redis managed: true - kind: horizontalpodautoscaler managed: true - kind: objectstorage managed: true - kind: route managed: true - kind: mirror managed: true - kind: monitoring managed: true - kind: tls managed: true - kind: clairpostgres managed: true
1.2. Using managed components
Unless your QuayRegistry
custom resource specifies otherwise, the Red Hat Quay Operator uses defaults for the following managed components:
- quay: Holds overrides for deployment of Red Hat Quay on OpenShift Container Platform, for example, environment variables and number of replicas. This component is new as of Red Hat Quay 3.7 and cannot be set to unmanaged.
postgres: For storing the registry metadata, As of Red Hat Quay 3.9, uses a version of PostgreSQL 13 from Software Collections.
NoteWhen upgrading from Red Hat Quay 3.8 → 3.9, the Operator automatically handles upgrading PostgreSQL 10 to PostgreSQL 13. This upgrade is required. PostgreSQL 10 had its final release on November 10, 2022 and is no longer supported.
- clair: Provides image vulnerability scanning.
- redis: Stores live builder logs and the Red Hat Quay tutorial. Also includes the locking mechanism that is required for garbage collection.
-
horizontalpodautoscaler: Adjusts the number of
Quay
pods depending on memory/cpu consumption. -
objectstorage: For storing image layer blobs, utilizes the
ObjectBucketClaim
Kubernetes API which is provided by Noobaa or Red Hat OpenShift Data Foundation. - route: Provides an external entrypoint to the Red Hat Quay registry from outside of OpenShift Container Platform.
- mirror: Configures repository mirror workers to support optional repository mirroring.
-
monitoring: Features include a Grafana dashboard, access to individual metrics, and notifications for frequently restarting
Quay
pods. - tls: Configures whether Red Hat Quay or OpenShift Container Platform handles SSL/TLS.
- clairpostgres: Configures a managed Clair database. This is a separate database than the PostgreSQL database used to deploy Red Hat Quay.
The Red Hat Quay Operator handles any required configuration and installation work needed for Red Hat Quay to use the managed components. If the opinionated deployment performed by the Red Hat Quay Operator is unsuitable for your environment, you can provide the Red Hat Quay Operator with unmanaged
resources, or overrides, as described in the following sections.
1.3. Using unmanaged components for dependencies
If you have existing components such as PostgreSQL, Redis, or object storage that you want to use with Red Hat Quay, you first configure them within the Red Hat Quay configuration bundle, or the config.yaml
file. Then, they must be referenced in your QuayRegistry
bundle as a Kubernetes Secret
while indicating which components are unmanaged.
If you are using an unmanaged PostgreSQL database, and the version is PostgreSQL 10, it is highly recommended that you upgrade to PostgreSQL 13. PostgreSQL 10 had its final release on November 10, 2022 and is no longer supported. For more information, see the PostgreSQL Versioning Policy.
See the following sections for configuring unmanaged components:
1.4. Config bundle secret
The spec.configBundleSecret
field is a reference to the metadata.name
of a Secret
in the same namespace as the QuayRegistry
resource. This Secret
must contain a config.yaml
key/value pair.
The config.yaml
file is a Red Hat Quay config.yaml
file. This field is optional, and is auto-filled by the Red Hat Quay Operator if not provided. If provided, it serves as the base set of config fields which are later merged with other fields from any managed components to form a final output Secret
, which is then mounted into the Red Hat Quay application pods.
1.5. Prerequisites for Red Hat Quay on OpenShift Container Platform
Consider the following prerequisites prior to deploying Red Hat Quay on OpenShift Container Platform using the Red Hat Quay Operator.
1.5.1. OpenShift Container Platform cluster
To deploy the Red Hat Quay Operator, you must have an OpenShift Container Platform 4.5 or later cluster and access to an administrative account. The administrative account must have the ability to create namespaces at the cluster scope.
1.5.2. Resource Requirements
Each Red Hat Quay application pod has the following resource requirements:
- 8 Gi of memory
- 2000 millicores of CPU
The Red Hat Quay Operator creates at least one application pod per Red Hat Quay deployment it manages. Ensure your OpenShift Container Platform cluster has sufficient compute resources for these requirements.
1.5.3. Object Storage
By default, the Red Hat Quay Operator uses the ObjectBucketClaim
Kubernetes API to provision object storage. Consuming this API decouples the Red Hat Quay Operator from any vendor-specific implementation. Red Hat OpenShift Data Foundation provides this API through its NooBaa component, which is used as an example throughout this documentation.
Red Hat Quay can be manually configured to use multiple storage cloud providers, including the following:
- Amazon S3 (see S3 IAM Bucket Policy for details on configuring an S3 bucket policy for Red Hat Quay)
- Microsoft Azure Blob Storage
- Google Cloud Storage
- Ceph Object Gateway (RADOS)
- OpenStack Swift
- CloudFront + S3
For a complete list of object storage providers, the Quay Enterprise 3.x support matrix.
1.5.4. StorageClass
When deploying Quay
and Clair
PostgreSQL databases using the Red Hat Quay Operator, a default StorageClass
is configured in your cluster.
The default StorageClass
used by the Red Hat Quay Operator provisions the Persistent Volume Claims required by the Quay
and Clair
databases. These PVCs are used to store data persistently, ensuring that your Red Hat Quay registry and Clair vulnerability scanner remain available and maintain their state across restarts or failures.
Before proceeding with the installation, verify that a default StorageClass
is configured in your cluster to ensure seamless provisioning of storage for Quay
and Clair
components.
Chapter 2. Installing the Red Hat Quay Operator from the OperatorHub
Use the following procedure to install the Red Hat Quay Operator from the OpenShift Container Platform OperatorHub.
Procedure
- Using the OpenShift Container Platform console, select Operators → OperatorHub.
- In the search box, type Red Hat Quay and select the official Red Hat Quay Operator provided by Red Hat. This directs you to the Installation page, which outlines the features, prerequisites, and deployment information.
- Select Install. This directs you to the Operator Installation page.
The following choices are available for customizing the installation:
-
Update Channel: Choose the update channel, for example,
stable-3.13
for the latest release. Installation Mode:
-
Choose
All namespaces on the cluster
if you want the Red Hat Quay Operator to be available cluster-wide. It is recommended that you install the Red Hat Quay Operator cluster-wide. If you choose a single namespace, the monitoring component will not be available by default. Choose
A specific namespace on the cluster
if you want it deployed only within a single namespace.- Approval Strategy: Choose to approve either automatic or manual updates. Automatic update strategy is recommended.
-
Choose
-
Update Channel: Choose the update channel, for example,
- Select Install.
Chapter 3. Configuring Red Hat Quay before deployment
The Red Hat Quay Operator can manage all of the Red Hat Quay components when deployed on OpenShift Container Platform. This is the default configuration, however, you can manage one or more components externally when you want more control over the set up.
Use the following pattern to configure unmanaged Red Hat Quay components.
Procedure
Create a
config.yaml
configuration file with the appropriate settings. Use the following reference for a minimal configuration:$ touch config.yaml
AUTHENTICATION_TYPE: Database BUILDLOGS_REDIS: host: <quay-server.example.com> password: <strongpassword> port: 6379 ssl: false DATABASE_SECRET_KEY: <0ce4f796-c295-415b-bf9d-b315114704b8> DB_URI: <postgresql://quayuser:quaypass@quay-server.example.com:5432/quay> DEFAULT_TAG_EXPIRATION: 2w DISTRIBUTED_STORAGE_CONFIG: default: - LocalStorage - storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: - default PREFERRED_URL_SCHEME: http SECRET_KEY: <e8f9fe68-1f84-48a8-a05f-02d72e6eccba> SERVER_HOSTNAME: <quay-server.example.com> SETUP_COMPLETE: true TAG_EXPIRATION_OPTIONS: - 0s - 1d - 1w - 2w - 4w USER_EVENTS_REDIS: host: <quay-server.example.com> port: 6379 ssl: false
Create a
Secret
using the configuration file by entering the following command:$ oc create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
Create a
quayregistry.yaml
file, identifying the unmanaged components and also referencing the createdSecret
, for example:Example
QuayRegistry
YAML fileapiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: configBundleSecret: <config_bundle_secret> components: - kind: objectstorage managed: false # ...
Enter the following command to deploy the registry by using the
quayregistry.yaml
file:$ oc create -n quay-enterprise -f quayregistry.yaml
3.1. Pre-configuring Red Hat Quay for automation
Red Hat Quay supports several configuration options that enable automation. Users can configure these options before deployment to reduce the need for interaction with the user interface.
3.1.1. Allowing the API to create the first user
To create the first user, users need to set the FEATURE_USER_INITIALIZE
parameter to true
and call the /api/v1/user/initialize
API. Unlike all other registry API calls that require an OAuth token generated by an OAuth application in an existing organization, the API endpoint does not require authentication.
Users can use the API to create a user such as quayadmin
after deploying Red Hat Quay, provided no other users have been created. For more information, see Using the API to create the first user.
3.1.2. Enabling general API access
Users should set the BROWSER_API_CALLS_XHR_ONLY
configuration option to false
to allow general access to the Red Hat Quay registry API.
3.1.3. Adding a superuser
After deploying Red Hat Quay, users can create a user and give the first user administrator privileges with full permissions. Users can configure full permissions in advance by using the SUPER_USER
configuration object. For example:
# ... SERVER_HOSTNAME: quay-server.example.com SETUP_COMPLETE: true SUPER_USERS: - quayadmin # ...
3.1.4. Restricting user creation
After you have configured a superuser, you can restrict the ability to create new users to the superuser group by setting the FEATURE_USER_CREATION
to false
. For example:
# ... FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false # ...
3.1.5. Enabling new functionality in Red Hat Quay 3.13
To use new Red Hat Quay 3.13 functions, enable some or all of the following features:
# ... FEATURE_UI_V2: true FEATURE_UI_V2_REPO_SETTINGS: true FEATURE_AUTO_PRUNE: true ROBOTS_DISALLOW: false # ...
3.1.6. Suggested configuration for automation
The following config.yaml
parameters are suggested for automation:
# ... FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false # ...
3.2. Configuring object storage
You need to configure object storage before installing Red Hat Quay, irrespective of whether you are allowing the Red Hat Quay Operator to manage the storage or managing it yourself.
If you want the Red Hat Quay Operator to be responsible for managing storage, see the section on Managed storage for information on installing and configuring NooBaa and the Red Hat OpenShift Data Foundations Operator.
If you are using a separate storage solution, set objectstorage
as unmanaged
when configuring the Operator. See the following section. Unmanaged storage, for details of configuring existing storage.
3.2.1. Using unmanaged storage
This section provides configuration examples for unmanaged storage for your convenience. Refer to the Red Hat Quay configuration guide for complete instructions on how to set up object storage.
3.2.1.1. AWS S3 storage
Use the following example when configuring AWS S3 storage for your Red Hat Quay deployment.
DISTRIBUTED_STORAGE_CONFIG: s3Storage: - S3Storage - host: s3.us-east-2.amazonaws.com s3_access_key: ABCDEFGHIJKLMN s3_secret_key: OL3ABCDEFGHIJKLMN s3_bucket: quay_bucket s3_region: <region> storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: - s3Storage
3.2.1.2. Google Cloud storage
Use the following example when configuring Google Cloud storage for your Red Hat Quay deployment.
DISTRIBUTED_STORAGE_CONFIG:
googleCloudStorage:
- GoogleCloudStorage
- access_key: GOOGQIMFB3ABCDEFGHIJKLMN
bucket_name: quay-bucket
secret_key: FhDAYe2HeuAKfvZCAGyOioNaaRABCDEFGHIJKLMN
storage_path: /datastorage/registry
boto_timeout: 120 1
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
- googleCloudStorage
- 1
- Optional. The time, in seconds, until a timeout exception is thrown when attempting to read from a connection. The default is
60
seconds. Also encompasses the time, in seconds, until a timeout exception is thrown when attempting to make a connection. The default is60
seconds.
3.2.1.3. Microsoft Azure storage
Use the following example when configuring Microsoft Azure storage for your Red Hat Quay deployment.
DISTRIBUTED_STORAGE_CONFIG:
azureStorage:
- AzureStorage
- azure_account_name: azure_account_name_here
azure_container: azure_container_here
storage_path: /datastorage/registry
azure_account_key: azure_account_key_here
sas_token: some/path/
endpoint_url: https://[account-name].blob.core.usgovcloudapi.net 1
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
- azureStorage
- 1
- The
endpoint_url
parameter for Microsoft Azure storage is optional and can be used with Microsoft Azure Government (MAG) endpoints. If left blank, theendpoint_url
will connect to the normal Microsoft Azure region.As of Red Hat Quay 3.7, you must use the Primary endpoint of your MAG Blob service. Using the Secondary endpoint of your MAG Blob service will result in the following error:
AuthenticationErrorDetail:Cannot find the claimed account when trying to GetProperties for the account whusc8-secondary
.
3.2.1.4. Ceph/RadosGW Storage
Use the following example when configuring Ceph/RadosGW storage for your Red Hat Quay deployment.
DISTRIBUTED_STORAGE_CONFIG: radosGWStorage: #storage config name - RadosGWStorage #actual driver - access_key: access_key_here #parameters secret_key: secret_key_here bucket_name: bucket_name_here hostname: hostname_here is_secure: 'true' port: '443' storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: #must contain name of the storage config - radosGWStorage
3.2.1.5. Swift storage
Use the following example when configuring Swift storage for your Red Hat Quay deployment.
DISTRIBUTED_STORAGE_CONFIG: swiftStorage: - SwiftStorage - swift_user: swift_user_here swift_password: swift_password_here swift_container: swift_container_here auth_url: https://example.org/swift/v1/quay auth_version: 1 ca_cert_path: /conf/stack/swift.cert" storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: - swiftStorage
3.2.1.6. NooBaa unmanaged storage
Use the following procedure to deploy NooBaa as your unmanaged storage configuration.
Procedure
- Create a NooBaa Object Bucket Claim in the Red Hat Quay console by navigating to Storage → Object Bucket Claims.
- Retrieve the Object Bucket Claim Data details, including the Access Key, Bucket Name, Endpoint (hostname), and Secret Key.
Create a
config.yaml
configuration file that uses the information for the Object Bucket Claim:DISTRIBUTED_STORAGE_CONFIG: default: - RHOCSStorage - access_key: WmrXtSGk8B3nABCDEFGH bucket_name: my-noobaa-bucket-claim-8b844191-dc6c-444e-9ea4-87ece0abcdef hostname: s3.openshift-storage.svc.cluster.local is_secure: true port: "443" secret_key: X9P5SDGJtmSuHFCMSLMbdNCMfUABCDEFGH+C5QD storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: - default
For more information about configuring an Object Bucket Claim, see Object Bucket Claim.
3.2.2. Using an unmanaged NooBaa instance
Use the following procedure to use an unmanaged NooBaa instance for your Red Hat Quay deployment.
Procedure
- Create a NooBaa Object Bucket Claim in the console at Storage → Object Bucket Claims.
-
Retrieve the Object Bucket Claim Data details including the
Access Key
,Bucket Name
,Endpoint (hostname)
, andSecret Key
. Create a
config.yaml
configuration file using the information for the Object Bucket Claim. For example:DISTRIBUTED_STORAGE_CONFIG: default: - RHOCSStorage - access_key: WmrXtSGk8B3nABCDEFGH bucket_name: my-noobaa-bucket-claim-8b844191-dc6c-444e-9ea4-87ece0abcdef hostname: s3.openshift-storage.svc.cluster.local is_secure: true port: "443" secret_key: X9P5SDGJtmSuHFCMSLMbdNCMfUABCDEFGH+C5QD storage_path: /datastorage/registry DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: [] DISTRIBUTED_STORAGE_PREFERENCE: - default
3.2.3. Managed storage
If you want the Red Hat Quay Operator to manage object storage for Red Hat Quay, your cluster needs to be capable of providing object storage through the ObjectBucketClaim
API. Using the Red Hat OpenShift Data Foundation Operator, there are two supported options available:
A standalone instance of the Multi-Cloud Object Gateway backed by a local Kubernetes
PersistentVolume
storage- Not highly available
- Included in the Red Hat Quay subscription
- Does not require a separate subscription for Red Hat OpenShift Data Foundation
A production deployment of Red Hat OpenShift Data Foundation with scale-out Object Service and Ceph
- Highly available
- Requires a separate subscription for Red Hat OpenShift Data Foundation
To use the standalone instance option, continue reading below. For production deployment of Red Hat OpenShift Data Foundation, please refer to the official documentation.
Object storage disk space is allocated automatically by the Red Hat Quay Operator with 50 GiB. This number represents a usable amount of storage for most small to medium Red Hat Quay installations but might not be sufficient for your use cases. Resizing the Red Hat OpenShift Data Foundation volume is currently not handled by the Red Hat Quay Operator. See the section below about resizing managed storage for more details.
3.2.3.1. Leveraging the Multicloud Object Gateway Component in the Red Hat OpenShift Data Foundation Operator for Red Hat Quay
As part of a Red Hat Quay subscription, users are entitled to use the Multicloud Object Gateway component of the Red Hat OpenShift Data Foundation Operator (formerly known as OpenShift Container Storage Operator). This gateway component allows you to provide an S3-compatible object storage interface to Red Hat Quay backed by Kubernetes PersistentVolume
-based block storage. The usage is limited to a Red Hat Quay deployment managed by the Operator and to the exact specifications of the multicloud Object Gateway instance as documented below.
Since Red Hat Quay does not support local filesystem storage, users can leverage the gateway in combination with Kubernetes PersistentVolume
storage instead, to provide a supported deployment. A PersistentVolume
is directly mounted on the gateway instance as a backing store for object storage and any block-based StorageClass
is supported.
By the nature of PersistentVolume
, this is not a scale-out, highly available solution and does not replace a scale-out storage system like Red Hat OpenShift Data Foundation. Only a single instance of the gateway is running. If the pod running the gateway becomes unavailable due to rescheduling, updates or unplanned downtime, this will cause temporary degradation of the connected Red Hat Quay instances.
Using the following procedures, you will install the Local Storage Operator, Red Hat OpenShift Data Foundation, and create a standalone Multicloud Object Gateway to deploy Red Hat Quay on OpenShift Container Platform.
The following documentation shares commonality with the official Red Hat OpenShift Data Foundation documentation.
3.2.3.1.1. Installing the Local Storage Operator on OpenShift Container Platform
Use the following procedure to install the Local Storage Operator from the OperatorHub before creating Red Hat OpenShift Data Foundation clusters on local storage devices.
- Log in to the OpenShift Web Console.
- Click Operators → OperatorHub.
- Type local storage into the search box to find the Local Storage Operator from the list of Operators. Click Local Storage.
- Click Install.
Set the following options on the Install Operator page:
- For Update channel, select stable.
- For Installation mode, select A specific namespace on the cluster.
- For Installed Namespace, select Operator recommended namespace openshift-local-storage.
- For Update approval, select Automatic.
- Click Install.
3.2.3.1.2. Installing Red Hat OpenShift Data Foundation on OpenShift Container Platform
Use the following procedure to install Red Hat OpenShift Data Foundation on OpenShift Container Platform.
Prerequisites
-
Access to an OpenShift Container Platform cluster using an account with
cluster-admin
and Operator installation permissions. - You must have at least three worker nodes in the OpenShift Container Platform cluster.
- For additional resource requirements, see the Planning your deployment guide.
Procedure
- Log in to the OpenShift Web Console.
- Click Operators → OperatorHub.
- Type OpenShift Data Foundation in the search box. Click OpenShift Data Foundation.
- Click Install.
Set the following options on the Install Operator page:
- For Update channel, select the most recent stable version.
- For Installation mode, select A specific namespace on the cluster.
- For Installed Namespace, select Operator recommended Namespace: openshift-storage.
For Update approval, select Automatic or Manual.
If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.
If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.
- For Console plugin, select Enable.
Click Install.
After the Operator is installed, a pop-up with a message,
Web console update is available
appears on the user interface. Click Refresh web console from this pop-up for the console changes to reflect.- Continue to the following section, "Creating a standalone Multicloud Object Gateway", to leverage the Multicloud Object Gateway Component for Red Hat Quay.
3.2.3.1.3. Creating a standalone Multicloud Object Gateway using the OpenShift Container Platform UI
Use the following procedure to create a standalone Multicloud Object Gateway.
Prerequisites
- You have installed the Local Storage Operator.
- You have installed the Red Hat OpenShift Data Foundation Operator.
Procedure
In the OpenShift Web Console, click Operators → Installed Operators to view all installed Operators.
Ensure that the namespace is
openshift-storage
.- Click Create StorageSystem.
On the Backing storage page, select the following:
- Select Multicloud Object Gateway for Deployment type.
- Select the Create a new StorageClass using the local storage devices option.
Click Next.
NoteYou are prompted to install the Local Storage Operator if it is not already installed. Click Install, and follow the procedure as described in "Installing the Local Storage Operator on OpenShift Container Platform".
On the Create local volume set page, provide the following information:
- Enter a name for the LocalVolumeSet and the StorageClass. By default, the local volume set name appears for the storage class name. You can change the name.
Choose one of the following:
Disk on all nodes
Uses the available disks that match the selected filters on all the nodes.
Disk on selected nodes
Uses the available disks that match the selected filters only on the selected nodes.
- From the available list of Disk Type, select SSD/NVMe.
Expand the Advanced section and set the following options:
Volume Mode
Filesystem is selected by default. Always ensure that Filesystem is selected for Volume Mode.
Device Type
Select one or more device type from the dropdown list.
Disk Size
Set a minimum size of 100GB for the device and maximum available size of the device that needs to be included.
Maximum Disks Limit
This indicates the maximum number of PVs that can be created on a node. If this field is left empty, then PVs are created for all the available disks on the matching nodes.
Click Next
A pop-up to confirm the creation of
LocalVolumeSet
is displayed.- Click Yes to continue.
In the Capacity and nodes page, configure the following:
- Available raw capacity is populated with the capacity value based on all the attached disks associated with the storage class. This takes some time to show up. The Selected nodes list shows the nodes based on the storage class.
- Click Next to continue.
Optional. Select the Connect to an external key management service checkbox. This is optional for cluster-wide encryption.
- From the Key Management Service Provider drop-down list, either select Vault or Thales CipherTrust Manager (using KMIP). If you selected Vault, go to the next step. If you selected Thales CipherTrust Manager (using KMIP), go to step iii.
Select an Authentication Method.
Using Token Authentication method
- Enter a unique Connection Name, host Address of the Vault server ('https://<hostname or ip>'), Port number and Token.
Expand Advanced Settings to enter additional settings and certificate details based on your
Vault
configuration:- Enter the Key Value secret path in Backend Path that is dedicated and unique to OpenShift Data Foundation.
- Optional: Enter TLS Server Name and Vault Enterprise Namespace.
- Upload the respective PEM encoded certificate file to provide the CA Certificate, Client Certificate, and Client Private Key.
Click Save and skip to step iv.
Using Kubernetes authentication method
- Enter a unique Vault Connection Name, host Address of the Vault server ('https://<hostname or ip>'), Port number and Role name.
Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:
- Enter the Key Value secret path in Backend Path that is dedicated and unique to Red Hat OpenShift Data Foundation.
- Optional: Enter TLS Server Name and Authentication Path if applicable.
- Upload the respective PEM encoded certificate file to provide the CA Certificate, Client Certificate, and Client Private Key.
- Click Save and skip to step iv.
To use Thales CipherTrust Manager (using KMIP) as the KMS provider, follow the steps below:
- Enter a unique Connection Name for the Key Management service within the project.
In the Address and Port sections, enter the IP of Thales CipherTrust Manager and the port where the KMIP interface is enabled. For example:
- Address: 123.34.3.2
- Port: 5696
- Upload the Client Certificate, CA certificate, and Client Private Key.
- If StorageClass encryption is enabled, enter the Unique Identifier to be used for encryption and decryption generated above.
-
The TLS Server field is optional and used when there is no DNS entry for the KMIP endpoint. For example,
kmip_all_<port>.ciphertrustmanager.local
.
- Select a Network.
- Click Next.
- In the Review and create page, review the configuration details. To modify any configuration settings, click Back.
- Click Create StorageSystem.
3.2.3.1.4. Create A standalone Multicloud Object Gateway using the CLI
Use the following procedure to install the Red Hat OpenShift Data Foundation (formerly known as OpenShift Container Storage) Operator and configure a single instance Multi-Cloud Gateway service.
The following configuration cannot be run in parallel on a cluster with Red Hat OpenShift Data Foundation installed.
Procedure
- On the OpenShift Web Console, and then select Operators → OperatorHub.
- Search for Red Hat OpenShift Data Foundation, and then select Install.
- Accept all default options, and then select Install.
Confirm that the Operator has installed by viewing the Status column, which should be marked as Succeeded.
WarningWhen the installation of the Red Hat OpenShift Data Foundation Operator is finished, you are prompted to create a storage system. Do not follow this instruction. Instead, create NooBaa object storage as outlined the following steps.
On your machine, create a file named
noobaa.yaml
with the following information:apiVersion: noobaa.io/v1alpha1 kind: NooBaa metadata: name: noobaa namespace: openshift-storage spec: dbResources: requests: cpu: '0.1' memory: 1Gi dbType: postgres coreResources: requests: cpu: '0.1' memory: 1Gi
This creates a single instance deployment of the Multi-cloud Object Gateway.
Apply the configuration with the following command:
$ oc create -n openshift-storage -f noobaa.yaml
Example output
noobaa.noobaa.io/noobaa created
After a few minutes, the Multi-cloud Object Gateway should finish provisioning. You can enter the following command to check its status:
$ oc get -n openshift-storage noobaas noobaa -w
Example output
NAME MGMT-ENDPOINTS S3-ENDPOINTS IMAGE PHASE AGE noobaa [https://10.0.32.3:30318] [https://10.0.32.3:31958] registry.redhat.io/ocs4/mcg-core-rhel8@sha256:56624aa7dd4ca178c1887343c7445a9425a841600b1309f6deace37ce6b8678d Ready 3d18h
Configure a backing store for the gateway by creating the following YAML file, named
noobaa-pv-backing-store.yaml
:apiVersion: noobaa.io/v1alpha1 kind: BackingStore metadata: finalizers: - noobaa.io/finalizer labels: app: noobaa name: noobaa-pv-backing-store namespace: openshift-storage spec: pvPool: numVolumes: 1 resources: requests: storage: 50Gi 1 storageClass: STORAGE-CLASS-NAME 2 type: pv-pool
Enter the following command to apply the configuration:
$ oc create -f noobaa-pv-backing-store.yaml
Example output
backingstore.noobaa.io/noobaa-pv-backing-store created
This creates the backing store configuration for the gateway. All images in Red Hat Quay will be stored as objects through the gateway in a
PersistentVolume
created by the above configuration.Run the following command to make the
PersistentVolume
backing store the default for allObjectBucketClaims
issued by the Red Hat Quay Operator:$ oc patch bucketclass noobaa-default-bucket-class --patch '{"spec":{"placementPolicy":{"tiers":[{"backingStores":["noobaa-pv-backing-store"]}]}}}' --type merge -n openshift-storage
Chapter 4. Configuring traffic ingress
4.1. Configuring SSL/TLS and Routes
Support for OpenShift Container Platform edge termination routes have been added by way of a new managed component, tls
. This separates the route
component from SSL/TLS and allows users to configure both separately.
EXTERNAL_TLS_TERMINATION: true
is the opinionated setting.
-
Managed
tls
means that the default cluster wildcard certificate is used. -
Unmanaged
tls
means that the user provided key and certificate pair is be injected into the route.
The ssl.cert
and ssl.key
are now moved to a separate, persistent secret, which ensures that the key and certificate pair are not regenerated upon every reconcile. The key and certificate pair are now formatted as edge
routes and mounted to the same directory in the Quay
container.
Multiple permutations are possible when configuring SSL/TLS and routes, but the following rules apply:
-
If SSL/TLS is
managed
, then your route must also bemanaged
. -
If SSL/TLS is
unmanaged
then you must supply certificates directly in the config bundle.
The following table describes the valid options:
Option | Route | TLS | Certs provided | Result |
---|---|---|---|---|
My own load balancer handles TLS | Managed | Managed | No | Edge route with default wildcard cert |
Red Hat Quay handles TLS | Managed | Unmanaged | Yes | Passthrough route with certs mounted inside the pod |
Red Hat Quay handles TLS | Unmanaged | Unmanaged | Yes |
Certificates are set inside of the |
4.1.1. Creating the config bundle secret with the SSL/TLS cert and key pair
Use the following procedure to create a config bundle secret that includes your own SSL/TLS certificate and key pair.
Procedure
Enter the following command to create config bundle secret that includes your own SSL/TLS certificate and key pair:
$ oc create secret generic --from-file config.yaml=./config.yaml --from-file ssl.cert=./ssl.cert --from-file ssl.key=./ssl.key config-bundle-secret
Chapter 5. Configuring resources for managed components on OpenShift Container Platform
You can manually adjust the resources on Red Hat Quay on OpenShift Container Platform for the following components that have running pods:
-
quay
-
clair
-
mirroring
-
clairpostgres
-
postgres
This feature allows users to run smaller test clusters, or to request more resources upfront in order to avoid partially degraded Quay
pods. Limitations and requests can be set in accordance with Kubernetes resource units.
The following components should not be set lower than their minimum requirements. This can cause issues with your deployment and, in some cases, result in failure of the pod’s deployment.
-
quay
: Minimum of 6 GB, 2vCPUs -
clair
: Recommended of 2 GB memory, 2 vCPUs -
clairpostgres
: Minimum of 200 MB
You can configure resource requests on the OpenShift Container Platform UI, or by directly by updating the QuayRegistry
YAML.
The default values set for these components are the suggested values. Setting resource requests too high or too low might lead to inefficient resource utilization, or performance degradation, respectively.
5.1. Configuring resource requests by using the OpenShift Container Platform UI
Use the following procedure to configure resources by using the OpenShift Container Platform UI.
Procedure
- On the OpenShift Container Platform developer console, click Operators → Installed Operators → Red Hat Quay.
- Click QuayRegistry.
- Click the name of your registry, for example, example-registry.
- Click YAML.
In the
spec.components
field, you can override the resource of thequay
,clair
,mirroring
clairpostgres
, andpostgres
resources by setting values for the.overrides.resources.limits
and theoverrides.resources.requests
fields. For example:spec: components: - kind: clair managed: true overrides: resources: limits: cpu: "5" # Limiting to 5 CPU (equivalent to 5000m or 5000 millicpu) memory: "18Gi" # Limiting to 18 Gibibytes of memory requests: cpu: "4" # Requesting 4 CPU memory: "4Gi" # Requesting 4 Gibibytes of memory - kind: postgres managed: true overrides: resources: limits: {} 1 requests: cpu: "700m" # Requesting 700 millicpu or 0.7 CPU memory: "4Gi" # Requesting 4 Gibibytes of memory - kind: mirror managed: true overrides: resources: limits: 2 requests: cpu: "800m" # Requesting 800 millicpu or 0.8 CPU memory: "1Gi" # Requesting 1 Gibibyte of memory - kind: quay managed: true overrides: resources: limits: cpu: "4" # Limiting to 4 CPU memory: "10Gi" # Limiting to 10 Gibibytes of memory requests: cpu: "4" # Requesting 4 CPU memory: "10Gi" # Requesting 10 Gibi of memory - kind: clairpostgres managed: true overrides: resources: limits: cpu: "800m" # Limiting to 800 millicpu or 0.8 CPU memory: "3Gi" # Limiting to 3 Gibibytes of memory requests: {}
5.2. Configuring resource requests by editing the QuayRegistry YAML
You can re-configure Red Hat Quay to configure resource requests after you have already deployed a registry. This can be done by editing the QuayRegistry
YAML file directly and then re-deploying the registry.
Procedure
Optional: If you do not have a local copy of the
QuayRegistry
YAML file, enter the following command to obtain it:$ oc get quayregistry <registry_name> -n <namespace> -o yaml > quayregistry.yaml
Open the
quayregistry.yaml
created from Step 1 of this procedure and make the desired changes. For example:- kind: quay managed: true overrides: resources: limits: {} requests: cpu: "0.7" # Requesting 0.7 CPU (equivalent to 500m or 500 millicpu) memory: "512Mi" # Requesting 512 Mebibytes of memory
- Save the changes.
Apply the Red Hat Quay registry using the updated configurations by running the following command:
$ oc replace -f quayregistry.yaml
Example output
quayregistry.quay.redhat.com/example-registry replaced
Chapter 6. Configuring the database
6.1. Using an existing PostgreSQL database
If you are using an externally managed PostgreSQL database, you must manually enable the pg_trgm
extension for a successful deployment.
You must not use the same externally managed PostgreSQL database for both Red Hat Quay and Clair deployments. Your PostgreSQL database must also not be shared with other workloads, as it might exhaust the natural connection limit on the PostgreSQL side when connection-intensive workloads, like Red Hat Quay or Clair, contend for resources. Additionally, pgBouncer is not supported with Red Hat Quay or Clair, so it is not an option to resolve this issue.
Use the following procedure to deploy an existing PostgreSQL database.
Procedure
Create a
config.yaml
file with the necessary database fields. For example:Example
config.yaml
file:DB_URI: postgresql://test-quay-database:postgres@test-quay-database:5432/test-quay-database
Create a
Secret
using the configuration file:$ kubectl create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
Create a
QuayRegistry.yaml
file which marks thepostgres
component asunmanaged
and references the createdSecret
. For example:Example
quayregistry.yaml
fileapiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: configBundleSecret: config-bundle-secret components: - kind: postgres managed: false
Next steps
- Continue to the following sections to deploy the registry.
6.1.1. Database configuration
This section describes the database configuration fields available for Red Hat Quay deployments.
6.1.1.1. Database URI
With Red Hat Quay, connection to the database is configured by using the required DB_URI
field.
The following table describes the DB_URI
configuration field:
Field | Type | Description |
---|---|---|
DB_URI | String | The URI for accessing the database, including any credentials.
Example postgresql://quayuser:quaypass@quay-server.example.com:5432/quay |
6.1.1.2. Database connection arguments
Optional connection arguments are configured by the DB_CONNECTION_ARGS
parameter. Some of the key-value pairs defined under DB_CONNECTION_ARGS
are generic, while others are database specific.
The following table describes database connection arguments:
Field | Type | Description |
---|---|---|
DB_CONNECTION_ARGS | Object | Optional connection arguments for the database, such as timeouts and SSL/TLS. |
.autorollback | Boolean |
Whether to use thread-local connections. |
.threadlocals | Boolean |
Whether to use auto-rollback connections. |
6.1.1.2.1. PostgreSQL SSL/TLS connection arguments
With SSL/TLS, configuration depends on the database you are deploying. The following example shows a PostgreSQL SSL/TLS configuration:
DB_CONNECTION_ARGS: sslmode: verify-ca sslrootcert: /path/to/cacert
The sslmode
option determines whether, or with, what priority a secure SSL/TLS TCP/IP connection will be negotiated with the server. There are six modes:
Mode | Description |
---|---|
disable | Your configuration only tries non-SSL/TLS connections. |
allow | Your configuration first tries a non-SSL/TLS connection. Upon failure, tries an SSL/TLS connection. |
prefer | Your configuration first tries an SSL/TLS connection. Upon failure, tries a non-SSL/TLS connection. |
require | Your configuration only tries an SSL/TLS connection. If a root CA file is present, it verifies the certificate in the same way as if verify-ca was specified. |
verify-ca | Your configuration only tries an SSL/TLS connection, and verifies that the server certificate is issued by a trusted certificate authority (CA). |
verify-full | Only tries an SSL/TLS connection, and verifies that the server certificate is issued by a trusted CA and that the requested server hostname matches that in the certificate. |
For more information on the valid arguments for PostgreSQL, see Database Connection Control Functions.
6.1.1.2.2. MySQL SSL/TLS connection arguments
The following example shows a sample MySQL SSL/TLS configuration:
DB_CONNECTION_ARGS: ssl: ca: /path/to/cacert
Information on the valid connection arguments for MySQL is available at Connecting to the Server Using URI-Like Strings or Key-Value Pairs.
6.1.2. Using the managed PostgreSQL database
With Red Hat Quay 3.9, if your database is managed by the Red Hat Quay Operator, updating from Red Hat Quay 3.8 → 3.9 automatically handles upgrading PostgreSQL 10 to PostgreSQL 13.
- Users with a managed database are required to upgrade their PostgreSQL database from 10 → 13.
- If your Red Hat Quay and Clair databases are managed by the Operator, the database upgrades for each component must succeed for the 3.9.0 upgrade to be successful. If either of the database upgrades fail, the entire Red Hat Quay version upgrade fails. This behavior is expected.
If you do not want the Red Hat Quay Operator to upgrade your PostgreSQL deployment from PostgreSQL 10 → 13, you must set the PostgreSQL parameter to managed: false
in your quayregistry.yaml
file. For more information about setting your database to unmanaged, see Using an existing Postgres database.
- It is highly recommended that you upgrade to PostgreSQL 13. PostgreSQL 10 had its final release on November 10, 2022 and is no longer supported. For more information, see the PostgreSQL Versioning Policy.
If you want your PostgreSQL database to match the same version as your Red Hat Enterprise Linux (RHEL) system, see Migrating to a RHEL 8 version of PostgreSQL for RHEL 8 or Migrating to a RHEL 9 version of PostgreSQL for RHEL 9.
For more information about the Red Hat Quay 3.8 → 3.9 procedure, see Upgrading the Red Hat Quay Operator overview.
6.1.2.1. PostgreSQL database recommendations
The Red Hat Quay team recommends the following for managing your PostgreSQL database.
- Database backups should be performed regularly using either the supplied tools on the PostgreSQL image or your own backup infrastructure. The Red Hat Quay Operator does not currently ensure that the PostgreSQL database is backed up.
-
Restoring the PostgreSQL database from a backup must be done using PostgreSQL tools and procedures. Be aware that your
Quay
pods should not be running while the database restore is in progress. - Database disk space is allocated automatically by the Red Hat Quay Operator with 50 GiB. This number represents a usable amount of storage for most small to medium Red Hat Quay installations but might not be sufficient for your use cases. Resizing the database volume is currently not handled by the Red Hat Quay Operator.
6.2. Configuring external Redis
Use the content in this section to set up an external Redis deployment.
6.2.1. Using an unmanaged Redis database
Use the following procedure to set up an external Redis database.
Procedure
Create a
config.yaml
file using the following Redis fields:# ... BUILDLOGS_REDIS: host: <quay-server.example.com> port: 6379 ssl: false # ... USER_EVENTS_REDIS: host: <quay-server.example.com> port: 6379 ssl: false # ...
Enter the following command to create a secret using the configuration file:
$ oc create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
Create a
quayregistry.yaml
file that sets the Redis component tounmanaged
and references the created secret:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: configBundleSecret: config-bundle-secret components: - kind: redis managed: false # ...
- Deploy the Red Hat Quay registry.
Additional resources
6.2.2. Using unmanaged Horizontal Pod Autoscalers
Horizontal Pod Autoscalers (HPAs) are now included with the Clair
, Quay
, and Mirror
pods, so that they now automatically scale during load spikes.
As HPA is configured by default to be managed, the number of Clair
, Quay
, and Mirror
pods is set to two. This facilitates the avoidance of downtime when updating or reconfiguring Red Hat Quay through the Operator or during rescheduling events.
6.2.2.1. Disabling the Horizontal Pod Autoscaler
To disable autoscaling or create your own HorizontalPodAutoscaler
, specify the component as unmanaged
in the QuayRegistry
instance. For example:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: components: - kind: horizontalpodautoscaler managed: false # ...
6.2.3. Disabling the Route component
Use the following procedure to prevent the Red Hat Quay Operator from creating a route.
Procedure
Set the component as
managed: false
in thequayregistry.yaml
file:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: components: - kind: route managed: false
Edit the
config.yaml
file to specify that Red Hat Quay handles SSL/TLS. For example:# ... EXTERNAL_TLS_TERMINATION: false # ... SERVER_HOSTNAME: example-registry-quay-quay-enterprise.apps.user1.example.com # ... PREFERRED_URL_SCHEME: https # ...
If you do not configure the unmanaged route correctly, the following error is returned:
{ { "kind":"QuayRegistry", "namespace":"quay-enterprise", "name":"example-registry", "uid":"d5879ba5-cc92-406c-ba62-8b19cf56d4aa", "apiVersion":"quay.redhat.com/v1", "resourceVersion":"2418527" }, "reason":"ConfigInvalid", "message":"required component `route` marked as unmanaged, but `configBundleSecret` is missing necessary fields" }
Disabling the default route means you are now responsible for creating a Route
, Service
, or Ingress
in order to access the Red Hat Quay instance. Additionally, whatever DNS you use must match the SERVER_HOSTNAME
in the Red Hat Quay config.
6.2.4. Disabling the monitoring component
If you install the Red Hat Quay Operator in a single namespace, the monitoring component is automatically set to managed: false
. Use the following reference to explicitly disable monitoring.
Unmanaged monitoring
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: components: - kind: monitoring managed: false
To enable monitoring in this scenario, see Enabling monitoring when the Red Hat Quay Operator is installed in a single namespace.
6.2.5. Disabling the mirroring component
To disable mirroring, use the following YAML configuration:
Unmanaged mirroring example YAML configuration
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: components: - kind: mirroring managed: false
Chapter 7. Deploying Red Hat Quay using the Operator
Red Hat Quay on OpenShift Container Platform can be deployed using command-line interface or from the OpenShift Container Platform console. The steps are fundamentally the same.
7.1. Deploying Red Hat Quay from the command line
Use the following procedure to deploy Red Hat Quay from using the command-line interface (CLI).
Prerequisites
- You have logged into OpenShift Container Platform using the CLI.
Procedure
Create a namespace, for example,
quay-enterprise
, by entering the following command:$ oc new-project quay-enterprise
Optional. If you want to pre-configure any aspects of your Red Hat Quay deployment, create a
Secret
for the config bundle:$ oc create secret generic quay-enterprise-config-bundle --from-file=config-bundle.tar.gz=/path/to/config-bundle.tar.gz
Create a
QuayRegistry
custom resource in a file calledquayregistry.yaml
For a minimal deployment, using all the defaults:
quayregistry.yaml:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise
Optional. If you want to have some components unmanaged, add this information in the
spec
field. A minimal deployment might look like the following example:Example quayregistry.yaml with unmanaged components
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: components: - kind: clair managed: false - kind: horizontalpodautoscaler managed: false - kind: mirror managed: false - kind: monitoring managed: false
Optional. If you have created a config bundle, for example,
init-config-bundle-secret
, reference it in thequayregistry.yaml
file:Example quayregistry.yaml with a config bundle
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: example-registry namespace: quay-enterprise spec: configBundleSecret: init-config-bundle-secret
Optional. If you have a proxy configured, you can add the information using overrides for Red Hat Quay, Clair, and mirroring:
Example quayregistry.yaml with proxy configured
kind: QuayRegistry metadata: name: quay37 spec: configBundleSecret: config-bundle-secret components: - kind: objectstorage managed: false - kind: route managed: true - kind: mirror managed: true overrides: env: - name: DEBUGLOG value: "true" - name: HTTP_PROXY value: quayproxy.qe.devcluster.openshift.com:3128 - name: HTTPS_PROXY value: quayproxy.qe.devcluster.openshift.com:3128 - name: NO_PROXY value: svc.cluster.local,localhost,quay370.apps.quayperf370.perfscale.devcluster.openshift.com - kind: tls managed: false - kind: clair managed: true overrides: env: - name: HTTP_PROXY value: quayproxy.qe.devcluster.openshift.com:3128 - name: HTTPS_PROXY value: quayproxy.qe.devcluster.openshift.com:3128 - name: NO_PROXY value: svc.cluster.local,localhost,quay370.apps.quayperf370.perfscale.devcluster.openshift.com - kind: quay managed: true overrides: env: - name: DEBUGLOG value: "true" - name: NO_PROXY value: svc.cluster.local,localhost,quay370.apps.quayperf370.perfscale.devcluster.openshift.com - name: HTTP_PROXY value: quayproxy.qe.devcluster.openshift.com:3128 - name: HTTPS_PROXY value: quayproxy.qe.devcluster.openshift.com:3128
Create the
QuayRegistry
in the specified namespace by entering the following command:$ oc create -n quay-enterprise -f quayregistry.yaml
Enter the following command to see when the
status.registryEndpoint
is populated:$ oc get quayregistry -n quay-enterprise example-registry -o jsonpath="{.status.registryEndpoint}" -w
Additional resources
- For more information about how to track the progress of your Red Hat Quay deployment, see Monitoring and debugging the deployment process.
7.1.1. Using the API to create the first user
Use the following procedure to create the first user in your Red Hat Quay organization.
Prerequisites
-
The config option
FEATURE_USER_INITIALIZE
must be set totrue
. - No users can already exist in the database.
This procedure requests an OAuth token by specifying "access_token": true
.
Open your Red Hat Quay configuration file and update the following configuration fields:
FEATURE_USER_INITIALIZE: true SUPER_USERS: - quayadmin
Stop the Red Hat Quay service by entering the following command:
$ sudo podman stop quay
Start the Red Hat Quay service by entering the following command:
$ sudo podman run -d -p 80:8080 -p 443:8443 --name=quay -v $QUAY/config:/conf/stack:Z -v $QUAY/storage:/datastorage:Z {productrepo}/{quayimage}:{productminv}
Run the following
CURL
command to generate a new user with a username, password, email, and access token:$ curl -X POST -k http://quay-server.example.com/api/v1/user/initialize --header 'Content-Type: application/json' --data '{ "username": "quayadmin", "password":"quaypass12345", "email": "quayadmin@example.com", "access_token": true}'
If successful, the command returns an object with the username, email, and encrypted password. For example:
{"access_token":"6B4QTRSTSD1HMIG915VPX7BMEZBVB9GPNY2FC2ED", "email":"quayadmin@example.com","encrypted_password":"1nZMLH57RIE5UGdL/yYpDOHLqiNCgimb6W9kfF8MjZ1xrfDpRyRs9NUnUuNuAitW","username":"quayadmin"} # gitleaks:allow
If a user already exists in the database, an error is returned:
{"message":"Cannot initialize user in a non-empty database"}
If your password is not at least eight characters or contains whitespace, an error is returned:
{"message":"Failed to initialize user: Invalid password, password must be at least 8 characters and contain no whitespace."}
Log in to your Red Hat Quay deployment by entering the following command:
$ sudo podman login -u quayadmin -p quaypass12345 http://quay-server.example.com --tls-verify=false
Example output
Login Succeeded!
7.1.2. Viewing created components using the command line
Use the following procedure to view deployed Red Hat Quay components.
Prerequisites
- You have deployed Red Hat Quay on OpenShift Container Platform.
Procedure
Enter the following command to view the deployed components:
$ oc get pods -n quay-enterprise
Example output
NAME READY STATUS RESTARTS AGE example-registry-clair-app-5ffc9f77d6-jwr9s 1/1 Running 0 3m42s example-registry-clair-app-5ffc9f77d6-wgp7d 1/1 Running 0 3m41s example-registry-clair-postgres-54956d6d9c-rgs8l 1/1 Running 0 3m5s example-registry-quay-app-79c6b86c7b-8qnr2 1/1 Running 4 3m42s example-registry-quay-app-79c6b86c7b-xk85f 1/1 Running 4 3m41s example-registry-quay-app-upgrade-5kl5r 0/1 Completed 4 3m50s example-registry-quay-database-b466fc4d7-tfrnx 1/1 Running 2 3m42s example-registry-quay-mirror-6d9bd78756-6lj6p 1/1 Running 0 2m58s example-registry-quay-mirror-6d9bd78756-bv6gq 1/1 Running 0 2m58s example-registry-quay-postgres-init-dzbmx 0/1 Completed 0 3m43s example-registry-quay-redis-8bd67b647-skgqx 1/1 Running 0 3m42s
7.1.3. Horizontal Pod Autoscaling
A default deployment shows the following running pods:
-
Two pods for the Red Hat Quay application itself (
example-registry-quay-app-*`
) -
One Redis pod for Red Hat Quay logging (
example-registry-quay-redis-*
) -
One database pod for PostgreSQL used by Red Hat Quay for metadata storage (
example-registry-quay-database-*
) -
Two
Quay
mirroring pods (example-registry-quay-mirror-*
) -
Two pods for the Clair application (
example-registry-clair-app-*
) -
One PostgreSQL pod for Clair (
example-registry-clair-postgres-*
)
Horizontal PPod Autoscaling is configured by default to be managed
, and the number of pods for Quay, Clair and repository mirroring is set to two. This facilitates the avoidance of downtime when updating or reconfiguring Red Hat Quay through the Red Hat Quay Operator or during rescheduling events. You can enter the following command to view information about HPA objects:
$ oc get hpa -n quay-enterprise
Example output
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE example-registry-clair-app Deployment/example-registry-clair-app 16%/90%, 0%/90% 2 10 2 13d example-registry-quay-app Deployment/example-registry-quay-app 31%/90%, 1%/90% 2 20 2 13d example-registry-quay-mirror Deployment/example-registry-quay-mirror 27%/90%, 0%/90% 2 20 2 13d
Additional resources
For more information on pre-configuring your Red Hat Quay deployment, see the section Pre-configuring Red Hat Quay for automation
7.1.4. Monitoring and debugging the deployment process
Users can now troubleshoot problems during the deployment phase. The status in the QuayRegistry
object can help you monitor the health of the components during the deployment an help you debug any problems that may arise.
Procedure
Enter the following command to check the status of your deployment:
$ oc get quayregistry -n quay-enterprise -o yaml
Example output
Immediately after deployment, the
QuayRegistry
object will show the basic configuration:apiVersion: v1 items: - apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: creationTimestamp: "2021-09-14T10:51:22Z" generation: 3 name: example-registry namespace: quay-enterprise resourceVersion: "50147" selfLink: /apis/quay.redhat.com/v1/namespaces/quay-enterprise/quayregistries/example-registry uid: e3fc82ba-e716-4646-bb0f-63c26d05e00e spec: components: - kind: postgres managed: true - kind: clair managed: true - kind: redis managed: true - kind: horizontalpodautoscaler managed: true - kind: objectstorage managed: true - kind: route managed: true - kind: mirror managed: true - kind: monitoring managed: true - kind: tls managed: true - kind: clairpostgres managed: true configBundleSecret: example-registry-config-bundle-kt55s kind: List metadata: resourceVersion: "" selfLink: ""
Use the
oc get pods
command to view the current state of the deployed components:$ oc get pods -n quay-enterprise
Example output
NAME READY STATUS RESTARTS AGE example-registry-clair-app-86554c6b49-ds7bl 0/1 ContainerCreating 0 2s example-registry-clair-app-86554c6b49-hxp5s 0/1 Running 1 17s example-registry-clair-postgres-68d8857899-lbc5n 0/1 ContainerCreating 0 17s example-registry-quay-app-upgrade-h2v7h 0/1 ContainerCreating 0 9s example-registry-quay-database-66f495c9bc-wqsjf 0/1 ContainerCreating 0 17s example-registry-quay-mirror-854c88457b-d845g 0/1 Init:0/1 0 2s example-registry-quay-mirror-854c88457b-fghxv 0/1 Init:0/1 0 17s example-registry-quay-postgres-init-bktdt 0/1 Terminating 0 17s example-registry-quay-redis-f9b9d44bf-4htpz 0/1 ContainerCreating 0 17s
While the deployment is in progress, the
QuayRegistry
object will show the current status. In this instance, database migrations are taking place, and other components are waiting until completion:status: conditions: - lastTransitionTime: "2021-09-14T10:52:04Z" lastUpdateTime: "2021-09-14T10:52:04Z" message: all objects created/updated successfully reason: ComponentsCreationSuccess status: "False" type: RolloutBlocked - lastTransitionTime: "2021-09-14T10:52:05Z" lastUpdateTime: "2021-09-14T10:52:05Z" message: running database migrations reason: MigrationsInProgress status: "False" type: Available lastUpdated: 2021-09-14 10:52:05.371425635 +0000 UTC unhealthyComponents: clair: - lastTransitionTime: "2021-09-14T10:51:32Z" lastUpdateTime: "2021-09-14T10:51:32Z" message: 'Deployment example-registry-clair-postgres: Deployment does not have minimum availability.' reason: MinimumReplicasUnavailable status: "False" type: Available - lastTransitionTime: "2021-09-14T10:51:32Z" lastUpdateTime: "2021-09-14T10:51:32Z" message: 'Deployment example-registry-clair-app: Deployment does not have minimum availability.' reason: MinimumReplicasUnavailable status: "False" type: Available mirror: - lastTransitionTime: "2021-09-14T10:51:32Z" lastUpdateTime: "2021-09-14T10:51:32Z" message: 'Deployment example-registry-quay-mirror: Deployment does not have minimum availability.' reason: MinimumReplicasUnavailable status: "False" type: Available
When the deployment process finishes successfully, the status in the
QuayRegistry
object shows no unhealthy components:status: conditions: - lastTransitionTime: "2021-09-14T10:52:36Z" lastUpdateTime: "2021-09-14T10:52:36Z" message: all registry component healthchecks passing reason: HealthChecksPassing status: "True" type: Available - lastTransitionTime: "2021-09-14T10:52:46Z" lastUpdateTime: "2021-09-14T10:52:46Z" message: all objects created/updated successfully reason: ComponentsCreationSuccess status: "False" type: RolloutBlocked currentVersion: {producty} lastUpdated: 2021-09-14 10:52:46.104181633 +0000 UTC registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.quayteam.org unhealthyComponents: {}
7.2. Deploying Red Hat Quay from the OpenShift Container Platform console
-
Create a namespace, for example,
quay-enterprise
. - Select Operators → Installed Operators, then select the Quay Operator to navigate to the Operator detail view.
- Click 'Create Instance' on the 'Quay Registry' tile under 'Provided APIs'.
-
Optionally change the 'Name' of the
QuayRegistry
. This will affect the hostname of the registry. All other fields have been populated with defaults. -
Click 'Create' to submit the
QuayRegistry
to be deployed by the Quay Operator. -
You should be redirected to the
QuayRegistry
list view. Click on theQuayRegistry
you just created to see the details view. - Once the 'Registry Endpoint' has a value, click it to access your new Quay registry via the UI. You can now select 'Create Account' to create a user and sign in.
7.2.1. Using the Red Hat Quay UI to create the first user
Use the following procedure to create the first user by the Red Hat Quay UI.
This procedure assumes that the FEATURE_USER_CREATION
config option has not been set to false.
If it is false
, the Create Account
functionality on the UI will be disabled, and you will have to use the API to create the first user.
Procedure
- In the OpenShift Container Platform console, navigate to Operators → Installed Operators, with the appropriate namespace / project.
Click on the newly installed
QuayRegistry
object to view the details. For example:-
After the
Registry Endpoint
has a value, navigate to this URL in your browser. Select Create Account in the Red Hat Quay registry UI to create a user. For example:
Enter the details for Username, Password, Email, and then click Create Account. For example:
After creating the first user, you are automatically logged in to the Red Hat Quay registry. For example:
Chapter 8. Viewing the status of the QuayRegistry object
Lifecycle observability for a given Red Hat Quay deployment is reported in the status
section of the corresponding QuayRegistry
object. The Red Hat Quay Operator constantly updates this section, and this should be the first place to look for any problems or state changes in Red Hat Quay or its managed dependencies.
8.1. Viewing the registry endpoint
Once Red Hat Quay is ready to be used, the status.registryEndpoint
field will be populated with the publicly available hostname of the registry.
8.2. Viewing the version of Red Hat Quay in use
The current version of Red Hat Quay that is running will be reported in status.currentVersion
.
8.3. Viewing the conditions of your Red Hat Quay deployment
Certain conditions will be reported in status.conditions
.
Chapter 9. Customizing Red Hat Quay on OpenShift Container Platform
After deployment, you can customize the Red Hat Quay application by editing the Red Hat Quay configuration bundle secret spec.configBundleSecret
. You can also change the managed status of components and configure resource requests for some components in the spec.components
object of the QuayRegistry
resource.
9.1. Editing the config bundle secret in the OpenShift Container Platform console
Use the following procedure to edit the config bundle secret in the OpenShift Container Platform console.
Procedure
On the Red Hat Quay Registry overview screen, click the link for the Config Bundle Secret.
To edit the secret, click Actions → Edit Secret.
Modify the configuration and save the changes.
- Monitor the deployment to ensure successful completion and that the configuration changes have taken effect.
9.2. Determining QuayRegistry endpoints and secrets
Use the following procedure to find QuayRegistry
endpoints and secrets.
Procedure
You can examine the
QuayRegistry
resource, usingoc describe quayregistry
oroc get quayregistry -o yaml
, to find the current endpoints and secrets by entering the following command:$ oc get quayregistry example-registry -n quay-enterprise -o yaml
Example output
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: example-registry namespace: quay-enterprise ... spec: components: - kind: quay managed: true ... - kind: clairpostgres managed: true configBundleSecret: init-config-bundle-secret 1 status: currentVersion: 3.7.0 lastUpdated: 2022-05-11 13:28:38.199476938 +0000 UTC registryEndpoint: https://example-registry-quay-quay-enterprise.apps.docs.gcp.quaydev.org 2
9.3. Downloading the existing configuration
The following procedure shows you how to download the existing configuration by locating the Config Bundle Secret
.
Procedure
Describe the
QuayRegistry
resource by entering the following command:$ oc describe quayregistry -n <quay_namespace>
# ... Config Bundle Secret: example-registry-config-bundle-v123x # ...
Obtain the secret data by entering the following command:
$ oc get secret -n <quay_namespace> <example-registry-config-bundle-v123x> -o jsonpath='{.data}'
Example output
{ "config.yaml": "RkVBVFVSRV9VU0 ... MDAwMAo=" }
Decode the data by entering the following command:
$ echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode
Example output
FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_BUILD_SUPPORT: true DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 102400000
Optional. You can export the data into a YAML file into the current directory by passing in the
>> config.yaml
flag. For example:$ echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode >> config.yaml
Chapter 10. Configuring custom SSL/TLS certificates for Red Hat Quay on OpenShift Container Platform
When Red Hat Quay is deployed on OpenShift Container Platform, the tls
component of the QuayRegistry
custom resource definition (CRD) is set to managed
by default. As a result, OpenShift Container Platform’s Certificate Authority is used to create HTTPS endpoints and to rotate SSL/TLS certificates.
You can configure custom SSL/TLS certificates before or after the initial deployment of Red Hat Quay on OpenShift Container Platform. This process involves creating or updating the configBundleSecret
resource within the QuayRegistry
YAML file to integrate your custom certificates and setting the tls
component to unmanaged
.
When configuring custom SSL/TLS certificates for Red Hat Quay, administrators are responsible for certificate rotation.
The following procedures enable you to apply custom SSL/TLS certificates to ensure secure communication and meet specific security requirements for your Red Hat Quay on OpenShift Container Platform deployment. These steps assumed you have already created a Certificate Authority (CA) bundle or an ssl.key
, and an ssl.cert
. The procedure then shows you how to integrate those files into your Red Hat Quay on OpenShift Container Platform deployment, which ensures that your registry operates with the specified security settings and conforms to your organization’s SSL/TLS policies.
-
The following procedure is used for securing Red Hat Quay with an HTTPS certificate. Note that this differs from managing Certificate Authority Trust Bundles. CA Trust Bundles are used by system processes within the
Quay
container to verify certificates against trusted CAs, and ensure that services like LDAP, storage backend, and OIDC connections are trusted. -
If you are adding the certificates to an existing deployment, you must include the existing
config.yaml
file in the new config bundle secret, even if you are not making any configuration changes.
10.1. Creating a Certificate Authority
Use the following procedure to set up your own CA and use it to issue a server certificate for your domain. This allows you to secure communications with SSL/TLS using your own certificates.
Procedure
Generate the root CA key by entering the following command:
$ openssl genrsa -out rootCA.key 2048
Generate the root CA certificate by entering the following command:
$ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem
Enter the information that will be incorporated into your certificate request, including the server hostname, for example:
Country Name (2 letter code) [XX]:IE State or Province Name (full name) []:GALWAY Locality Name (eg, city) [Default City]:GALWAY Organization Name (eg, company) [Default Company Ltd]:QUAY Organizational Unit Name (eg, section) []:DOCS Common Name (eg, your name or your server's hostname) []:quay-server.example.com
Generate the server key by entering the following command:
$ openssl genrsa -out ssl.key 2048
Generate a signing request by entering the following command:
$ openssl req -new -key ssl.key -out ssl.csr
Enter the information that will be incorporated into your certificate request, including the server hostname, for example:
Country Name (2 letter code) [XX]:IE State or Province Name (full name) []:GALWAY Locality Name (eg, city) [Default City]:GALWAY Organization Name (eg, company) [Default Company Ltd]:QUAY Organizational Unit Name (eg, section) []:DOCS Common Name (eg, your name or your server's hostname) []:quay-server.example.com Email Address []:
Create a configuration file
openssl.cnf
, specifying the server hostname, for example:Example
openssl.cnf
file[req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = <quay-server.example.com> IP.1 = 192.168.1.112
Use the configuration file to generate the certificate
ssl.cert
:$ openssl x509 -req -in ssl.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out ssl.cert -days 356 -extensions v3_req -extfile openssl.cnf
Confirm your created certificates and files by entering the following command:
$ ls /path/to/certificates
Example output
rootCA.key ssl-bundle.cert ssl.key custom-ssl-config-bundle-secret.yaml rootCA.pem ssl.cert openssl.cnf rootCA.srl ssl.csr
10.2. Creating a custom SSL/TLS configBundleSecret resource
After creating your custom SSL/TLS certificates, you can create a custom configBundleSecret
resource for Red Hat Quay on OpenShift Container Platform, which allows you to upload ssl.cert
and ssl.key
files.
Prerequisites
-
You have base64 decoded the original config bundle into a
config.yaml
file. For more information, see Downloading the existing configuration. - You have generated custom SSL certificates and keys.
Procedure
Create a new YAML file, for example,
custom-ssl-config-bundle-secret.yaml
:$ touch custom-ssl-config-bundle-secret.yaml
Create the
custom-ssl-config-bundle-secret
resource.Create the resource by entering the following command:
$ oc -n <namespace> create secret generic custom-ssl-config-bundle-secret \ --from-file=config.yaml=</path/to/config.yaml> \ 1 --from-file=ssl.cert=</path/to/ssl.cert> \ 2 --from-file=extra_ca_cert_<name-of-certificate>.crt=ca-certificate-bundle.crt \ 3 --from-file=ssl.key=</path/to/ssl.key> \ 4 --dry-run=client -o yaml > custom-ssl-config-bundle-secret.yaml
- 1
- Where
<config.yaml>
is yourbase64 decoded
config.yaml
file. - 2
- Where
<ssl.cert>
is yourssl.cert
file. - 3
- Optional. The
--from-file=extra_ca_cert_<name-of-certificate>.crt=ca-certificate-bundle.crt
field allows Red Hat Quay to recognize custom Certificate Authority (CA) files. If you are using LDAP, OIDC, or another service that uses custom CAs, you must add them via theextra_ca_cert
path. For more information, see "Adding additional Certificate Authorities to Red Hat Quay on OpenShift Container Platform." - 4
- Where
<ssl.key>
is yourssl.key
file.
Optional. You can check the content of the
custom-ssl-config-bundle-secret.yaml
file by entering the following command:$ cat custom-ssl-config-bundle-secret.yaml
Example output
apiVersion: v1 data: config.yaml: QUxMT1dfUFVMTFNfV0lUSE9VVF9TVFJJQ1RfTE9HR0lORzogZmFsc2UKQVVUSEVOVElDQVRJT05fVFlQRTogRGF0YWJhc2UKREVGQVVMVF9UQUdfRVhQSVJBVElPTjogMncKRElTVFJJQlVURURfU1R... ssl.cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVYakNDQTBhZ0F3SUJBZ0lVTUFBRk1YVWlWVHNoMGxNTWI3U1l0eFV5eTJjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dZZ3hDekFKQmdOVkJBWVR... extra_ca_cert_<name-of-certificate>:LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVYakNDQTBhZ0F3SUJBZ0lVTUFBRk1YVWlWVHNoMGxNTWI3U1l0eFV5eTJjd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2dZZ3hDe... ssl.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ2c0VWxZOVV1SVJPY1oKcFhpZk9MVEdqaS9neUxQMlpiMXQ... kind: Secret metadata: creationTimestamp: null name: custom-ssl-config-bundle-secret namespace: <namespace>
Create the
configBundleSecret
resource by entering the following command:$ oc create -n <namespace> -f custom-ssl-config-bundle-secret.yaml
Example output
secret/custom-ssl-config-bundle-secret created
Update the
QuayRegistry
YAML file to reference thecustom-ssl-config-bundle-secret
object by entering the following command:$ oc patch quayregistry <registry_name> -n <namespace> --type=merge -p '{"spec":{"configBundleSecret":"custom-ssl-config-bundle-secret"}}'
Example output
quayregistry.quay.redhat.com/example-registry patched
Set the
tls
component of theQuayRegistry
YAML tofalse
by entering the following command:$ oc patch quayregistry <registry_name> -n <namespace> --type=merge -p '{"spec":{"components":[{"kind":"tls","managed":false}]}}'
Example output
quayregistry.quay.redhat.com/example-registry patched
Ensure that your
QuayRegistry
YAML file has been updated to use the custom SSLconfigBundleSecret
resource, and that your andtls
resource is set tofalse
by entering the following command:$ oc get quayregistry <registry_name> -n <namespace> -o yaml
Example output
# ... configBundleSecret: custom-ssl-config-bundle-secret # ... spec: components: - kind: tls managed: false # ...
Verification
Confirm a TLS connection to the server and port by entering the following command:
$ openssl s_client -connect <quay-server.example.com>:443
Example output
# ... SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 0E995850DC3A8EB1A838E2FF06CE56DBA81BD8443E7FA05895FBD6FBDE9FE737 Session-ID-ctx: Resumption PSK: 1EA68F33C65A0F0FA2655BF9C1FE906152C6E3FEEE3AEB6B1B99BA7C41F06077989352C58E07CD2FBDC363FA8A542975 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) # ...