Chapter 3. Setting up and configuring the registry
3.1. Configuring the registry for AWS user-provisioned infrastructure
3.1.1. Configuring a secret for the Image Registry Operator
In addition to the configs.imageregistry.operator.openshift.io
and ConfigMap resources, configuration is provided to the Operator by a separate secret resource located within the openshift-image-registry
namespace.
The image-registry-private-configuration-user
secret provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.
For S3 on AWS storage, the secret is expected to contain two keys:
-
REGISTRY_STORAGE_S3_ACCESSKEY
-
REGISTRY_STORAGE_S3_SECRETKEY
Procedure
Create an OpenShift Container Platform secret that contains the required keys.
$ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=myaccesskey --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=mysecretkey --namespace openshift-image-registry
3.1.2. Configuring registry storage for AWS with user-provisioned infrastructure
During installation, your cloud credentials are sufficient to create an Amazon S3 bucket and the Registry Operator will automatically configure storage.
If the Registry Operator cannot create an S3 bucket and automatically configure storage, you can create an S3 bucket and configure storage with the following procedure.
Prerequisites
- You have a cluster on AWS with user-provisioned infrastructure.
For Amazon S3 storage, the secret is expected to contain two keys:
-
REGISTRY_STORAGE_S3_ACCESSKEY
-
REGISTRY_STORAGE_S3_SECRETKEY
-
Procedure
Use the following procedure if the Registry Operator cannot create an S3 bucket and automatically configure storage.
- Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old.
Fill in the storage configuration in
configs.imageregistry.operator.openshift.io/cluster
:$ oc edit configs.imageregistry.operator.openshift.io/cluster
Example configuration
storage: s3: bucket: <bucket-name> region: <region-name>
To secure your registry images in AWS, block public access to the S3 bucket.
3.1.3. Image Registry Operator configuration parameters for AWS S3
The following configuration parameters are available for AWS S3 registry storage.
ImageRegistryConfigStorageS3
holds the information to configure the registry to use the AWS S3 service for back-end storage. See the S3 storage driver documentation for more information.
Parameter | Description |
---|---|
| Bucket is the bucket name in which you want to store the registry’s data. It is optional and is generated if not provided. |
| Region is the AWS region in which your bucket exists. It is optional and is set based on the installed AWS Region. |
| RegionEndpoint is the endpoint for S3 compatible storage services. It is optional and defaults based on the Region that is provided. |
| VirtualHostedStyle enables using S3 virtual hosted style bucket paths with a custom RegionEndpoint. It is optional and defaults to false. Set this parameter to deploy OpenShift Container Platform to hidden regions. |
| Encrypt specifies whether or not the registry stores the image in encrypted format. It is optional and defaults to false. |
| KeyID is the KMS key ID to use for encryption. It is optional. Encrypt must be true, or this parameter is ignored. |
| CloudFront configures Amazon Cloudfront as the storage middleware in a registry. It is optional. |
When the value of the regionEndpoint
parameter is configured to a URL of a Rados Gateway, an explicit port must not be specified. For example:
regionEndpoint: http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local
3.2. Configuring the registry for GCP user-provisioned infrastructure
3.2.1. Configuring a secret for the Image Registry Operator
In addition to the configs.imageregistry.operator.openshift.io
and ConfigMap resources, configuration is provided to the Operator by a separate secret resource located within the openshift-image-registry
namespace.
The image-registry-private-configuration-user
secret provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.
For GCS on GCP storage, the secret is expected to contain one key whose value is the contents of a credentials file provided by GCP:
-
REGISTRY_STORAGE_GCS_KEYFILE
Procedure
Create an OpenShift Container Platform secret that contains the required keys.
$ oc create secret generic image-registry-private-configuration-user --from-file=REGISTRY_STORAGE_GCS_KEYFILE=<path_to_keyfile> --namespace openshift-image-registry
3.2.2. Registry storage for GCP with user-provisioned infrastructure
You must set up the storage medium manually and configure the settings in the registry custom resource (CR).
Prerequisites
- A cluster on GCP with user-provisioned infrastructure.
- To configure registry storage for GCP, you need to provide Registry Operator cloud credentials.
For GCS on GCP storage, the secret is expected to contain one key whose value is the contents of a credentials file provided by GCP:
-
REGISTRY_STORAGE_GCS_KEYFILE
-
3.2.3. Image Registry Operator configuration parameters for GCP GCS
Procedure
The following configuration parameters are available for GCP GCS registry storage.
Parameter | Description |
---|---|
| Bucket is the bucket name in which you want to store the registry’s data. It is optional and is generated if not provided. |
| Region is the GCS location in which your bucket exists. It is optional and is set based on the installed GCS Region. |
| ProjectID is the Project ID of the GCP project that this bucket should be associated with. It is optional. |
| KeyID is the KMS key ID to use for encryption. It is optional because buckets are encrypted by default on GCP. This allows for the use of a custom encryption key. |
3.3. Configuring the registry for OpenStack user-provisioned infrastructure
You can configure the registry of a cluster that runs on your own Red Hat OpenStack Platform (RHOSP) infrastructure.
3.3.1. Configuring the Image Registry Operator to trust Swift storage
You must configure the Image Registry Operator to trust Red Hat OpenStack Platform (RHOSP) Swift storage.
Procedure
From a command line, enter the following command to change the value of the
spec.disableRedirect
field in theconfig.imageregistry
object totrue
:$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"disableRedirect":true}}'
3.3.2. Configuring a secret for the Image Registry Operator
In addition to the configs.imageregistry.operator.openshift.io
and ConfigMap resources, configuration is provided to the Operator by a separate secret resource located within the openshift-image-registry
namespace.
The image-registry-private-configuration-user
secret provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.
For Swift on Red Hat OpenStack Platform (RHOSP) storage, the secret is expected to contain the following two keys:
-
REGISTRY_STORAGE_SWIFT_USER
-
REGISTRY_STORAGE_SWIFT_PASSWORD
Procedure
Create an OpenShift Container Platform secret that contains the required keys.
$ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_SWIFT_USER=<username> --from-literal=REGISTRY_STORAGE_SWIFT_PASSWORD=<password> -n openshift-image-registry
3.3.3. Registry storage for RHOSP with user-provisioned infrastructure
You must set up the storage medium manually and configure the settings in the registry custom resource (CR).
Prerequisites
- A cluster on Red Hat OpenStack Platform (RHOSP) with user-provisioned infrastructure.
- To configure registry storage for RHOSP, you need to provide Registry Operator cloud credentials.
For Swift on RHOSP storage, the secret is expected to contain the following two keys:
-
REGISTRY_STORAGE_SWIFT_USER
-
REGISTRY_STORAGE_SWIFT_PASSWORD
-
3.3.4. Image Registry Operator configuration parameters for RHOSP Swift
The following configuration parameters are available for Red Hat OpenStack Platform (RHOSP) Swift registry storage.
Parameter | Description |
---|---|
| This value is optional. |
| This value is optional. |
| This value is optional. |
| This value is optional. |
| This value is optional. |
| This value is optional. |
| This value is optional. |
| This value is optional. |
3.4. Configuring the registry for Azure user-provisioned infrastructure
3.4.1. Configuring a secret for the Image Registry Operator
In addition to the configs.imageregistry.operator.openshift.io
and ConfigMap resources, configuration is provided to the Operator by a separate secret resource located within the openshift-image-registry
namespace.
The image-registry-private-configuration-user
secret provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.
For Azure registry storage, the secret is expected to contain one key whose value is the contents of a credentials file provided by Azure:
-
REGISTRY_STORAGE_AZURE_ACCOUNTKEY
Procedure
Create an OpenShift Container Platform secret that contains the required key.
$ oc create secret generic image-registry-private-configuration-user --from-literal=REGISTRY_STORAGE_AZURE_ACCOUNTKEY=<accountkey> --namespace openshift-image-registry
3.4.2. Configuring registry storage for Azure
During installation, your cloud credentials are sufficient to create Azure Blob Storage, and the Registry Operator automatically configures storage.
Prerequisites
- A cluster on Azure with user-provisioned infrastructure.
- To configure registry storage for Azure, provide Registry Operator cloud credentials.
For Azure storage the secret is expected to contain one key:
-
REGISTRY_STORAGE_AZURE_ACCOUNTKEY
-
Procedure
- Create an Azure storage container.
Fill in the storage configuration in
configs.imageregistry.operator.openshift.io/cluster
:$ oc edit configs.imageregistry.operator.openshift.io/cluster
Example configuration
storage: azure: accountName: <storage-account-name> container: <container-name>
3.4.3. Configuring registry storage for Azure Government
During installation, your cloud credentials are sufficient to create Azure Blob Storage, and the Registry Operator automatically configures storage.
Prerequisites
- A cluster on Azure with user-provisioned infrastructure in a government region.
- To configure registry storage for Azure, provide Registry Operator cloud credentials.
For Azure storage, the secret is expected to contain one key:
-
REGISTRY_STORAGE_AZURE_ACCOUNTKEY
-
Procedure
- Create an Azure storage container.
Fill in the storage configuration in
configs.imageregistry.operator.openshift.io/cluster
:$ oc edit configs.imageregistry.operator.openshift.io/cluster
Example configuration
storage: azure: accountName: <storage-account-name> container: <container-name> cloudName: AzureUSGovernmentCloud 1
- 1
cloudName
is the name of the Azure cloud environment, which can be used to configure the Azure SDK with the appropriate Azure API endpoints. Defaults toAzurePublicCloud
. You can also setcloudName
toAzureUSGovernmentCloud
,AzureChinaCloud
, orAzureGermanCloud
with sufficient credentials.
3.5. Configuring the registry for RHOSP
3.5.1. Configuring an image registry with custom storage on clusters that run on RHOSP
After you install a cluster on Red Hat OpenStack Platform (RHOSP), you can use a Cinder volume that is in a specific availability zone for registry storage.
Procedure
Create a YAML file that specifies the storage class and availability zone to use. For example:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: custom-csi-storageclass provisioner: cinder.csi.openstack.org volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: availability: <availability_zone_name>
NoteOpenShift Container Platform does not verify the existence of the availability zone you choose. Verify the name of the availability zone before you apply the configuration.
From a command line, apply the configuration:
$ oc apply -f <storage_class_file_name>
Example output
storageclass.storage.k8s.io/custom-csi-storageclass created
Create a YAML file that specifies a persistent volume claim (PVC) that uses your storage class and the
openshift-image-registry
namespace. For example:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-pvc-imageregistry namespace: openshift-image-registry 1 annotations: imageregistry.openshift.io: "true" spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 100Gi 2 storageClassName: <your_custom_storage_class> 3
From a command line, apply the configuration:
$ oc apply -f <pvc_file_name>
Example output
persistentvolumeclaim/csi-pvc-imageregistry created
Replace the original persistent volume claim in the image registry configuration with the new claim:
$ oc patch configs.imageregistry.operator.openshift.io/cluster --type 'json' -p='[{"op": "replace", "path": "/spec/storage/pvc/claim", "value": "csi-pvc-imageregistry"}]'
Example output
config.imageregistry.operator.openshift.io/cluster patched
Over the next several minutes, the configuration is updated.
Verification
To confirm that the registry is using the resources that you defined:
Verify that the PVC claim value is identical to the name that you provided in your PVC definition:
$ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml
Example output
... status: ... managementState: Managed pvc: claim: csi-pvc-imageregistry ...
Verify that the status of the PVC is
Bound
:$ oc get pvc -n openshift-image-registry csi-pvc-imageregistry
Example output
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE csi-pvc-imageregistry Bound pvc-72a8f9c9-f462-11e8-b6b6-fa163e18b7b5 100Gi RWO custom-csi-storageclass 11m
3.6. Configuring the registry for bare metal
3.6.1. Image registry removed during installation
On platforms that do not provide shareable object storage, the OpenShift Image Registry Operator bootstraps itself as Removed
. This allows openshift-installer
to complete installations on these platform types.
After installation, you must edit the Image Registry Operator configuration to switch the managementState
from Removed
to Managed
.
The Prometheus console provides an ImageRegistryRemoved
alert, for example:
"Image Registry has been removed. ImageStreamTags
, BuildConfigs
and DeploymentConfigs
which reference ImageStreamTags
may not work as expected. Please configure storage and update the config to Managed
state by editing configs.imageregistry.operator.openshift.io."
3.6.2. Changing the image registry’s management state
To start the image registry, you must change the Image Registry Operator configuration’s managementState
from Removed
to Managed
.
Procedure
Change
managementState
Image Registry Operator configuration fromRemoved
toManaged
. For example:$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
3.6.3. Image registry storage configuration
The Image Registry Operator is not initially available for platforms that do not provide default storage. After installation, you must configure your registry to use storage so that the Registry Operator is made available.
Instructions are shown for configuring a persistent volume, which is required for production clusters. Where applicable, instructions are shown for configuring an empty directory as the storage location, which is available for only non-production clusters.
Additional instructions are provided for allowing the image registry to use block storage types by using the Recreate
rollout strategy during upgrades.
3.6.4. Configuring registry storage for bare metal and other manual installations
As a cluster administrator, following installation you must configure your registry to use storage.
Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role. - You have a cluster that uses manually-provisioned Red Hat Enterprise Linux CoreOS (RHCOS) nodes, such as bare metal.
You have provisioned persistent storage for your cluster, such as Red Hat OpenShift Container Storage.
ImportantOpenShift Container Platform supports
ReadWriteOnce
access for image registry storage when you have only one replica.ReadWriteOnce
access also requires that the registry uses theRecreate
rollout strategy. To deploy an image registry that supports high availability with two or more replicas,ReadWriteMany
access is required.- Must have 100Gi capacity.
Procedure
To configure your registry to use storage, change the
spec.storage.pvc
in theconfigs.imageregistry/cluster
resource.NoteWhen using shared storage, review your security settings to prevent outside access.
Verify that you do not have a registry pod:
$ oc get pod -n openshift-image-registry -l docker-registry=default
Example output
No resourses found in openshift-image-registry namespace
NoteIf you do have a registry pod in your output, you do not need to continue with this procedure.
Check the registry configuration:
$ oc edit configs.imageregistry.operator.openshift.io
Example output
storage: pvc: claim:
Leave the
claim
field blank to allow the automatic creation of animage-registry-storage
PVC.Check the
clusteroperator
status:$ oc get clusteroperator image-registry
Example output
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE image-registry 4.7 True False False 6h50m
Ensure that your registry is set to managed to enable building and pushing of images.
Run:
$ oc edit configs.imageregistry/cluster
Then, change the line
managementState: Removed
to
managementState: Managed
3.6.5. Configuring storage for the image registry in non-production clusters
You must configure storage for the Image Registry Operator. For non-production clusters, you can set the image registry to an empty directory. If you do so, all images are lost if you restart the registry.
Procedure
To set the image registry storage to an empty directory:
$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
WarningConfigure this option for only non-production clusters.
If you run this command before the Image Registry Operator initializes its components, the
oc patch
command fails with the following error:Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
Wait a few minutes and run the command again.
3.6.6. Configuring block registry storage
To allow the image registry to use block storage types during upgrades as a cluster administrator, you can use the Recreate
rollout strategy.
Block storage volumes are supported but not recommended for use with the image registry on production clusters. An installation where the registry is configured on block storage is not highly available because the registry cannot have more than one replica.
Procedure
To set the image registry storage as a block storage type, patch the registry so that it uses the
Recreate
rollout strategy and runs with only one (1
) replica:$ oc patch config.imageregistry.operator.openshift.io/cluster --type=merge -p '{"spec":{"rolloutStrategy":"Recreate","replicas":1}}'
- Provision the PV for the block storage device, and create a PVC for that volume. The requested block volume uses the ReadWriteOnce (RWO) access mode.
- Edit the registry configuration so that it references the correct PVC.
3.6.7. Additional resources
- For more details about configuring registry storage for bare metal, see Recommended configurable storage technology.
3.7. Configuring the registry for vSphere
3.7.1. Image registry removed during installation
On platforms that do not provide shareable object storage, the OpenShift Image Registry Operator bootstraps itself as Removed
. This allows openshift-installer
to complete installations on these platform types.
After installation, you must edit the Image Registry Operator configuration to switch the managementState
from Removed
to Managed
.
The Prometheus console provides an ImageRegistryRemoved
alert, for example:
"Image Registry has been removed. ImageStreamTags
, BuildConfigs
and DeploymentConfigs
which reference ImageStreamTags
may not work as expected. Please configure storage and update the config to Managed
state by editing configs.imageregistry.operator.openshift.io."
3.7.2. Changing the image registry’s management state
To start the image registry, you must change the Image Registry Operator configuration’s managementState
from Removed
to Managed
.
Procedure
Change
managementState
Image Registry Operator configuration fromRemoved
toManaged
. For example:$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
3.7.2.1. Image registry storage configuration
The Image Registry Operator is not initially available for platforms that do not provide default storage. After installation, you must configure your registry to use storage so that the Registry Operator is made available.
Instructions are shown for configuring a persistent volume, which is required for production clusters. Where applicable, instructions are shown for configuring an empty directory as the storage location, which is available for only non-production clusters.
Additional instructions are provided for allowing the image registry to use block storage types by using the Recreate
rollout strategy during upgrades.
3.7.3. Configuring registry storage for VMware vSphere
As a cluster administrator, following installation you must configure your registry to use storage.
Prerequisites
- Cluster administrator permissions.
- A cluster on VMware vSphere.
Persistent storage provisioned for your cluster, such as Red Hat OpenShift Container Storage.
ImportantOpenShift Container Platform supports
ReadWriteOnce
access for image registry storage when you have only one replica.ReadWriteOnce
access also requires that the registry uses theRecreate
rollout strategy. To deploy an image registry that supports high availability with two or more replicas,ReadWriteMany
access is required.- Must have "100Gi" capacity.
Testing shows issues with using the NFS server on RHEL as storage backend for core services. This includes the OpenShift Container Registry and Quay, Prometheus for monitoring storage, and Elasticsearch for logging storage. Therefore, using RHEL NFS to back PVs used by core services is not recommended.
Other NFS implementations on the marketplace might not have these issues. Contact the individual NFS implementation vendor for more information on any testing that was possibly completed against these OpenShift Container Platform core components.
Procedure
To configure your registry to use storage, change the
spec.storage.pvc
in theconfigs.imageregistry/cluster
resource.NoteWhen using shared storage, review your security settings to prevent outside access.
Verify that you do not have a registry pod:
$ oc get pod -n openshift-image-registry -l docker-registry=default
Example output
No resourses found in openshift-image-registry namespace
NoteIf you do have a registry pod in your output, you do not need to continue with this procedure.
Check the registry configuration:
$ oc edit configs.imageregistry.operator.openshift.io
Example output
storage: pvc: claim: 1
- 1
- Leave the
claim
field blank to allow the automatic creation of animage-registry-storage
persistent volume claim (PVC). The PVC is generated based on the default storage class. However, be aware that the default storage class might provide ReadWriteOnce (RWO) volumes, such as a RADOS Block Device (RBD), which can cause issues when replicating to more than one replica.
Check the
clusteroperator
status:$ oc get clusteroperator image-registry
Example output
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE image-registry 4.7 True False False 6h50m
3.7.4. Configuring storage for the image registry in non-production clusters
You must configure storage for the Image Registry Operator. For non-production clusters, you can set the image registry to an empty directory. If you do so, all images are lost if you restart the registry.
Procedure
To set the image registry storage to an empty directory:
$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
WarningConfigure this option for only non-production clusters.
If you run this command before the Image Registry Operator initializes its components, the
oc patch
command fails with the following error:Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
Wait a few minutes and run the command again.
3.7.5. Configuring block registry storage for VMware vSphere
To allow the image registry to use block storage types such as vSphere Virtual Machine Disk (VMDK) during upgrades as a cluster administrator, you can use the Recreate
rollout strategy.
Block storage volumes are supported but not recommended for use with image registry on production clusters. An installation where the registry is configured on block storage is not highly available because the registry cannot have more than one replica.
Procedure
To set the image registry storage as a block storage type, patch the registry so that it uses the
Recreate
rollout strategy and runs with only1
replica:$ oc patch config.imageregistry.operator.openshift.io/cluster --type=merge -p '{"spec":{"rolloutStrategy":"Recreate","replicas":1}}'
Provision the PV for the block storage device, and create a PVC for that volume. The requested block volume uses the ReadWriteOnce (RWO) access mode.
Create a
pvc.yaml
file with the following contents to define a VMware vSpherePersistentVolumeClaim
object:kind: PersistentVolumeClaim apiVersion: v1 metadata: name: image-registry-storage 1 namespace: openshift-image-registry 2 spec: accessModes: - ReadWriteOnce 3 resources: requests: storage: 100Gi 4
- 1
- A unique name that represents the
PersistentVolumeClaim
object. - 2
- The namespace for the
PersistentVolumeClaim
object, which isopenshift-image-registry
. - 3
- The access mode of the persistent volume claim. With
ReadWriteOnce
, the volume can be mounted with read and write permissions by a single node. - 4
- The size of the persistent volume claim.
Create the
PersistentVolumeClaim
object from the file:$ oc create -f pvc.yaml -n openshift-image-registry
Edit the registry configuration so that it references the correct PVC:
$ oc edit config.imageregistry.operator.openshift.io -o yaml
Example output
storage: pvc: claim: 1
- 1
- Creating a custom PVC allows you to leave the
claim
field blank for the default automatic creation of animage-registry-storage
PVC.
For instructions about configuring registry storage so that it references the correct PVC, see Configuring the registry for vSphere.
3.7.6. Additional resources
- For more details about configuring registry storage for vSphere, see Recommended configurable storage technology.