Chapter 2. Certificate types and descriptions
2.1. Certificate validation
OpenShift Container Platform monitors certificates for proper validity, for the cluster certificates it issues and manages. The OpenShift Container Platform alerting framework has rules to help identify when a certificate issue is about to occur. These rules consist of the following checks:
- API server client certificate expiration is less than five minutes.
2.2. User-provided certificates for the API server
Purpose
The API server is accessible by clients external to the cluster at api.<cluster_name>.<base_domain>
. You might want clients to access the API server at a different host name or without the need to distribute the cluster-managed certificate authority (CA) certificates to the clients. The administrator must set a custom default certificate to be used by the API server when serving content.
Location
The user-provided certificates must be provided in a kubernetes.io/tls
type Secret
in the openshift-config
namespace. Update the API server cluster configuration, the apiserver/cluster
resource, to enable the use of the user-provided certificate.
Management
User-provided certificates are managed by the user.
Expiration
User-provided certificates are managed by the user.
Customization
Update the secret containing the user-managed certificate as needed.
2.3. Proxy certificates
Purpose
Proxy certificates allow users to specify one or more custom certificate authority (CA) certificates used by platform components when making egress connections.
The trustedCA
field of the Proxy object is a reference to a ConfigMap that contains a user-provided trusted certificate authority (CA) bundle. This bundle is merged with the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle and injected into the trust store of platform components that make egress HTTPS calls. For example, image-registry-operator
calls an external image registry to download images. If trustedCA
is not specified, only the RHCOS trust bundle is used for proxied HTTPS connections. Provide custom CA certificates to the RHCOS trust bundle if you want to use your own certificate infrastructure.
The trustedCA
field should only be consumed by a proxy validator. The validator is responsible for reading the certificate bundle from required key ca-bundle.crt
and copying it to a ConfigMap named trusted-ca-bundle
in the openshift-config-managed
namespace. The namespace for the ConfigMap referenced by trustedCA
is openshift-config
:
apiVersion: v1 kind: ConfigMap metadata: name: user-ca-bundle namespace: openshift-config data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- Custom CA certificate bundle. -----END CERTIFICATE-----
Managing proxy certificates during installation
The additionalTrustBundle
value of the installer configuration is used to specify any proxy-trusted CA certificates during installation. For example:
$ cat install-config.yaml . . . proxy: httpProxy: http://<HTTP_PROXY> httpsProxy: https://<HTTPS_PROXY> additionalTrustBundle: | -----BEGIN CERTIFICATE----- <MY_HTTPS_PROXY_TRUSTED_CA_CERT> -----END CERTIFICATE----- . . .
Location
The user-provided trust bundle is represented as a ConfigMap. The ConfigMap is mounted into the file system of platform components that make egress HTTPS calls. Typically, Operators mount the ConfigMap to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
, but this is not required by the proxy. A proxy can modify or inspect the HTTPS connection. In either case, the proxy must generate and sign a new certificate for the connection.
Complete proxy support means connecting to the specified proxy and trusting any signatures it has generated. Therefore, it is necessary to let the user specify a trusted root, such that any certificate chain connected to that trusted root is also trusted.
If using the RHCOS trust bundle, place CA certificates in /etc/pki/ca-trust/source/anchors
.
See Using shared system certificates in the Red Hat Enterprise Linux documentation for more information.
Expiration
The user sets the expiration term of the user-provided trust bundle.
The default expiration term is defined by the CA certificate itself. It is up to the CA administrator to configure this for the certificate before it can be used by OpenShift Container Platform or RHCOS.
Red Hat does not monitor for when CAs expire. However, due to the long life of CAs, this is generally not an issue. However, you might need to periodically update the trust bundle.
Services
By default, all platform components that make egress HTTPS calls will use the RHCOS trust bundle. If trustedCA
is defined, it will also be used.
Any service that is running on the RHCOS node is able to use the trust bundle of the node.
Management
These certificates are managed by the system and not the user.
Customization
Updating the user-provided trust bundle consists of either:
-
updating the PEM-encoded certificates in the ConfigMap referenced by
trustedCA,
or -
creating a ConfigMap in the namespace
openshift-config
that contains the new trust bundle and updatingtrustedCA
to reference the name of the new ConfigMap.
The mechanism for writing CA certificates to the RHCOS trust bundle is exactly the same as writing any other file to RHCOS, which is done through the use of MachineConfigs. When the Machine Config Operator (MCO) applies the new MachineConfig that contains the new CA certificates, the node is rebooted. During the next boot, the service coreos-update-ca-trust.service
runs on the RHCOS nodes, which automatically update the trust bundle with the new CA certificates. For example:
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 50-examplecorp-ca-cert spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVORENDQXh5Z0F3SUJBZ0lKQU51bkkwRDY2MmNuTUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdsTVFzd0NRWUQKV1FRR0V3SlZVekVYTUJVR0ExVUVDQXdPVG05eWRHZ2dRMkZ5YjJ4cGJtRXhFREFPQmdOVkJBY01CMUpoYkdWcApBMmd4RmpBVUJnTlZCQW9NRFZKbFpDQklZWFFzSUVsdVl5NHhFekFSQmdOVkJBc01DbEpsWkNCSVlYUWdTVlF4Ckh6QVpCZ05WQkFNTUVsSmxaQ0JJWVhRZ1NWUWdVbTl2ZENCRFFURWhNQjhHQ1NxR1NJYjNEUUVKQVJZU2FXNW0KWGpDQnBURUxNQWtHQTFVRUJoTUNWVk14RnpBVkJnTlZCQWdNRGs1dmNuUm9JRU5oY205c2FXNWhNUkF3RGdZRApXUVFIREFkU1lXeGxhV2RvTVJZd0ZBWURWUVFLREExU1pXUWdTR0YwTENCSmJtTXVNUk13RVFZRFZRUUxEQXBTCkFXUWdTR0YwSUVsVU1Sc3dHUVlEVlFRRERCSlNaV1FnU0dGMElFbFVJRkp2YjNRZ1EwRXhJVEFmQmdrcWhraUcKMHcwQkNRRVdFbWx1Wm05elpXTkFjbVZrYUdGMExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUApCRENDQVFvQ2dnRUJBTFF0OU9KUWg2R0M1TFQxZzgwcU5oMHU1MEJRNHNaL3laOGFFVHh0KzVsblBWWDZNSEt6CmQvaTdsRHFUZlRjZkxMMm55VUJkMmZRRGsxQjBmeHJza2hHSUlaM2lmUDFQczRsdFRrdjhoUlNvYjNWdE5xU28KSHhrS2Z2RDJQS2pUUHhEUFdZeXJ1eTlpckxaaW9NZmZpM2kvZ0N1dDBaV3RBeU8zTVZINXFXRi9lbkt3Z1BFUwpZOXBvK1RkQ3ZSQi9SVU9iQmFNNzYxRWNyTFNNMUdxSE51ZVNmcW5obzNBakxRNmRCblBXbG82MzhabTFWZWJLCkNFTHloa0xXTVNGa0t3RG1uZTBqUTAyWTRnMDc1dkNLdkNzQ0F3RUFBYU5qTUdFd0hRWURWUjBPQkJZRUZIN1IKNXlDK1VlaElJUGV1TDhacXczUHpiZ2NaTUI4R0ExVWRJd1FZTUJhQUZIN1I0eUMrVWVoSUlQZXVMOFpxdzNQegpjZ2NaTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RGdZRFZSMFBBUUgvQkFRREFnR0dNQTBHQ1NxR1NJYjNEUUVCCkR3VUFBNElCQVFCRE52RDJWbTlzQTVBOUFsT0pSOCtlbjVYejloWGN4SkI1cGh4Y1pROGpGb0cwNFZzaHZkMGUKTUVuVXJNY2ZGZ0laNG5qTUtUUUNNNFpGVVBBaWV5THg0ZjUySHVEb3BwM2U1SnlJTWZXK0tGY05JcEt3Q3NhawpwU29LdElVT3NVSks3cUJWWnhjckl5ZVFWMnFjWU9lWmh0UzV3QnFJd09BaEZ3bENFVDdaZTU4UUhtUzQ4c2xqCjVlVGtSaml2QWxFeHJGektjbGpDNGF4S1Fsbk92VkF6eitHbTMyVTB4UEJGNEJ5ZVBWeENKVUh3MVRzeVRtZWwKU3hORXA3eUhvWGN3bitmWG5hK3Q1SldoMWd4VVp0eTMKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= filesystem: root mode: 0644 path: /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
The trust store of machines must also support updating the trust store of nodes.
Renewal
There are no Operators that can auto-renew certificates on the RHCOS nodes.
Red Hat does not monitor for when CAs expire. However, due to the long life of CAs, this is generally not an issue. However, you might need to periodically update the trust bundle.
2.4. Service CA certificates
Purpose
service-ca
is an Operator that creates a self-signed CA when an OpenShift Container Platform cluster is deployed.
Expiration
A custom expiration term is not supported. The self-signed CA is stored in a secret with qualified name service-ca/signing-key
in fields tls.crt
(certificate(s)), tls.key
(private key), and ca-bundle.crt
(CA bundle).
Other services can request a service serving certificate by annotating a service resource with service.beta.openshift.io/serving-cert-secret-name: <secret name>
. In response, the Operator generates a new certificate, as tls.crt
, and private key, as tls.key
to the named secret. The certificate is valid for two years.
Other services can request that the CA bundle for the service CA be injected into APIService or ConfigMap resources by annotating with service.beta.openshift.io/inject-cabundle: true
to support validating certificates generated from the service CA. In response, the Operator writes its current CA bundle to the CABundle
field of APIService or as service-ca.crt
to a ConfigMap.
As of OpenShift Container Platform 4.3.5, automated rotation is supported and is backported to some 4.2.z and 4.3.z releases. For any release supporting automated rotation, the service CA is valid for 26 months and is automatically refreshed when there is less than 13 months validity left. If necessary, you can manually refresh the service CA.
The service CA expiration of 26 months is longer than the expected upgrade interval for a supported OpenShift Container Platform cluster, such that non-control plane consumers of service CA certificates will be refreshed after CA rotation and prior to the expiration of the pre-rotation CA.
A manually-rotated service CA does not maintain trust with the previous service CA. You might experience a temporary service disruption until the Pods in the cluster are restarted, which ensures that Pods are using service serving certificates issued by the new service CA.
Management
These certificates are managed by the system and not the user.
Services
Services that use service CA certificates include:
- cluster-autoscaler-operator
- cluster-monitoring-operator
- cluster-authentication-operator
- cluster-image-registry-operator
- cluster-ingress-operator
- cluster-kube-apiserver-operator
- cluster-kube-controller-manager-operator
- cluster-kube-scheduler-operator
- cluster-networking-operator
- cluster-openshift-apiserver-operator
- cluster-openshift-controller-manager-operator
- cluster-samples-operator
- cluster-svcat-apiserver-operator
- cluster-svcat-controller-manager-operator
- machine-config-operator
- console-operator
- insights-operator
- machine-api-operator
- operator-lifecycle-manager
This is not a comprehensive list.
2.5. Node certificates
Purpose
Node certificates are signed by the cluster; they come from a certificate authority (CA) that is generated by the bootstrap process. Once the cluster is installed, the node certificates are auto-rotated.
Management
These certificates are managed by the system and not the user.
2.6. Bootstrap certificates
Purpose
The kubelet, in OpenShift Container Platform 4 and later, uses the bootstrap certificate located in /etc/kubernetes/kubeconfig
to initially bootstrap. This is followed by the bootstrap initialization process and authorization of the kubelet to create a CSR.
In that process, the kubelet generates a CSR while communicating over the bootstrap channel. The controller manager signs the CSR, resulting in a certificate that the kubelet manages.
Management
These certificates are managed by the system and not the user.
Expiration
This bootstrap CA is valid for 10 years.
The kubelet-managed certificate is valid for one year and rotates automatically at around the 80 percent mark of that one year.
Customization
You cannot customize the bootstrap certificates.
2.7. etcd certificates
Purpose
etcd certificates are signed by the etcd-signer; they come from a certificate authority (CA) that is generated by the bootstrap process.
Location
CA certificates:
-
etcd CA certificate:
/etc/ssl/etcd/ca.crt
-
etcd metric CA certificate:
/etc/ssl/etcd/metric-ca.crt
-
etcd CA certificate:
-
Server certificates:
/etc/ssl/etcd/system:etcd-server
-
Client certificates:
<api_server_pod_directory>/secrets/etcd-client/
-
Peer certificates:
/etc/ssl/etcd/system:etcd-peer
-
Metric certificates:
/etc/ssl/etcd/metric-signer
Expiration
The CA certificates are valid for 10 years. The peer, client, and server certificates are valid for three years.
Management
These certificates are managed by the system and not the user.
Services
etcd certificates are used for encrypted communication between etcd member peers, as well as encrypted client traffic. The following certificates are generated and used by etcd and other processes that communicate with etcd:
- Peer certificates: Used for communication between etcd members.
-
Client certificates: Used for encrypted server-client communication. Client certificates are currently used by the API server only, and no other service should connect to etcd directly except for the proxy. Client secrets (
etcd-client
,etcd-metric-client
,etcd-metric-signer
, andetcd-signer
) are added to theopenshift-config
,openshift-monitoring
, andopenshift-kube-apiserver
namespaces. - Server certificates: Used by the etcd server for authenticating client requests.
- Metric certificates: All metric consumers connect to proxy with metric-client certificates.
2.8. OLM certificates
Management
All certificates for OpenShift Lifecycle Manager (OLM) components (olm-operator
, catalog-operator
, packageserver
, and marketplace-operator
) are managed by the system.
Operators installed via OLM can have certificates generated for them if they are providing API services. packageserver
is one example.
Certificates in the openshift-operator-lifecycle-manager
namespace are managed by OLM with the exception of certificates used by Operators that require a validating or mutating webhook.
Operators that install validating or mutating webhooks must currently manage those certificates themselves. They do not require the user to manage the certificates.
OLM will not update the certificates of Operators that it manages in proxy environments. These certificates must be managed by the user via the subscription config.
2.9. User-provided certificates for default ingress
Purpose
Applications are usually exposed at <route_name>.apps.<cluster_name>.<base_domain>
. The <cluster_name>
and <base_domain>
come from the installation config file. <route_name>
is the host field of the route, if specified, or the route name. For example, hello-openshift-default.apps.username.devcluster.openshift.com
. hello-openshift
is the name of the route and the route is in the default namespace. You might want clients to access the applications without the need to distribute the cluster-managed CA certificates to the clients. The administrator must set a custom default certificate when serving application content.
The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use operator-generated default certificates in production clusters.
Location
The user-provided certificates must be provided in a kubernetes.io/tls
type Secret
in the openshift-config
namespace. Update the ingresscontroller.operator/default
resource in the openshift-ingress-operator
namespace to enable the use of the user-provided certificate.
Management
User-provided certificates are managed by the user.
Expiration
User-provided certificates are managed by the user.
Services
Applications deployed on the cluster use user-provided certificates for default ingress.
Customization
Update the secret containing the user-managed certificate as needed.
2.10. Ingress certificates
Purpose
The Ingress Operator uses certificates for:
- Securing access to metrics for Prometheus.
- Securing access to routes.
Location
To secure access to Ingress Operator and Ingress Controller metrics, the Ingress Operator uses service serving certificates. The Operator requests a certificate from the service-ca
controller for its own metrics, and the service-ca
controller puts the certificate in a secret named metrics-tls
in the openshift-ingress-operator
namespace. Additionally, the Ingress Operator requests a certificate for each Ingress Controller, and the service-ca
controller puts the certificate in a secret named router-metrics-certs-<name>
, where <name>
is the name of the Ingress Controller, in the openshift-ingress
namespace.
Each Ingress Controller has a default certificate that it uses for secured routes that do not specify their own certificates. Unless you specify a custom certificate, the Operator uses a self-signed certificate by default. The Operator uses its own self-signed signing certificate to sign any default certificate that it generates. The Operator generates this signing certificate and puts it in a secret named router-ca
in the openshift-ingress-operator
namespace. When the Operator generates a default certificate, it puts the default certificate in a secret named router-certs-<name>
(where <name>
is the name of the Ingress Controller) in the openshift-ingress
namespace.
The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use Operator-generated default certificates in production clusters.
Workflow
Figure 2.1. Custom certificate workflow
Figure 2.2. Default certificate workflow
An empty defaultCertificate
field causes the Ingress Operator to use its self-signed CA to generate a serving certificate for the specified domain.
The default CA certificate and key generated by the Ingress Operator. Used to sign Operator-generated default serving certificates.
In the default workflow, the wildcard default serving certificate, created by the Ingress Operator and signed using the generated default CA certificate. In the custom workflow, this is the user-provided certificate.
The router deployment. Uses the certificate in secrets/router-certs-default
as its default front-end server certificate.
In the default workflow, the contents of the wildcard default serving certificate (public and private parts) are copied here to enable OAuth integration. In the custom workflow, this is the user-provided certificate.
Transitional resource containing the certificate (public part) of the Operator-generated default CA certificate; read by OAuth and the web console to establish trust. This object will be removed in a future release.
The public (certificate) part of the default serving certificate. Replaces the configmaps/router-ca
resource.
The user updates the cluster proxy configuration with the CA certificate that signed the ingresscontroller
serving certificate. This enables components like auth
, console
, and the registry to trust the serving certificate.
The cluster-wide trusted CA bundle containing the combined Red Hat Enterprise Linux CoreOS (RHCOS) and user-provided CA bundles or an RHCOS-only bundle if a user bundle is not provided.
The custom CA certificate bundle, which instructs other components (for example, auth
and console
) to trust an ingresscontroller
configured with a custom certificate.
The trustedCA
field is used to reference the user-provided CA bundle.
The Cluster Network Operator injects the trusted CA bundle into the proxy-ca
ConfigMap.
OpenShift Container Platform 4.3 and earlier versions use router-ca
.
Expiration
The expiration terms for the Ingress Operator’s certificates are as follows:
-
The expiration date for metrics certificates that the
service-ca
controller creates is two years after the date of creation. - The expiration date for the Operator’s signing certificate is two years after the date of creation.
- The expiration date for default certificates that the Operator generates is two years after the date of creation.
You cannot specify custom expiration terms on certificates that the Ingress Operator or service-ca
controller creates.
You cannot specify expiration terms when installing OpenShift Container Platform for certificates that the Ingress Operator or service-ca
controller creates.
Services
Prometheus uses the certificates that secure metrics.
The Ingress Operator uses its signing certificate to sign default certificates that it generates for Ingress Controllers for which you do not set custom default certificates.
Cluster components that use secured routes may use the default Ingress Controller’s default certificate.
Ingress to the cluster via a secured route uses the default certificate of the Ingress Controller by which the route is accessed unless the route specifies its own certificate.
Management
Ingress certificates are managed by the user. See Replacing the default ingress certificate for more information.
Renewal
The service-ca
controller automatically rotates the certificates that it issues. However, it is possible to use oc delete secret <secret>
to manually rotate service serving certificates.
The Ingress Operator does not rotate its own signing certificate or the default certificates that it generates. Operator-generated default certificates are intended as placeholders for custom default certificates that you configure.