Chapter 8. Configuring certificates
8.1. Replacing the default ingress certificate
8.1.1. Understanding the default ingress certificate
By default OpenShift Container Platform uses the Ingress Operator to create an internal CA and issue a wildcard certificate that is valid for applications under the .apps
sub-domain. Both the web console and CLI use this certificate as well.
The internal infrastructure CA certificates are self-signed. While this process might be perceived as bad practice by some security or PKI teams, any risk here is minimal. The only clients that implicitly trust these certificates are other components within the cluster. Replacing the default wildcard certificate with one that is issued by a public CA already included in the CA bundle as provided by the container userspace allows external clients to connect securely to applications running under the .apps
sub-domain.
8.1.2. Replacing the default ingress certificate
You can replace the default ingress certificate for all applications under the .apps
subdomain. After you replace the certificate, all applications, including the web console and CLI, will have encryption provided by specified certificate.
Prerequisites
-
You must have a wildcard certificate for the fully qualified
.apps
subdomain and its corresponding private key. Each should be in a separate PEM format file. - The private key must be unencrypted. If your key is encrypted, decrypt it before importing it into OpenShift Container Platform.
-
The certificate must include the
subjectAltName
extension showing*.apps.<clustername>.<domain>
. - The certificate file can contain one or more certificates in a chain. The wildcard certificate must be the first certificate in the file. It can then be followed with any intermediate certificates, and the file should end with the root CA certificate.
- Copy the root CA certificate into an additional PEM format file.
Procedure
Create a ConfigMap that includes only the root CA certificate used to sign the wildcard certificate:
$ oc create configmap custom-ca \ --from-file=ca-bundle.crt=</path/to/example-ca.crt> \1 -n openshift-config
- 1
</path/to/example-ca.crt>
is the path to the root CA certificate file on your local file system.
Update the cluster-wide proxy configuration with the newly created ConfigMap:
$ oc patch proxy/cluster \ --type=merge \ --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
Create a secret that contains the wildcard certificate chain and key:
$ oc create secret tls <secret> \1 --cert=</path/to/cert.crt> \2 --key=</path/to/cert.key> \3 -n openshift-ingress
Update the Ingress Controller configuration with the newly created secret:
$ oc patch ingresscontroller.operator default \ --type=merge -p \ '{"spec":{"defaultCertificate": {"name": "<secret>"}}}' \1 -n openshift-ingress-operator
- 1
- Replace
<secret>
with the name used for the secret in the previous step.
8.2. Adding API server certificates
The default API server certificate is issued by an internal OpenShift Container Platform cluster CA. Clients outside of the cluster will not be able to verify the API server’s certificate by default. This certificate can be replaced by one that is issued by a CA that clients trust.
8.2.1. Add an API server named certificate
The default API server certificate is issued by an internal OpenShift Container Platform cluster CA. You can add one or more alternative certificates that the API server will return based on the fully qualified domain name (FQDN) requested by the client, for example when a reverse proxy or load balancer is used.
Prerequisites
- You must have a certificate for the FQDN and its corresponding private key. Each should be in a separate PEM format file.
- The private key must be unencrypted. If your key is encrypted, decrypt it before importing it into OpenShift Container Platform.
-
The certificate must include the
subjectAltName
extension showing the FQDN. - The certificate file can contain one or more certificates in a chain. The certificate for the API server FQDN must be the first certificate in the file. It can then be followed with any intermediate certificates, and the file should end with the root CA certificate.
Do not provide a named certificate for the internal load balancer (host name api-int.<cluster_name>.<base_domain>
). Doing so will leave your cluster in a degraded state.
Procedure
Create a secret that contains the certificate chain and private key in the
openshift-config
namespace.$ oc create secret tls <secret> \1 --cert=</path/to/cert.crt> \2 --key=</path/to/cert.key> \3 -n openshift-config
Update the API server to reference the created secret.
$ oc patch apiserver cluster \ --type=merge -p \ '{"spec":{"servingCerts": {"namedCertificates": [{"names": ["<FQDN>"], 1 "servingCertificate": {"name": "<secret>"}}]}}}' 2
Examine the
apiserver/cluster
object and confirm the secret is now referenced.$ oc get apiserver cluster -o yaml ... spec: servingCerts: namedCertificates: - names: - <FQDN> servingCertificate: name: <secret> ...
8.3. Securing service traffic using service serving certificate secrets
8.3.1. Understanding service serving certificates
Service serving certificates are intended to support complex middleware applications that require encryption. These certificates are issued as TLS web server certificates.
The service-ca
controller uses the x509.SHA256WithRSA
signature algorithm to generate service certificates.
The generated certificate and key are in PEM format, stored in tls.crt
and tls.key
respectively, within a created secret. The certificate and key are automatically replaced when they get close to expiration.
The service CA certificate, which issues the service certificates, is valid for 26 months and is automatically rotated when there is less than six months validity left. After rotation, the previous service CA configuration is still trusted until its expiration. This allows a grace period for all affected services to refresh their key material before the expiration. If you do not upgrade your cluster during this grace period, which restarts services and refreshes their key material, you might need to manually restart services to avoid failures after the previous service CA expires.
You can use the following command to manually restart all Pods in the cluster. Be aware that running this command causes a service interruption, because it deletes every running Pod in every namespace. These Pods will automatically restart after they are deleted.
$ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \ do oc delete pods --all -n $I; \ sleep 1; \ done
8.3.2. Add a service certificate
To secure communication to your service, generate a signed serving certificate and key pair into a secret in the same namespace as the service.
The generated certificate is only valid for the internal service DNS name <service.name>.<service.namespace>.svc
, and are only valid for internal communications.
Prerequisites:
- You must have a service defined.
Procedure
Annotate the service with
service.beta.openshift.io/serving-cert-secret-name
.$ oc annotate service <service-name> \1 service.beta.openshift.io/serving-cert-secret-name=<secret-name> 2
For instance, use the following command to annotate the service
foo
:$ oc annotate service foo service.beta.openshift.io/serving-cert-secret-name=foo
Examine the service to confirm the annotations are present.
$ oc describe service <service-name> ... Annotations: service.beta.openshift.io/serving-cert-secret-name: <service-name> service.beta.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1556850837 ...
- After the cluster generates a secret for your service, your PodSpec can mount it, and the Pod will run after it becomes available.
8.3.3. Add a service certificate to a ConfigMap
A Pod can access the service CA certificate by mounting a ConfigMap that is annotated with service.beta.openshift.io/inject-cabundle=true
. Once annotated, the cluster automatically injects the service CA certificate into the service-ca.crt
key on the ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using service serving certificates.
After adding this annotation to a ConfigMap all existing data in it is deleted. It is recommended to use a separate ConfigMap to contain the service-ca.crt
, instead of using the same ConfigMap that stores your Pod’s configuration.
Procedure
Annotate the ConfigMap with
service.beta.openshift.io/inject-cabundle=true
.$ oc annotate configmap <configmap-name> \1 service.beta.openshift.io/inject-cabundle=true
- 1
- Replace
<configmap-name>
with the name of the ConfigMap to annotate.
NoteExplicitly referencing the
service-ca.crt
key in a volumeMount will prevent a Pod from starting until the ConfigMap has been injected with the CA bundle.For instance, to annotate the ConfigMap
foo
the following command would be used:$ oc annotate configmap foo service.beta.openshift.io/inject-cabundle=true
View the ConfigMap to ensure the certificate has been generated. This appears as a
service-ca.crt
in the YAML output.$ oc get configmap <configmap-name> -o yaml apiVersion: v1 data: service-ca.crt: | -----BEGIN CERTIFICATE----- ...
8.3.4. Manually rotate the generated service certificate
You can rotate the service certificate by deleting the associated secret. Deleting the secret results in a new one being automatically created, resulting in a new certificate.
Prerequisites
- A secret containing the certificate and key pair must have been generated for the service.
Procedure
Examine the service to determine the secret containing the certificate. This is found in the
serving-cert-secret-name
annotation, as seen below.$ oc describe service <service-name> ... service.beta.openshift.io/serving-cert-secret-name: <secret> ...
Delete the generated secret for the service. This process will automatically recreate the secret.
$ oc delete secret <secret> 1
- 1
- Replace
<secret>
with the name of the secret from the previous step.
Confirm that the certificate has been recreated by obtaining the new secret and examining the
AGE
.$ oc get secret <service-name> NAME TYPE DATA AGE <service.name> kubernetes.io/tls 2 1s
8.3.5. Manually rotate the service CA certificate
The service CA is valid for 26 months and is automatically refreshed when there is less than six months validity left.
If necessary, you can manually refresh the service CA by using the following procedure.
A manually-rotated service CA does not maintain trust with the previous service CA. You might experience a temporary service disruption until the Pods in the cluster are restarted, which ensures that Pods are using service serving certificates issued by the new service CA.
Prerequisites
- You must be logged in as a cluster admin.
Procedure
View the expiration date of the current service CA certificate by using the following command.
$ oc get secrets/signing-key -n openshift-service-ca \ -o template='{{index .data "tls.crt"}}' \ | base64 -d \ | openssl x509 -noout -enddate
Manually rotate the service CA. This process generates a new service CA which will be used to sign the new service certificates.
$ oc delete secret/signing-key -n openshift-service-ca
To apply the new certificates to all services, restart all the Pods in your cluster. This command ensures that all services use the updated certificates.
$ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \ do oc delete pods --all -n $I; \ sleep 1; \ done
WarningThis command will cause a service interruption, as it goes through and deletes every running pod in every namespace. These pods will automatically restart after they are deleted.