Chapter 3. Configuring certificates

Procedure

1. Create a config map that includes only the root CA certificate used to sign the wildcard certificate:

   $ oc create configmap custom-ca \
        --from-file=ca-bundle.crt=</path/to/example-ca.crt> \1
        -n openshift-config

   1

   </path/to/example-ca.crt> is the path to the root CA certificate file on your local file system.

2. Update the cluster-wide proxy configuration with the newly created config map:

   $ oc patch proxy/cluster \
        --type=merge \
        --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'

3. Create a secret that contains the wildcard certificate chain and key:

   $ oc create secret tls <secret> \1
        --cert=</path/to/cert.crt> \2
        --key=</path/to/cert.key> \3
        -n openshift-ingress

   1

   <secret> is the name of the secret that will contain the certificate chain and private key.

   2

   </path/to/cert.crt> is the path to the certificate chain on your local file system.

   3

   </path/to/cert.key> is the path to the private key associated with this certificate.

4. Update the Ingress Controller configuration with the newly created secret:

   $ oc patch ingresscontroller.operator default \
        --type=merge -p \
        '{"spec":{"defaultCertificate": {"name": "<secret>"}}}' \1
        -n openshift-ingress-operator

   1

   Replace <secret> with the name used for the secret in the previous step.

Procedure

1. Login to the new API as the kubeadmin user.

   $ oc login -u kubeadmin -p <password> https://FQDN:6443

2. Get the kubeconfig file.

   $ oc config view --flatten > kubeconfig-newapi

3. Create a secret that contains the certificate chain and private key in the openshift-config namespace.

   $ oc create secret tls <secret> \1
        --cert=</path/to/cert.crt> \2
        --key=</path/to/cert.key> \3
        -n openshift-config

   1

   <secret> is the name of the secret that will contain the certificate chain and private key.

   2

   </path/to/cert.crt> is the path to the certificate chain on your local file system.

   3

   </path/to/cert.key> is the path to the private key associated with this certificate.

4. Update the API server to reference the created secret.

   $ oc patch apiserver cluster \
        --type=merge -p \
        '{"spec":{"servingCerts": {"namedCertificates":
        [{"names": ["<FQDN>"], 1
        "servingCertificate": {"name": "<secret>"}}]}}}' 2

   1

   Replace <FQDN> with the FQDN that the API server should provide the certificate for.

   2

   Replace <secret> with the name used for the secret in the previous step.

5. Examine the apiserver/cluster object and confirm the secret is now referenced.

   $ oc get apiserver cluster -o yaml

   Example output

   ...
   spec:
     servingCerts:
       namedCertificates:
       - names:
         - <FQDN>
         servingCertificate:
           name: <secret>
   ...

6. Check the kube-apiserver operator, and verify that a new revision of the Kubernetes API server rolls out. It may take a minute for the operator to detect the configuration change and trigger a new deployment. While the new revision is rolling out, PROGRESSING will report True.

   $ oc get clusteroperators kube-apiserver

   Do not continue to the next step until PROGRESSING is listed as False, as shown in the following output:

   Example output

   NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   kube-apiserver   4.14.0     True        False         False      145m

   If PROGRESSING is showing True, wait a few minutes and try again.

   Note

   A new revision of the Kubernetes API server only rolls out if the API server named certificate is added for the first time. When the API server named certificate is renewed, a new revision of the Kubernetes API server does not roll out because the kube-apiserver pods dynamically reload the updated certificate.

Procedure

1. Annotate the service with service.beta.openshift.io/serving-cert-secret-name:

   $ oc annotate service <service_name> \1
        service.beta.openshift.io/serving-cert-secret-name=<secret_name> 2

   1

   Replace <service_name> with the name of the service to secure.

   2

   <secret_name> will be the name of the generated secret containing the certificate and key pair. For convenience, it is recommended that this be the same as <service_name>.

   For example, use the following command to annotate the service test1:

   $ oc annotate service test1 service.beta.openshift.io/serving-cert-secret-name=test1

2. Examine the service to confirm that the annotations are present:

   $ oc describe service <service_name>

   Example output

   ...
   Annotations:              service.beta.openshift.io/serving-cert-secret-name: <service_name>
                             service.beta.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1556850837
   ...

3. After the cluster generates a secret for your service, your Pod spec can mount it, and the pod will run after it becomes available.

Procedure

1. Annotate the config map with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate configmap <config_map_name> \1
        service.beta.openshift.io/inject-cabundle=true

   1

   Replace <config_map_name> with the name of the config map to annotate.

   Note

   Explicitly referencing the service-ca.crt key in a volume mount will prevent a pod from starting until the config map has been injected with the CA bundle. This behavior can be overridden by setting the optional field to true for the volume’s serving certificate configuration.

   For example, use the following command to annotate the config map test1:

   $ oc annotate configmap test1 service.beta.openshift.io/inject-cabundle=true

2. View the config map to ensure that the service CA bundle has been injected:

   $ oc get configmap <config_map_name> -o yaml

   The CA bundle is displayed as the value of the service-ca.crt key in the YAML output:

   apiVersion: v1
   data:
     service-ca.crt: |
       -----BEGIN CERTIFICATE-----
   ...

Procedure

1. Annotate the API service with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate apiservice <api_service_name> \1
        service.beta.openshift.io/inject-cabundle=true

   1

   Replace <api_service_name> with the name of the API service to annotate.

   For example, use the following command to annotate the API service test1:

   $ oc annotate apiservice test1 service.beta.openshift.io/inject-cabundle=true

2. View the API service to ensure that the service CA bundle has been injected:

   $ oc get apiservice <api_service_name> -o yaml

   The CA bundle is displayed in the spec.caBundle field in the YAML output:

   apiVersion: apiregistration.k8s.io/v1
   kind: APIService
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   spec:
     caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the CRD with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate crd <crd_name> \1
        service.beta.openshift.io/inject-cabundle=true

   1

   Replace <crd_name> with the name of the CRD to annotate.

   For example, use the following command to annotate the CRD test1:

   $ oc annotate crd test1 service.beta.openshift.io/inject-cabundle=true

2. View the CRD to ensure that the service CA bundle has been injected:

   $ oc get crd <crd_name> -o yaml

   The CA bundle is displayed in the spec.conversion.webhook.clientConfig.caBundle field in the YAML output:

   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   spec:
     conversion:
       strategy: Webhook
       webhook:
         clientConfig:
           caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the mutating webhook configuration with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate mutatingwebhookconfigurations <mutating_webhook_name> \1
        service.beta.openshift.io/inject-cabundle=true

   1

   Replace <mutating_webhook_name> with the name of the mutating webhook configuration to annotate.

   For example, use the following command to annotate the mutating webhook configuration test1:

   $ oc annotate mutatingwebhookconfigurations test1 service.beta.openshift.io/inject-cabundle=true

2. View the mutating webhook configuration to ensure that the service CA bundle has been injected:

   $ oc get mutatingwebhookconfigurations <mutating_webhook_name> -o yaml

   The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

   apiVersion: admissionregistration.k8s.io/v1
   kind: MutatingWebhookConfiguration
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   webhooks:
   - myWebhook:
     - v1beta1
     clientConfig:
       caBundle: <CA_BUNDLE>
   ...

Procedure

1. Annotate the validating webhook configuration with service.beta.openshift.io/inject-cabundle=true:

   $ oc annotate validatingwebhookconfigurations <validating_webhook_name> \1
        service.beta.openshift.io/inject-cabundle=true

   1

   Replace <validating_webhook_name> with the name of the validating webhook configuration to annotate.

   For example, use the following command to annotate the validating webhook configuration test1:

   $ oc annotate validatingwebhookconfigurations test1 service.beta.openshift.io/inject-cabundle=true

2. View the validating webhook configuration to ensure that the service CA bundle has been injected:

   $ oc get validatingwebhookconfigurations <validating_webhook_name> -o yaml

   The CA bundle is displayed in the clientConfig.caBundle field of all webhooks in the YAML output:

   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingWebhookConfiguration
   metadata:
     annotations:
       service.beta.openshift.io/inject-cabundle: "true"
   ...
   webhooks:
   - myWebhook:
     - v1beta1
     clientConfig:
       caBundle: <CA_BUNDLE>
   ...

Procedure

1. Examine the service to determine the secret containing the certificate. This is found in the serving-cert-secret-name annotation, as seen below.

   $ oc describe service <service_name>

   Example output

   ...
   service.beta.openshift.io/serving-cert-secret-name: <secret>
   ...

2. Delete the generated secret for the service. This process will automatically recreate the secret.

   $ oc delete secret <secret> 1

   1

   Replace <secret> with the name of the secret from the previous step.

3. Confirm that the certificate has been recreated by obtaining the new secret and examining the AGE.

   $ oc get secret <service_name>

   Example output

   NAME              TYPE                DATA   AGE
   <service.name>    kubernetes.io/tls   2      1s

Procedure

1. View the expiration date of the current service CA certificate by using the following command.

   $ oc get secrets/signing-key -n openshift-service-ca \
        -o template='{{index .data "tls.crt"}}' \
        | base64 --decode \
        | openssl x509 -noout -enddate

2. Manually rotate the service CA. This process generates a new service CA which will be used to sign the new service certificates.

   $ oc delete secret/signing-key -n openshift-service-ca

3. To apply the new certificates to all services, restart all the pods in your cluster. This command ensures that all services use the updated certificates.

   $ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
         do oc delete pods --all -n $I; \
         sleep 1; \
         done

   Warning

   This command will cause a service interruption, as it goes through and deletes every running pod in every namespace. These pods will automatically restart after they are deleted.

Procedure

1. Create a config map that includes the root CA certificate used to sign the wildcard certificate:

   $ oc create configmap custom-ca \
        --from-file=ca-bundle.crt=</path/to/example-ca.crt> \1
        -n openshift-config

   1

   </path/to/example-ca.crt> is the path to the CA certificate bundle on your local file system.

2. Update the cluster-wide proxy configuration with the newly created config map:

   $ oc patch proxy/cluster \
        --type=merge \
        --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'