SupportConsoleDevelopersStart a trialContact

Chapter 9. Redeploying Certificates

9.1. Overview

OpenShift Enterprise uses certificates to provide secure connections for the following components:

  • masters (API server and controllers)
  • etcd
  • nodes
  • registry
  • router

You can use Ansible playbooks provided with the installer to automate checking expiration dates for cluster certificates. Playbooks are also provided to automate backing up and redeploying these certificates, which can fix common certificate errors.

Possible use cases for redeploying certificates include:

  • The installer detected the wrong host names and the issue was identified too late.
  • The certificates are expired and you need to update them.
  • You have a new CA and would like to create certificates using it instead.

9.2. Checking Certificate Expirations

You can use the installer to warn you about any certificates expiring within a configurable window of days and notify you about any certificates that have already expired. Certificate expiry playbooks use the Ansible role openshift_certificate_expiry.

Certificates examined by the role include:

  • Master and node service certificates
  • Router and registry service certificates from etcd secrets
  • Master, node, router, registry, and kubeconfig files for cluster-admin users
  • etcd certificates (including embedded)

9.2.1. Role Variables

The openshift_certificate_expiry role uses the following variables:

Table 9.1. Core Variables
Variable NameDefault ValueDescription

openshift_certificate_expiry_config_base

/etc/origin

Base OpenShift Enterprise configuration directory.

openshift_certificate_expiry_warning_days

30

Flag certificates that will expire in this many days from now.

openshift_certificate_expiry_show_all

no

Include healthy (non-expired and non-warning) certificates in results.

Table 9.2. Optional Variables
Variable NameDefault ValueDescription

openshift_certificate_expiry_generate_html_report

no

Generate an HTML report of the expiry check results.

openshift_certificate_expiry_html_report_path

/tmp/cert-expiry-report.html

The full path for saving the HTML report.

openshift_certificate_expiry_save_json_results

no

Save expiry check results as a JSON file.

openshift_certificate_expiry_json_results_path

/tmp/cert-expiry-report.json

The full path for saving the JSON report.

9.2.2. Running Certificate Expiration Playbooks

The OpenShift Enterprise installer provides a set of example certificate expiration playbooks, using different sets of configuration for the openshift_certificate_expiry role.

These playbooks must be used with an inventory file that is representative of the cluster. For best results, run ansible-playbook with the -v option.

Using the easy-mode.yaml example playbook, you can try the role out before tweaking it to your specifications as needed. This playbook:

  • Produces JSON and stylized HTML reports in /tmp/.
  • Sets the warning window very large, so you will almost always get results back.
  • Includes all certificates (healthy or not) in the results.

easy-mode.yaml Playbook

- name: Check cert expirys
  hosts: nodes:masters:etcd
  become: yes
  gather_facts: no
  vars:
    openshift_certificate_expiry_warning_days: 1500
    openshift_certificate_expiry_save_json_results: yes
    openshift_certificate_expiry_generate_html_report: yes
    openshift_certificate_expiry_show_all: yes
  roles:
    - role: openshift_certificate_expiry

To run the easy-mode.yaml playbook: 

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/easy-mode.yaml
Other Example Playbooks

The other example playbooks are also available to run directly out of the /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/ directory.

Table 9.3. Other Example Playbooks
File NameUsage

default.yaml

Produces the default behavior of the openshift_certificate_expiry role.

html_and_json_default_paths.yaml

Generates HTML and JSON artifacts in their default paths.

longer_warning_period.yaml

Changes the expiration warning window to 1500 days.

longer-warning-period-json-results.yaml

Changes the expiration warning window to 1500 days and saves the results as a JSON file.

To run any of these example playbooks: 

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/<playbook>

9.2.3. Output Formats

As noted above, there are two ways to format your check report. In JSON format for machine parsing, or as a stylized HTML page for easy skimming.

HTML Report

An example of an HTML report is provided with the installer. You can open the following file in your browser to view it:

/usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.html

JSON Report

There are two top-level keys in the saved JSON results: data and summary.

The data key is a hash where the keys are the names of each host examined and the values are the check results for the certificates identified on each respective host.

The summary key is a hash that summarizes the total number of certificates:

  • examined on the entire cluster
  • that are OK
  • expiring within the configured warning window
  • already expired

For an example of the full JSON report, see /usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.json.

The summary from the JSON data can be easily checked for warnings or expirations using a variety of command-line tools. For example, using grep you can look for the word summary and print out the two lines after the match (-A2): 

$ grep -A2 summary /tmp/cert-expiry-report.json
    "summary": {
        "warning": 16,
        "expired": 0

If available, the jq tool can also be used to pick out specific values. The first two examples below show how to select just one value, either warning or expired. The third example shows how to select both values at once: 

$ jq '.summary.warning' /tmp/cert-expiry-report.json
16

$ jq '.summary.expired' /tmp/cert-expiry-report.json
0

$ jq '.summary.warning,.summary.expired' /tmp/cert-expiry-report.json
16
0

9.3. Redeploying Certificates

Use the following playbooks to redeploy master, etcd, node, registry, and router certificates on all relevant hosts. You can redeploy all of them at once using the current CA, redeploy certificates for specific components only, or redeploy a newly generated or custom CA on its own.

Just like the certificate expiry playbooks, these playbooks must be run with an inventory file that is representative of the cluster.

In particular, the inventory must specify or override all host names and IP addresses set via the following variables such that they match the current cluster configuration:

  • openshift_hostname
  • openshift_public_hostname
  • openshift_ip
  • openshift_public_ip
  • openshift_master_cluster_hostname
  • openshift_master_cluster_public_hostname

9.3.1. Redeploying All Certificates Using the Current OpenShift Enterprise and etcd CA

The redeploy-certificates.yml playbook does not regenerate the OpenShift Enterprise CA certificate. New master, etcd, node, registry, and router certificates are created using the current CA certificate to sign new certificates.

This also includes serial restarts of:

  • etcd
  • master services
  • node services

To redeploy master, etcd, and node certificates using the current OpenShift Enterprise CA, run this playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-certificates.yml

9.3.2. Redeploying a New or Custom OpenShift Enterprise CA

The redeploy-openshift-ca.yml playbook redeploys the OpenShift Enterprise CA certificate by generating a new CA certificate and distributing an updated bundle to all components including client kubeconfig files and the node’s database of trusted CAs (the CA-trust).

This also includes serial restarts of:

  • master services
  • node services
  • docker

Additionally, you can specify a custom CA certificate when redeploying certificates instead of relying on a CA generated by OpenShift Enterprise.

When the master services are restarted, the registry and routers can continue to communicate with the master without being redeployed because the master’s serving certificate is the same, and the CA the registry and routers have are still valid.

To redeploy a newly generated or custom CA:

  1. If you want to use a custom CA, set the following variable in your inventory file: 

    # Configure custom ca certificate
# NOTE: CA certificate will not be replaced with existing clusters.
# This option may only be specified when creating a new cluster or
# when redeploying cluster certificates with the redeploy-certificates
# playbook.
openshift_master_ca_certificate={'certfile': '</path/to/ca.crt>', 'keyfile': '</path/to/ca.key>'}

    If you do not set the above, then the current CA will be regenerated in the next step.

  2. Run the redeploy-openshift-ca.yml playbook, specifying your inventory file: 

    $ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-openshift-ca.yml

With the new OpenShift Enterprise CA in place, you can then use the redeploy-certificates.yml playbook at your discretion whenever you want to redeploy certificates signed by the new CA on all components.

9.3.3. Redeploying a New etcd CA

The redeploy-etcd-ca.yml playbook redeploys the etcd CA certificate by generating a new CA certificate and distributing an updated bundle to all etcd peers and master clients.

This also includes serial restarts of:

  • etcd
  • master services

To redeploy a newly generated etcd CA:

  1. Run the redeploy-etcd-ca.yml playbook, specifying your inventory file: 

    $ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-etcd-ca.yml

With the new etcd CA in place, you can then use the redeploy-etcd-certificates.yml playbook at your discretion whenever you want to redeploy certificates signed by the new etcd CA on etcd peers and master clients. Alternatively, you can use the redeploy-certificates.yml playbook to redeploy certificates for OpenShift Enterprise components in addition to etcd peers and master clients.

9.3.4. Redeploying Master Certificates Only

The redeploy-master-certificates.yml playbook only redeploys master certificates. This also includes serial restarts of master services.

To redeploy master certificates, run this playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-master-certificates.yml

9.3.5. Redeploying etcd Certificates Only

The redeploy-etcd-certificates.yml playbook only redeploys etcd certificates including master client certificates.

This also include serial restarts of:

  • etcd
  • master services.

To redeploy etcd certificates, run this playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-etcd-certificates.yml

9.3.6. Redeploying Node Certificates Only

The redeploy-node-certificates.yml playbook only redeploys node certificates. This also include serial restarts of node services.

To redeploy node certificates, run this playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-node-certificates.yml

9.3.7. Redeploying Registry or Router Certificates Only

The redeploy-registry-certificates.yml and redeploy-router-certificates.yml playbooks replace installer-created certificates for the registry and router. If custom certificates are in use for these components, see Redeploying Custom Registry or Router Certificates to replace them manually.

9.3.7.1. Redeploying Registry Certificates Only

To redeploy registry certificates, run the following playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-registry-certificates.yml

9.3.7.2. Redeploying Router Certificates Only

To redeploy router certificates, run the following playbook, specifying your inventory file: 

$ ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-router-certificates.yml

9.3.8. Redeploying Custom Registry or Router Certificates

When nodes are evacuated due to a redeployed CA, registry and router pods are restarted. If the registry and router certificates were not also redeployed with the new CA, this can cause outages because they cannot reach the masters using their old certificates.

The playbooks for redeploying certificates cannot redeploy custom registry or router certificates, so to address this issue, you can manually redeploy the registry and router certificates.

9.3.8.1. Redeploying Registry Certificates Manually

To redeploy registry certificates manually, you must add new registry certificates to a secret named registry-certificates, then redeploy the registry:

  1. Switch to the default project for the remainder of these steps: 

    $ oc project default

  2. If your registry was initially created on OpenShift Enterprise 3.1 or earlier, it may still be using environment variables to store certificates (which has been deprecated in favor of using secrets).

    1. Run the following and look for the OPENSHIFT_CA_DATA, OPENSHIFT_CERT_DATA, OPENSHIFT_KEY_DATA environment variables: 

      $ oc env dc/docker-registry --list

    2. If they do not exist, skip this step. If they do, create the following ClusterRoleBinding: 

      $ cat <<EOF |
apiVersion: v1
groupNames: null
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: registry-registry-role
roleRef:
  kind: ClusterRole
  name: system:registry
subjects:
- kind: ServiceAccount
  name: registry
  namespace: default
userNames:
- system:serviceaccount:default:registry
EOF
oc create -f -

      Then, run the following to remove the environment variables: 

      $ oc env dc/docker-registry OPENSHIFT_CA_DATA- OPENSHIFT_CERT_DATA- OPENSHIFT_KEY_DATA- OPENSHIFT_MASTER-

  3. Set the following environment variables locally to make later commands less complex: 

    $ REGISTRY_IP=`oc get service docker-registry -o jsonpath='{.spec.clusterIP}'`
$ REGISTRY_HOSTNAME=`oc get route/docker-registry -o jsonpath='{.spec.host}'`

  4. Create new registry certificates: 

    $ oc adm ca create-server-cert \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --hostnames=$REGISTRY_IP,docker-registry.default.svc.cluster.local,$REGISTRY_HOSTNAME \
    --cert=/etc/origin/master/registry.crt \
    --key=/etc/origin/master/registry.key \
    --signer-serial=/etc/origin/master/ca.serial.txt

  5. Update the registry-certificates secret with the new registry certificates: 

    $ oc secret new registry-certificates \
    /etc/origin/master/registry.crt \
    /etc/origin/master/registry.key \
    -o json | oc replace -f -

  6. Redeploy the registry: 

    $ oc deploy dc/docker-registry --latest

9.3.8.2. Redeploying Router Certificates Manually

When routers are initially deployed, an annotation is added to the router’s service that automatically creates a service serving certificate secret.

To redeploy router certificates manually, that service serving certificate can be triggered to be recreated by deleting the secret, adding a new secret, then redeploying the router:

  1. Switch to the default project for the remainder of these steps: 

    $ oc project default

  2. If your router was initially created on OpenShift Enterprise 3.1 or earlier, it may still be using environment variables to store certificates (which has been deprecated in favor of using service serving certificate secret).

    1. Run the following and look for the OPENSHIFT_CA_DATA, OPENSHIFT_CERT_DATA, OPENSHIFT_KEY_DATA environment variables: 

      $ oc env dc/router --list

    2. If they do not exist, skip this step. If they do, create the following ClusterRoleBinding: 

      $ cat <<EOF |
apiVersion: v1
groupNames: null
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: router-router-role
roleRef:
  kind: ClusterRole
  name: system:router
subjects:
- kind: ServiceAccount
  name: router
  namespace: default
userNames:
- system:serviceaccount:default:router
EOF
oc create -f -

      Then, run the following to remove the environment variables: 

      $ oc env dc/router OPENSHIFT_CA_DATA- OPENSHIFT_CERT_DATA- OPENSHIFT_KEY_DATA- OPENSHIFT_MASTER-

  3. To obtain a new certificate, run: 

    # cd /root
# mkdir cert ; cd cert
# CA=/etc/origin/master
# oadm ca create-server-cert --signer-cert=$CA/ca.crt --signer-key=$CA/ca.key \
--signer-serial=$CA/ca.serial.txt --hostnames=hostnames.for.the.certificate \
--cert=router.crt --key=router.key

    This will generate:

    • A new certificate (router.crt in this example).
    • Its corresponding private key (router.key in this example).
    • A copy of the signing certificate authority (CA) certificate chain ($CA/ca.crt in this example; it can contain more than one certificate if intermediate CAs are involved).

  4. Create a new file by concatenating these three files in that specific order: 

    # cat router.crt router.key $CA/ca.crt > router.pem

  5. To back up the old certificate: 

    # oc export secret router-certs > ~/old-router-certs-secret.yaml  1
    1
    router-certs is the default name of the secret, as it is the name used by the oadm router --default-cert option.

  6. Delete the router-certs secret: 

    $ oc delete secret router-certs

  7. Create a new secret. 

    # oc secrets new router-certs tls.crt=router.pem tls.key=router.key \ 1
       -o json --type='kubernetes.io/tls' --confirm | \
   oc replace -f -
    1
    router.pem contains the certificate, its key, and its signing CA.

  8. Redeploy the router: 

    $ oc deploy dc/router --latest
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.