Chapter 16. Managing TLS certificates

1. Generates a new CA certificate, but retains the existing key.

   The new certificate replaces the old one with the name ca.crt within the corresponding Secret.

2. Generates new client certificates (for ZooKeeper nodes, Kafka brokers, and the Entity Operator).

   This is not strictly necessary because the signing key has not changed, but it keeps the validity period of the client certificate in sync with the CA certificate.

3. Restarts ZooKeeper nodes so that they will trust the new CA certificate and use the new client certificates.
4. Restarts Kafka brokers so that they will trust the new CA certificate and use the new client certificates.

5. Restarts the Topic and User Operators so that they will trust the new CA certificate and use the new client certificates.

   User certificates are signed by the clients CA. User certificates generated by the User Operator are renewed when the clients CA is renewed.

Procedure

1. Apply the strimzi.io/force-renew annotation to the secret that contains the CA certificate that you want to renew.

   Renewing the Cluster CA secret

   oc annotate secret my-cluster-cluster-ca-cert -n my-project strimzi.io/force-renew="true"

   Renewing the Clients CA secret

   oc annotate secret my-cluster-clients-ca-cert -n my-project strimzi.io/force-renew="true"

2. At the next reconciliation, the Cluster Operator generates new certificates.

   If maintenance time windows are configured, the Cluster Operator generates the new CA certificate at the first reconciliation within the next maintenance time window.

3. Check the period of validity for the new CA certificates.

   Checking the period of validity for the new cluster CA certificate

   oc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

   Checking the period of validity for the new clients CA certificate

   oc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

   The command returns a notBefore and notAfter date, which is the valid start and end date for the CA certificate.

4. Update client configurations to trust the new cluster CA certificate.

   See:

   • Section 16.4, “Configuring internal clients to trust the cluster CA”
   • Section 16.5, “Configuring external clients to trust the cluster CA”

Procedure

1. Delete the secret containing the expired CA certificate.

   Deleting the Cluster CA secret

   oc delete secret my-cluster-cluster-ca-cert -n my-project

   Deleting the Clients CA secret

   oc delete secret my-cluster-clients-ca-cert -n my-project

2. Wait for the Cluster Operator to generate new certificates.

   • A new CA cluster certificate to verify the identity of the Kafka brokers is created in a secret of the same name (my-cluster-cluster-ca-cert).
   • A new CA clients certificate to verify the identity of Kafka users is created in a secret of the same name (my-cluster-clients-ca-cert).

3. Check the period of validity for the new CA certificates.

   Checking the period of validity for the new cluster CA certificate

   oc get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

   Checking the period of validity for the new clients CA certificate

   oc get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

   The command returns a notBefore and notAfter date, which is the valid start and end date for the CA certificate.

4. Delete the component pods and secrets that use the CA certificates.

   1. Delete the ZooKeeper secret.
   2. Wait for the Cluster Operator to detect the missing ZooKeeper secret and recreate it.
   3. Delete all ZooKeeper pods.
   4. Delete the Kafka secret.
   5. Wait for the Cluster Operator to detect the missing Kafka secret and recreate it.
   6. Delete all Kafka pods.

   If you are only recovering the clients CA certificate, you only need to delete the Kafka secret and pods.

   You can use the following oc command to find resources and also verify that they have been removed.

   oc get <resource_type> --all-namespaces | grep <kafka_cluster_name>

   Replace <resource_type> with the type of the resource, such as Pod or Secret.

5. Wait for the Cluster Operator to detect the missing Kafka and ZooKeeper pods and recreate them with the updated CA certificates.

   On reconciliation, the Cluster Operator automatically updates other components to trust the new CA certificates.

6. Verify that there are no issues related to certificate validation in the Cluster Operator log.

7. Update client configurations to trust the new cluster CA certificate.

   See:

   • Section 16.4, “Configuring internal clients to trust the cluster CA”
   • Section 16.5, “Configuring external clients to trust the cluster CA”

Procedure

1. Create a new secret that contains the CA certificate.

   Client secret creation with a certificate in PEM format only

   oc create secret generic <cluster_name>-clients-ca-cert --from-file=ca.crt=ca.crt

   Cluster secret creation with certificates in PEM and PKCS #12 format

   oc create secret generic <cluster_name>-cluster-ca-cert \
     --from-file=ca.crt=ca.crt \
     --from-file=ca.p12=ca.p12 \
     --from-literal=ca.password=P12-PASSWORD

   Replace <cluster_name> with the name of your Kafka cluster.

2. Create a new secret that contains the private key.

   oc create secret generic <ca_key_secret> --from-file=ca.key=ca.key

3. Label the secrets.

   oc label secret <ca_certificate_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"

   oc label secret <ca_key_secret> strimzi.io/kind=Kafka strimzi.io/cluster="<cluster_name>"

   • Label strimzi.io/kind=Kafka identifies the Kafka custom resource.
   • Label strimzi.io/cluster="<cluster_name>" identifies the Kafka cluster.

4. Annotate the secrets

   oc annotate secret <ca_certificate_secret> strimzi.io/ca-cert-generation="<ca_certificate_generation>"

   oc annotate secret <ca_key_secret> strimzi.io/ca-key-generation="<ca_key_generation>"

   • Annotation strimzi.io/ca-cert-generation="<ca_certificate_generation>" defines the generation of a new CA certificate.

   • Annotation strimzi.io/ca-key-generation="<ca_key_generation>" defines the generation of a new CA key.

     Start from 0 (zero) as the incremental value (strimzi.io/ca-cert-generation=0) for your own CA certificate. Set a higher incremental value when you renew the certificates.

5. Create the Kafka resource for your cluster, configuring either the Kafka.spec.clusterCa or the Kafka.spec.clientsCa object to not use generated CAs.

   Example fragment Kafka resource configuring the cluster CA to use certificates you supply for yourself

   kind: Kafka
   version: kafka.strimzi.io/v1beta2
   spec:
     # ...
     clusterCa:
       generateCertificateAuthority: false

Procedure

1. Update the Secret for the CA certificate.

   Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

   oc edit secret <ca_certificate_secret_name>

   <ca_certificate_secret_name> is the name of the Secret, which is <kafka_cluster_name>-cluster-ca-cert for the cluster CA certificate and <kafka_cluster_name>-clients-ca-cert for the clients CA certificate.

   The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

   Example secret configuration for a cluster CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "0" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   Current base64-encoded CA certificate

   2

   Current CA certificate generation annotation value

2. Encode your new CA certificate into base64.

   cat <path_to_new_certificate> | base64

3. Update the CA certificate.

   Copy the base64-encoded CA certificate from the previous step as the value for the ca.crt property under data.

4. Increase the value of the CA certificate generation annotation.

   Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

   When Streams for Apache Kafka generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

5. Save the secret with the new CA certificate and certificate generation annotation value.

   Example secret configuration updated with a new CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "1" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   New base64-encoded CA certificate

   2

   New CA certificate generation annotation value

Procedure

1. Pause the reconciliation of the Kafka custom resource.

   1. Annotate the custom resource in OpenShift, setting the pause-reconciliation annotation to true:

      oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="true"

      For example, for a Kafka custom resource named my-cluster:

      oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"

   2. Check that the status conditions of the custom resource show a change to ReconciliationPaused:

      oc describe Kafka <name_of_custom_resource>

      The type condition changes to ReconciliationPaused at the lastTransitionTime.

2. Check the settings for the generateCertificateAuthority properties in your Kafka custom resource.

   If a property is set to false, a CA certificate is not generated by the Cluster Operator. You require this setting if you are using your own certificates.

3. If needed, edit the existing Kafka custom resource and set the generateCertificateAuthority properties to false.

   oc edit Kafka <name_of_custom_resource>

   The following example shows a Kafka custom resource with both cluster and clients CA certificates generation delegated to the user.

   Example Kafka configuration using your own CA certificates

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   # ...
   spec:
   # ...
     clusterCa:
       generateCertificateAuthority: false 1
     clientsCa:
       generateCertificateAuthority: false 2
   # ...

   1

   Use your own cluster CA

   2

   Use your own clients CA

4. Update the Secret for the CA certificate.

   1. Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

      oc edit secret <ca_certificate_secret_name>

      <ca_certificate_secret_name> is the name of the Secret, which is <kafka_cluster_name>-cluster-ca-cert for the cluster CA certificate and <kafka_cluster_name>-clients-ca-cert for the clients CA certificate.

      The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      Current base64-encoded CA certificate

      2

      Current CA certificate generation annotation value

   2. Rename the current CA certificate to retain it.

      Rename the current ca.crt property under data as ca-<date>.crt, where <date> is the certificate expiry date in the format YEAR-MONTH-DAYTHOUR-MINUTE-SECONDZ. For example ca-2023-01-26T17-32-00Z.crt:. Leave the value for the property as it is to retain the current CA certificate.

   3. Encode your new CA certificate into base64.

      cat <path_to_new_certificate> | base64

   4. Update the CA certificate.

      Create a new ca.crt property under data and copy the base64-encoded CA certificate from the previous step as the value for ca.crt property.

   5. Increase the value of the CA certificate generation annotation.

      Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

      When Streams for Apache Kafka generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

   6. Save the secret with the new CA certificate and certificate generation annotation value.

      Example secret configuration updated with a new CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
        ca-2023-01-26T17-32-00Z.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 2
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "1" 3
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      New base64-encoded CA certificate

      2

      Old base64-encoded CA certificate

      3

      New CA certificate generation annotation value

5. Update the Secret for the CA key used to sign your new CA certificate.

   1. Edit the existing secret to add the new CA key and update the key generation annotation value.

      oc edit secret <ca_key_name>

      <ca_key_name> is the name of CA key, which is <kafka_cluster_name>-cluster-ca for the cluster CA key and <kafka_cluster_name>-clients-ca for the clients CA key.

      The following example shows a secret for a cluster CA key that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA key

      apiVersion: v1
      kind: Secret
      data:
        ca.key: SA1cKF1GFDzOIiPOIUQBHDNFGDFS... 1
      metadata:
        annotations:
          strimzi.io/ca-key-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca
        #...
      type: Opaque

      1

      Current base64-encoded CA key

      2

      Current CA key generation annotation value

   2. Encode the CA key into base64.

      cat <path_to_new_key> | base64

   3. Update the CA key.

      Copy the base64-encoded CA key from the previous step as the value for the ca.key property under data.

   4. Increase the value of the CA key generation annotation.

      Update the strimzi.io/ca-key-generation annotation with a higher incremental value. For example, change strimzi.io/ca-key-generation=0 to strimzi.io/ca-key-generation=1. If the Secret is missing the annotation, it is treated as 0, so add the annotation with a value of 1.

      When Streams for Apache Kafka generates certificates, the key generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates together with a new CA key, set the annotation with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates and keys. The strimzi.io/ca-key-generation has to be incremented on each CA certificate renewal.

   5. Save the secret with the new CA key and key generation annotation value.

      Example secret configuration updated with a new CA key

      apiVersion: v1
      kind: Secret
      data:
        ca.key: AB0cKF1GFDzOIiPOIUQWERZJQ0F... 1
      metadata:
        annotations:
          strimzi.io/ca-key-generation: "1" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca
        #...
      type: Opaque

      1

      New base64-encoded CA key

      2

      New CA key generation annotation value

6. Resume from the pause.

   To resume the Kafka custom resource reconciliation, set the pause-reconciliation annotation to false.

   oc annotate --overwrite Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="false"

   You can also do the same by removing the pause-reconciliation annotation.

   oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation-

   On the next reconciliation, the Cluster Operator performs a rolling update of ZooKeeper, Kafka, and other components to trust the new CA certificate. When the rolling update is complete, the Cluster Operator will start a new one to generate new server certificates signed by the new CA key.

   If maintenance time windows are configured, the Cluster Operator will roll the pods at the first reconciliation within the next maintenance time window.

7. Wait until the rolling updates to move to the new CA certificate are complete.

8. Remove any outdated certificates from the secret configuration to ensure that the cluster no longer trusts them.

   oc edit secret <ca_certificate_secret_name>

   Example secret configuration with the old certificate removed

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F...
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "1"
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

9. Start a manual rolling update of your cluster to pick up the changes made to the secret configuration.

   See Section 29.2, “Starting rolling updates of Kafka and other operands using annotations”.