Chapter 9. Managing TLS certificates

1. Generates a new CA certificate, but retains the existing key.

   The new certificate replaces the old one with the name ca.crt within the corresponding Secret.

2. Generates new client certificates (for ZooKeeper nodes, Kafka brokers, and the Entity Operator).

   This is not strictly necessary because the signing key has not changed, but it keeps the validity period of the client certificate in sync with the CA certificate.

3. Restarts ZooKeeper nodes so that they will trust the new CA certificate and use the new client certificates.
4. Restarts Kafka brokers so that they will trust the new CA certificate and use the new client certificates.

5. Restarts the Topic and User Operators so that they will trust the new CA certificate and use the new client certificates.

   User certificates are signed by the clients CA. User certificates generated by the User Operator are renewed when the clients CA is renewed.

Procedure

1. Apply the strimzi.io/force-renew annotation to the Secret that contains the CA certificate that you want to renew.

   Table 9.13. Annotation for the Secret that forces renewal of certificates

   Certificate	Secret	Annotate command
   Cluster CA	KAFKA-CLUSTER-NAME-cluster-ca-cert	oc annotate secret KAFKA-CLUSTER-NAME-cluster-ca-cert strimzi.io/force-renew=true
   Clients CA	KAFKA-CLUSTER-NAME-clients-ca-cert	oc annotate secret KAFKA-CLUSTER-NAME-clients-ca-cert strimzi.io/force-renew=true

   At the next reconciliation the Cluster Operator will generate a new CA certificate for the Secret that you annotated. If maintenance time windows are configured, the Cluster Operator will generate the new CA certificate at the first reconciliation within the next maintenance time window.

   Client applications must reload the cluster and clients CA certificates that were renewed by the Cluster Operator.

2. Check the period the CA certificate is valid:

   For example, using an openssl command:

   oc get secret CA-CERTIFICATE-SECRET -o 'jsonpath={.data.CA-CERTIFICATE}' | base64 -d | openssl x509 -subject -issuer -startdate -enddate -noout

   CA-CERTIFICATE-SECRET is the name of the Secret, which is KAFKA-CLUSTER-NAME-cluster-ca-cert for the cluster CA certificate and KAFKA-CLUSTER-NAME-clients-ca-cert for the clients CA certificate.

   CA-CERTIFICATE is the name of the CA certificate, such as jsonpath={.data.ca\.crt}.

   The command returns a notBefore and notAfter date, which is the validity period for the CA certificate.

   For example, for a cluster CA certificate:

   subject=O = io.strimzi, CN = cluster-ca v0
   issuer=O = io.strimzi, CN = cluster-ca v0
   notBefore=Jun 30 09:43:54 2020 GMT
   notAfter=Jun 30 09:43:54 2021 GMT

3. Delete old certificates from the Secret.

   When components are using the new certificates, older certificates might still be active. Delete the old certificates to remove any potential security risk.

Procedure

1. Create a new secret that contains the CA certificate.

   Client secret creation with a certificate in PEM format only

   oc create secret generic <cluster_name>-clients-ca-cert --from-file=ca.crt=ca.crt

   Cluster secret creation with certificates in PEM and PKCS #12 format

   oc create secret generic <cluster_name>-cluster-ca-cert \
     --from-file=ca.crt=ca.crt \
     --from-file=ca.p12=ca.p12 \
     --from-literal=ca.password=P12-PASSWORD

   Replace <cluster_name> with the name of your Kafka cluster.

2. Create a new secret that contains the private key.

   oc create secret generic CA-KEY-SECRET --from-file=ca.key=ca.key

3. Label the secrets.

   oc label secret CA-CERTIFICATE-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=<cluster_name>

   oc label secret CA-KEY-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=<cluster_name>

   • Label strimzi.io/kind=Kafka identifies the Kafka custom resource.
   • Label strimzi.io/cluster=<cluster_name> identifies the Kafka cluster.

4. Annotate the secrets

   oc annotate secret CA-CERTIFICATE-SECRET strimzi.io/ca-cert-generation=CA-CERTIFICATE-GENERATION

   oc annotate secret CA-KEY-SECRET strimzi.io/ca-key-generation=CA-KEY-GENERATION

   • Annotation strimzi.io/ca-cert-generation=CA-CERTIFICATE-GENERATION defines the generation of a new CA certificate.

   • Annotation strimzi.io/ca-key-generation=CA-KEY-GENERATION defines the generation of a new CA key.

     Start from 0 (zero) as the incremental value (strimzi.io/ca-cert-generation=0) for your own CA certificate. Set a higher incremental value when you renew the certificates.

5. Create the Kafka resource for your cluster, configuring either the Kafka.spec.clusterCa or the Kafka.spec.clientsCa object to not use generated CAs.

   Example fragment Kafka resource configuring the cluster CA to use certificates you supply for yourself

   kind: Kafka
   version: kafka.strimzi.io/v1beta2
   spec:
     # ...
     clusterCa:
       generateCertificateAuthority: false

Procedure

1. Update the Secret for the CA certificate.

   Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

   oc edit secret <ca_certificate_secret_name>

   <ca_certificate_secret_name> is the name of the Secret, which is <kafka_cluster_name>-cluster-ca-cert for the cluster CA certificate and <kafka_cluster_name>-clients-ca-cert for the clients CA certificate.

   The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

   Example secret configuration for a cluster CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "0" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   Current base64-encoded CA certificate

   2

   Current CA certificate generation annotation value

2. Encode your new CA certificate into base64.

   cat <path_to_new_certificate> | base64

3. Update the CA certificate.

   Copy the base64-encoded CA certificate from the previous step as the value for the ca.crt property under data.

4. Increase the value of the CA certificate generation annotation.

   Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

   When AMQ Streams generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

5. Save the secret with the new CA certificate and certificate generation annotation value.

   Example secret configuration updated with a new CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "1" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   New base64-encoded CA certificate

   2

   New CA certificate generation annotation value

Procedure

1. Pause the reconciliation of the Kafka custom resource.

   1. Annotate the custom resource in OpenShift, setting the pause-reconciliation annotation to true:

      oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="true"

      For example, for a Kafka custom resource named my-cluster:

      oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"

   2. Check that the status conditions of the custom resource show a change to ReconciliationPaused:

      oc describe Kafka <name_of_custom_resource>

      The type condition changes to ReconciliationPaused at the lastTransitionTime.

2. Update the Secret for the CA certificate.

   1. Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

      oc edit secret <ca_certificate_secret_name>

      <ca_certificate_secret_name> is the name of the Secret, which is KAFKA-CLUSTER-NAME-cluster-ca-cert for the cluster CA certificate and KAFKA-CLUSTER-NAME-clients-ca-cert for the clients CA certificate.

      The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      Current base64-encoded CA certificate

      2

      Current CA certificate generation annotation value

   2. Rename the current CA certificate to retain it.

      Rename the current ca.crt property under data as ca-<date>.crt, where <date> is the certificate expiry date in the format YEAR-MONTH-DAYTHOUR-MINUTE-SECONDZ. For example ca-2022-01-26T17-32-00Z.crt:. Leave the value for the property as it is to retain the current CA certificate.

   3. Encode your new CA certificate into base64.

      cat <path_to_new_certificate> | base64

   4. Update the CA certificate.

      Create a new ca.crt property under data and copy the base64-encoded CA certificate from the previous step as the value for ca.crt property.

   5. Increase the value of the CA certificate generation annotation.

      Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

      When AMQ Streams generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

   6. Save the secret with the new CA certificate and certificate generation annotation value.

      Example secret configuration updated with a new CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
        ca-2022-01-26T17-32-00Z.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 2
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "1" 3
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      New base64-encoded CA certificate

      2

      Old base64-encoded CA certificate

      3

      New CA certificate generation annotation value

3. Update the Secret for the CA key used to sign your new CA certificate.

   1. Edit the existing secret to add the new CA key and update the key generation annotation value.

      oc edit secret <ca_key_name>

      <ca_key_name> is the name of CA key, which is <kafka_cluster_name>-cluster-ca for the cluster CA key and <kafka_cluster_name>-clients-ca for the clients CA key.

      The following example shows a secret for a cluster CA key that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA key

      apiVersion: v1
      kind: Secret
      data:
        ca.key: SA1cKF1GFDzOIiPOIUQBHDNFGDFS... 1
      metadata:
        annotations:
          strimzi.io/ca-key-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca
        #...
      type: Opaque

      1

      Current base64-encoded CA key

      2

      Current CA key generation annotation value

   2. Encode the CA key into base64.

      cat <path_to_new_key> | base64

   3. Update the CA key.

      Copy the base64-encoded CA key from the previous step as the value for the ca.key property under data.

   4. Increase the value of the CA key generation annotation.

      Update the strimzi.io/ca-key-generation annotation with a higher incremental value. For example, change strimzi.io/ca-key-generation=0 to strimzi.io/ca-key-generation=1. If the Secret is missing the annotation, it is treated as 0, so add the annotation with a value of 1.

      When AMQ Streams generates certificates, the key generation annotation is automatically incremented by the Cluster Operator. For your own CA certificates together with a new CA key, set the annotation with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates and keys. The strimzi.io/ca-key-generation has to be incremented on each CA certificate renewal.

4. Save the secret with the new CA key and key generation annotation value.

   Example secret configuration updated with a new CA key

   apiVersion: v1
   kind: Secret
   data:
     ca.key: AB0cKF1GFDzOIiPOIUQWERZJQ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-key-generation: "1" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca
     #...
   type: Opaque

   1

   New base64-encoded CA key

   2

   New CA key generation annotation value

5. Resume from the pause.

   To resume the Kafka custom resource reconciliation, set the pause-reconciliation annotation to false.

   oc annotate --overwrite Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="false"

   You can also do the same by removing the pause-reconciliation annotation.

   oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation-