Chapter 11. Managing TLS certificates

Procedure

1. Replace the CA certificate generated by the Cluster Operator.

   1. Delete the existing secret.

      oc delete secret CA-CERTIFICATE-SECRET

      CA-CERTIFICATE-SECRET is the name of the Secret:

      • CLUSTER-NAME-cluster-ca-cert for the cluster CA certificate
      • CLUSTER-NAME-clients-ca-cert for the clients CA certificate

      Replace CLUSTER-NAME with the name of your Kafka cluster.

      Ignore any "Not Exists" errors.

   2. Create the new secret.

      Client secret creation with a certificate in PEM format only

      oc create secret generic CLUSTER-NAME-clients-ca-cert --from-file=ca.crt=ca.crt

      Cluster secret creation with certificates in PEM and PKCS #12 format

      oc create secret generic CLUSTER-NAME-cluster-ca-cert \
        --from-file=ca.crt=ca.crt \
        --from-file=ca.p12=ca.p12 \
        --from-literal=ca.password=P12-PASSWORD

2. Replace the private key generated by the Cluster Operator.

   1. Delete the existing secret.

      oc delete secret CA-KEY-SECRET

      CA-KEY-SECRET is the name of CA key:

      • CLUSTER-NAME-cluster-ca for the cluster CA key
      • CLUSTER-NAME-clients-ca for the clients CA key

   2. Create the new secret.

      oc create secret generic CA-KEY-SECRET --from-file=ca.key=ca.key

3. Label the secrets.

   oc label secret CA-CERTIFICATE-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=CLUSTER-NAME

   oc label secret CA-KEY-SECRET strimzi.io/kind=Kafka strimzi.io/cluster=CLUSTER-NAME

   • Label strimzi.io/kind=Kafka identifies the Kafka custom resource.
   • Label strimzi.io/cluster=CLUSTER-NAME identifies the Kafka cluster.

4. Annotate the secrets

   oc annotate secret CA-CERTIFICATE-SECRET strimzi.io/ca-cert-generation=CA-CERTIFICATE-GENERATION

   oc annotate secret CA-KEY-SECRET strimzi.io/ca-key-generation=CA-KEY-GENERATION

   • Annotation strimzi.io/ca-cert-generation=CA-CERTIFICATE-GENERATION defines the generation of a new CA certificate.

   • Annotation strimzi.io/ca-key-generation=CA-KEY-GENERATION defines the generation of a new CA key.

     If you are replacing CA certificates automatically generated by the Cluster Operator, use the next higher incremental value from the existing annotation and follow the replacing CA keys procedure. If there are no CA certificates automatically generated by the Cluster Operator, start from 0 (zero) as the incremental value (strimzi.io/ca-cert-generation=0) for your own CA certificate. Set a higher incremental value when you renew the certificates.

5. Create the Kafka resource for your cluster, configuring either the Kafka.spec.clusterCa or the Kafka.spec.clientsCa object to not use generated CAs.

   Example fragment Kafka resource configuring the cluster CA to use certificates you supply for yourself

   kind: Kafka
   version: kafka.strimzi.io/v1beta2
   spec:
     # ...
     clusterCa:
       generateCertificateAuthority: false

1. Generate a new CA certificate, but retain the existing key. The new certificate replaces the old one with the name ca.crt within the corresponding Secret.
2. Generate new client certificates (for ZooKeeper nodes, Kafka brokers, and the Entity Operator). This is not strictly necessary because the signing key has not changed, but it keeps the validity period of the client certificate in sync with the CA certificate.
3. Restart ZooKeeper nodes so that they will trust the new CA certificate and use the new client certificates.
4. Restart Kafka brokers so that they will trust the new CA certificate and use the new client certificates.
5. Restart the Topic and User Operators so that they will trust the new CA certificate and use the new client certificates.

Procedure

1. Apply the strimzi.io/force-renew annotation to the Secret that contains the CA certificate that you want to renew.

   Table 11.13. Annotation for the Secret that forces renewal of certificates

   Certificate	Secret	Annotate command
   Cluster CA	KAFKA-CLUSTER-NAME-cluster-ca-cert	oc annotate secret KAFKA-CLUSTER-NAME-cluster-ca-cert strimzi.io/force-renew=true
   Clients CA	KAFKA-CLUSTER-NAME-clients-ca-cert	oc annotate secret KAFKA-CLUSTER-NAME-clients-ca-cert strimzi.io/force-renew=true

   At the next reconciliation the Cluster Operator will generate a new CA certificate for the Secret that you annotated. If maintenance time windows are configured, the Cluster Operator will generate the new CA certificate at the first reconciliation within the next maintenance time window.

   Client applications must reload the cluster and clients CA certificates that were renewed by the Cluster Operator.

2. Check the period the CA certificate is valid:

   For example, using an openssl command:

   oc get secret CA-CERTIFICATE-SECRET -o 'jsonpath={.data.CA-CERTIFICATE}' | base64 -d | openssl x509 -subject -issuer -startdate -enddate -noout

   CA-CERTIFICATE-SECRET is the name of the Secret, which is KAFKA-CLUSTER-NAME-cluster-ca-cert for the cluster CA certificate and KAFKA-CLUSTER-NAME-clients-ca-cert for the clients CA certificate.

   CA-CERTIFICATE is the name of the CA certificate, such as jsonpath={.data.ca\.crt}.

   The command returns a notBefore and notAfter date, which is the validity period for the CA certificate.

   For example, for a cluster CA certificate:

   subject=O = io.strimzi, CN = cluster-ca v0
   issuer=O = io.strimzi, CN = cluster-ca v0
   notBefore=Jun 30 09:43:54 2020 GMT
   notAfter=Jun 30 09:43:54 2021 GMT

3. Delete old certificates from the Secret.

   When components are using the new certificates, older certificates might still be active. Delete the old certificates to remove any potential security risk.

Procedure

1. Update the Secret for the CA certificate.

   Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

   oc edit secret <ca_certificate_secret_name>

   <ca_certificate_secret_name> is the name of the Secret, which is <kafka_cluster_name>-cluster-ca-cert for the cluster CA certificate and <kafka_cluster_name>-clients-ca-cert for the clients CA certificate.

   The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

   Example secret configuration for a cluster CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "0" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   Current base64-encoded CA certificate

   2

   Current CA certificate generation annotation value

2. Encode your new CA certificate into base64.

   cat <path_to_new_certificate> | base64

3. Update the CA certificate.

   Copy the base64-encoded CA certificate from the previous step as the value for the ca.crt property under data.

4. Increase the value of the CA certificate generation annotation.

   Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

   When AMQ Streams generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For manual renewal of your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

5. Save the secret with the new CA certificate and certificate generation annotation value.

   Example secret configuration updated with a new CA certificate

   apiVersion: v1
   kind: Secret
   data:
     ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-cert-generation: "1" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca-cert
     #...
   type: Opaque

   1

   New base64-encoded CA certificate

   2

   New CA certificate generation annotation value

Procedure

1. Pause the reconciliation of the Kafka custom resource.

   1. Annotate the custom resource in OpenShift, setting the pause-reconciliation annotation to true:

      oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation="true"

      For example, for a Kafka custom resource named my-cluster:

      oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"

   2. Check that the status conditions of the custom resource show a change to ReconciliationPaused:

      oc describe Kafka <name_of_custom_resource>

      The type condition changes to ReconciliationPaused at the lastTransitionTime.

2. Update the Secret for the CA certificate.

   1. Edit the existing secret to add the new CA certificate and update the certificate generation annotation value.

      oc edit secret <ca_certificate_secret_name>

      <ca_certificate_secret_name> is the name of the Secret, which is KAFKA-CLUSTER-NAME-cluster-ca-cert for the cluster CA certificate and KAFKA-CLUSTER-NAME-clients-ca-cert for the clients CA certificate.

      The following example shows a secret for a cluster CA certificate that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 1
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      Current base64-encoded CA certificate

      2

      Current CA certificate generation annotation value

   2. Rename the current CA certificate to retain it.

      Rename the current ca.crt property under data as ca-<date>.crt, where <date> is the certificate expiry date in the format YEAR-MONTH-DAYTHOUR-MINUTE-SECONDZ. For example ca-2022-01-26T17-32-00Z.crt:. Leave the value for the property as it is to retain the current CA certificate.

   3. Encode your new CA certificate into base64.

      cat <path_to_new_certificate> | base64

   4. Update the CA certificate.

      Create a new ca.crt property under data and copy the base64-encoded CA certificate from the previous step as the value for ca.crt property.

   5. Increase the value of the CA certificate generation annotation.

      Update the strimzi.io/ca-cert-generation annotation with a higher incremental value. For example, change strimzi.io/ca-cert-generation=0 to strimzi.io/ca-cert-generation=1. If the Secret is missing the annotation, the value is treated as 0, so add the annotation with a value of 1.

      When AMQ Streams generates certificates, the certificate generation annotation is automatically incremented by the Cluster Operator. For manual renewal of your own CA certificates, set the annotations with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates. The strimzi.io/ca-cert-generation has to be incremented on each CA certificate renewal.

   6. Save the secret with the new CA certificate and certificate generation annotation value.

      Example secret configuration updated with a new CA certificate

      apiVersion: v1
      kind: Secret
      data:
        ca.crt: GCa6LS3RTHeKFiFDGBOUDYFAZ0F... 1
        ca-2022-01-26T17-32-00Z.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F... 2
      metadata:
        annotations:
          strimzi.io/ca-cert-generation: "1" 3
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca-cert
        #...
      type: Opaque

      1

      New base64-encoded CA certificate

      2

      Old base64-encoded CA certificate

      3

      New CA certificate generation annotation value

3. Update the Secret for the CA key used to sign your new CA certificate.

   1. Edit the existing secret to add the new CA key and update the key generation annotation value.

      oc edit secret <ca_key_name>

      <ca_key_name> is the name of CA key, which is <kafka_cluster_name>-cluster-ca for the cluster CA key and <kafka_cluster_name>-clients-ca for the clients CA key.

      The following example shows a secret for a cluster CA key that’s associated with a Kafka cluster named my-cluster.

      Example secret configuration for a cluster CA key

      apiVersion: v1
      kind: Secret
      data:
        ca.key: SA1cKF1GFDzOIiPOIUQBHDNFGDFS... 1
      metadata:
        annotations:
          strimzi.io/ca-key-generation: "0" 2
        labels:
          strimzi.io/cluster: my-cluster
          strimzi.io/kind: Kafka
        name: my-cluster-cluster-ca
        #...
      type: Opaque

      1

      Current base64-encoded CA key

      2

      Current CA key generation annotation value

   2. Encode the CA key into base64.

      cat <path_to_new_key> | base64

   3. Update the CA key.

      Copy the base64-encoded CA key from the previous step as the value for the ca.key property under data.

   4. Increase the value of the CA key generation annotation.

      Update the strimzi.io/ca-key-generation annotation with a higher incremental value. For example, change strimzi.io/ca-key-generation=0 to strimzi.io/ca-key-generation=1. If the Secret is missing the annotation, it is treated as 0, so add the annotation with a value of 1.

      When AMQ Streams generates certificates, the key generation annotation is automatically incremented by the Cluster Operator. For manual renewal of your own CA certificates together with a new CA key, set the annotation with a higher incremental value. The annotation needs a higher value than the one from the current secret so that the Cluster Operator can roll the pods and update the certificates and keys. The strimzi.io/ca-key-generation has to be incremented on each CA certificate renewal.

4. Save the secret with the new CA key and key generation annotation value.

   Example secret configuration updated with a new CA key

   apiVersion: v1
   kind: Secret
   data:
     ca.key: AB0cKF1GFDzOIiPOIUQWERZJQ0F... 1
   metadata:
     annotations:
       strimzi.io/ca-key-generation: "1" 2
     labels:
       strimzi.io/cluster: my-cluster
       strimzi.io/kind: Kafka
     name: my-cluster-cluster-ca
     #...
   type: Opaque

   1

   New base64-encoded CA key

   2

   New CA key generation annotation value

5. Resume from the pause.

   To resume the Kafka custom resource reconciliation, set the pause-reconciliation annotation to false.

   oc annotate Kafka NAME-OF-CUSTOM-RESOURCE strimzi.io/pause-reconciliation="false"

   You can also do the same by removing the pause-reconciliation annotation.

   oc annotate Kafka <name_of_custom_resource> strimzi.io/pause-reconciliation-

Procedure

1. Create a Secret containing your private key and server certificate:

   oc create secret generic my-secret --from-file=my-listener-key.key --from-file=my-listener-certificate.crt

2. Edit the Kafka resource for your cluster. Configure the listener to use your Secret, certificate file, and private key file in the configuration.brokerCertChainAndKey property.

   Example configuration for a loadbalancer external listener with TLS encryption enabled

   # ...
   listeners:
     - name: plain
       port: 9092
       type: internal
       tls: false
     - name: external
       port: 9094
       type: loadbalancer
       tls: true
       authentication:
         type: tls
       configuration:
         brokerCertChainAndKey:
           secretName: my-secret
           certificate: my-listener-certificate.crt
           key: my-listener-key.key
   # ...

   Example configuration for a TLS listener

   # ...
   listeners:
     - name: plain
       port: 9092
       type: internal
       tls: false
     - name: tls
       port: 9093
       type: internal
       tls: true
       authentication:
         type: tls
       configuration:
         brokerCertChainAndKey:
           secretName: my-secret
           certificate: my-listener-certificate.crt
           key: my-listener-key.key
   # ...

3. Apply the new configuration to create or update the resource:

   oc apply -f kafka.yaml

   The Cluster Operator starts a rolling update of the Kafka cluster, which updates the configuration of the listeners.

   Note

   A rolling update is also started if you update a Kafka listener certificate in a Secret that is already used by a TLS or external listener.