Chapter 6. Setting up client access to the Kafka cluster

Procedure

1. Deploy a Kafka producer.

   oc run kafka-producer -ti --image=registry.redhat.io/amq7/amq-streams-kafka-27-rhel7:1.7.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --broker-list cluster-name-kafka-bootstrap:9092 --topic my-topic

2. Type a message into the console where the producer is running.
3. Press Enter to send the message.

4. Deploy a Kafka consumer.

   oc run kafka-consumer -ti --image=registry.redhat.io/amq7/amq-streams-kafka-27-rhel7:1.7.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server cluster-name-kafka-bootstrap:9092 --topic my-topic --from-beginning

5. Confirm that you see the incoming messages in the consumer console.

1. An external listener is configured for the Kafka cluster, with TLS encryption and authentication, and Kafka simple authorization is enabled.
2. A KafkaUser is created for the client, with TLS authentication and Access Control Lists (ACLs) defined for simple authorization.

Procedure

1. Configure the Kafka cluster with an external Kafka listener.

   • Define the authentication required to access the Kafka broker through the listener

   • Enable authorization on the Kafka broker

     For example:

     apiVersion: kafka.strimzi.io/v1beta2
     kind: Kafka
     metadata:
       name: my-cluster
       namespace: myproject
     spec:
       kafka:
         # ...
         listeners: 1
         - name: external 2
           port: 9094 3
           type: LISTENER-TYPE 4
           tls: true 5
           authentication:
             type: tls 6
           configuration:
             preferredNodePortAddressType: InternalDNS 7
             bootstrap and broker service overrides 8
             #...
         authorization: 9
           type: simple
           superUsers:
             - super-user-name 10
       # ...

     1

     Configuration options for enabling external listeners are described in the Generic Kafka listener schema reference.

     2

     Name to identify the listener. Must be unique within the Kafka cluster.

     3

     Port number used by the listener inside Kafka. The port number has to be unique within a given Kafka cluster. Allowed port numbers are 9092 and higher with the exception of ports 9404 and 9999, which are already used for Prometheus and JMX. Depending on the listener type, the port number might not be the same as the port number that connects Kafka clients.

     4

     External listener type specified as route, loadbalancer, nodeport or ingress. An internal listener is specified as internal.

     5

     Enables TLS encryption on the listener. Default is false. TLS encryption is not required for route listeners.

     6

     Authentication specified as tls.

     7

     (Optional, for nodeport listeners only) Configuration to specify a preference for the first address type used by AMQ Streams as the node address.

     8

     (Optional) AMQ Streams automatically determines the addresses to advertise to clients. The addresses are automatically assigned by OpenShift. You can override bootstrap and broker service addresses if the infrastructure on which you are running AMQ Streams does not provide the right address. Validation is not performed on the overrides. The override configuration differs according to the listener type. For example, you can override hosts for route, DNS names or IP addresses for loadbalancer, and node ports for nodeport.

     9

     Authoization specified as simple, which uses the AclAuthorizer Kafka plugin.

     10

     (Optional) Super users can access all brokers regardless of any access restrictions defined in ACLs.

     Warning

     An OpenShift Route address comprises the name of the Kafka cluster, the name of the listener, and the name of the namespace it is created in. For example, my-cluster-kafka-listener1-bootstrap-myproject (CLUSTER-NAME-kafka-LISTENER-NAME-bootstrap-NAMESPACE). If you are using a route listener type, be careful that the whole length of the address does not exceed a maximum limit of 63 characters.

2. Create or update the Kafka resource.

   oc apply -f KAFKA-CONFIG-FILE

   The Kafka cluster is configured with a Kafka broker listener using TLS authentication.

   A service is created for each Kafka broker pod.

   A service is created to serve as the bootstrap address for connection to the Kafka cluster.

   A service is also created as the external bootstrap address for external connection to the Kafka cluster using nodeport listeners.

   The cluster CA certificate to verify the identity of the kafka brokers is also created with the same name as the Kafka resource.

3. Find the bootstrap address and port from the status of the Kafka resource.

   oc get kafka KAFKA-CLUSTER-NAME -o jsonpath='{.status.listeners[?(@.type=="external")].bootstrapServers}'

   Use the bootstrap address in your Kafka client to connect to the Kafka cluster.

4. Extract the public cluster CA certificate and password from the generated KAFKA-CLUSTER-NAME-cluster-ca-cert Secret.

   oc get secret KAFKA-CLUSTER-NAME-cluster-ca-cert -o jsonpath='{.data.ca\.p12}' | base64 -d > ca.p12

   oc get secret KAFKA-CLUSTER-NAME-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 -d > ca.password

   Use the certificate and password in your Kafka client to connect to the Kafka cluster with TLS encryption.

   Note

   Cluster CA certificates renew automatically by default. If you are using your own Kafka listener certificates, you will need to renew the certificates manually.

5. Create or modify a user representing the client that requires access to the Kafka cluster.

   • Specify the same authentication type as the Kafka listener.

   • Specify the authorization ACLs for simple authorization.

     For example:

     apiVersion: kafka.strimzi.io/v1beta2
     kind: KafkaUser
     metadata:
       name: my-user
       labels:
         strimzi.io/cluster: my-cluster 1
     spec:
       authentication:
         type: tls 2
       authorization:
         type: simple
         acls: 3
           - resource:
               type: topic
               name: my-topic
               patternType: literal
             operation: Read
           - resource:
               type: topic
               name: my-topic
               patternType: literal
             operation: Describe
           - resource:
               type: group
               name: my-group
               patternType: literal
             operation: Read

     1

     The label must match the label of the Kafka cluster for the user to be created.

     2

     Authentication specified as tls.

     3

     Simple authorization requires an accompanying list of ACL rules to apply to the user. The rules define the operations allowed on Kafka resources based on the username (my-user).

6. Create or modify the KafkaUser resource.

   oc apply -f USER-CONFIG-FILE

   The user is created, as well as a Secret with the same name as the KafkaUser resource. The Secret contains a private and public key for TLS client authentication.

   For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-user
     labels:
       strimzi.io/kind: KafkaUser
       strimzi.io/cluster: my-cluster
   type: Opaque
   data:
     ca.crt: PUBLIC-KEY-OF-THE-CLIENT-CA
     user.crt: USER-CERTIFICATE-CONTAINING-PUBLIC-KEY-OF-USER
     user.key: PRIVATE-KEY-OF-USER
     user.p12: P12-ARCHIVE-FILE-STORING-CERTIFICATES-AND-KEYS
     user.password: PASSWORD-PROTECTING-P12-ARCHIVE

7. Configure your client to connect to the Kafka cluster with the properties required to make a secure connection to the Kafka cluster.

   1. Add the authentication details for the public cluster certificates:

      security.protocol: SSL 1
      ssl.truststore.location: PATH-TO/ssl/keys/truststore 2
      ssl.truststore.password: CLUSTER-CA-CERT-PASSWORD 3
      ssl.truststore.type=PKCS12 4

      1

      Enables TLS encryption (with or without TLS client authentication).

      2

      Specifies the truststore location where the certificates were imported.

      3

      Specifies the password for accessing the truststore. This property can be omitted if it is not needed by the truststore.

      4

      Identifies the truststore type.

      Note

      Use security.protocol: SASL_SSL when using SCRAM-SHA authentication over TLS.

   2. Add the bootstrap address and port for connecting to the Kafka cluster:

      bootstrap.servers: BOOTSTRAP-ADDRESS:PORT

   3. Add the authentication details for the public user certificates:

      ssl.keystore.location: PATH-TO/ssl/keys/user1.keystore 1
      ssl.keystore.password: USER-CERT-PASSWORD 2

      1

      Specifies the keystore location where the certificates were imported.

      2

      Specifies the password for accessing the keystore. This property can be omitted if it is not needed by the keystore.

      The public user certificate is signed by the client CA when it is created.