Chapter 14. Setting up client access to a Kafka cluster
After you have deployed Streams for Apache Kafka, you can set up client access to your Kafka cluster. To verify the deployment, you can deploy example producer and consumer clients. Otherwise, create listeners that provide client access within or outside the OpenShift cluster.
14.1. Deploying example clients
Deploy example producer and consumer clients to send and receive messages. You can use these clients to verify a deployment of Streams for Apache Kafka.
Prerequisites
- The Kafka cluster is available for the clients.
Procedure
Deploy a Kafka producer.
oc run kafka-producer -ti --image=registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server cluster-name-kafka-bootstrap:9092 --topic my-topic
- Type a message into the console where the producer is running.
- Press Enter to send the message.
Deploy a Kafka consumer.
oc run kafka-consumer -ti --image=registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server cluster-name-kafka-bootstrap:9092 --topic my-topic --from-beginning
- Confirm that you see the incoming messages in the consumer console.
14.2. Configuring listeners to connect to Kafka brokers
Use listeners for client connection to Kafka brokers. Streams for Apache Kafka provides a generic GenericKafkaListener
schema with properties to configure listeners through the Kafka
resource. The GenericKafkaListener
provides a flexible approach to listener configuration. You can specify properties to configure internal listeners for connecting within the OpenShift cluster or external listeners for connecting outside the OpenShift cluster.
Specify a connection type
to expose Kafka in the listener configuration. The type chosen depends on your requirements, and your environment and infrastructure. The following listener types are supported:
- Internal listeners
-
internal
to connect within the same OpenShift cluster -
cluster-ip
to expose Kafka using per-brokerClusterIP
services
-
- External listeners
-
nodeport
to use ports on OpenShift nodes -
loadbalancer
to use loadbalancer services -
ingress
to use KubernetesIngress
and the Ingress NGINX Controller for Kubernetes (Kubernetes only) -
route
to use OpenShiftRoute
and the default HAProxy router (OpenShift only)
-
Do not use ingress
on OpenShift, use the route
type instead. The Ingress NGINX Controller is only intended for use on Kubernetes. The route
type is only supported on OpenShift.
An internal
type listener configuration uses a headless service and the DNS names given to the broker pods. You might want to join your OpenShift network to an outside network. In which case, you can configure an internal
type listener (using the useServiceDnsDomain
property) so that the OpenShift service DNS domain (typically .cluster.local
) is not used. You can also configure a cluster-ip
type of listener that exposes a Kafka cluster based on per-broker ClusterIP
services. This is a useful option when you can’t route through the headless service or you wish to incorporate a custom access mechanism. For example, you might use this listener when building your own type of external listener for a specific Ingress controller or the OpenShift Gateway API.
External listeners handle access to a Kafka cluster from networks that require different authentication mechanisms. You can configure external listeners for client access outside an OpenShift environment using a specified connection mechanism, such as a loadbalancer or route. For example, loadbalancers might not be suitable for certain infrastructure, such as bare metal, where node ports provide a better option.
Each listener is defined as an array in the Kafka
resource.
Example listener configuration
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... listeners: - name: plain port: 9092 type: internal tls: false configuration: useServiceDnsDomain: true - name: tls port: 9093 type: internal tls: true authentication: type: tls - name: external1 port: 9094 type: route tls: true configuration: brokerCertChainAndKey: secretName: my-secret certificate: my-certificate.crt key: my-key.key # ...
You can configure as many listeners as required, as long as their names and ports are unique. You can also configure listeners for secure connection using authentication.
If you want to know more about the pros and cons of each connection type, refer to Accessing Apache Kafka in Strimzi.
If you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.
Additional resources
14.3. Listener naming conventions
From the listener configuration, the resulting listener bootstrap and per-broker service names are structured according to the following naming conventions:
Listener type | Bootstrap service name | Per-Broker service name |
---|---|---|
| <cluster_name>-kafka-bootstrap | Not applicable |
| <cluster_name>-kafka-<listener-name>-bootstrap | <cluster_name>-kafka-<listener-name>-<idx> |
For example, my-cluster-kafka-bootstrap
, my-cluster-kafka-external1-bootstrap
, and my-cluster-kafka-external1-0
. The names are assigned to the services, routes, load balancers, and ingresses created through the listener configuration.
You can use certain backwards compatible names and port numbers to transition listeners initially configured under the retired KafkaListeners
schema. The resulting external listener naming convention varies slightly. These specific combinations of listener name and port configuration values are backwards compatible:
Listener name | Port | Bootstrap service name | Per-Broker service name |
---|---|---|---|
|
| <cluster_name>-kafka-bootstrap | Not applicable |
|
| <cluster-name>-kafka-bootstrap | Not applicable |
|
| <cluster_name>-kafka-bootstrap | <cluster_name>-kafka-bootstrap-<idx> |
14.4. Setting up client access to a Kafka cluster using listeners
Using the address of the Kafka cluster, you can provide access to a client in the same OpenShift cluster; or provide external access to a client on a different OpenShift namespace or outside OpenShift entirely. This procedure shows how to configure client access to a Kafka cluster from outside OpenShift or from another OpenShift cluster.
A Kafka listener provides access to the Kafka cluster. Client access is secured using the following configuration:
-
An external listener is configured for the Kafka cluster, with TLS encryption and mTLS authentication, and Kafka
simple
authorization enabled. -
A
KafkaUser
is created for the client, with mTLS authentication, and Access Control Lists (ACLs) defined forsimple
authorization.
You can configure your listener to use mutual tls
, scram-sha-512
, or oauth
authentication. mTLS always uses encryption, but encryption is also recommended when using SCRAM-SHA-512 and OAuth 2.0 authentication.
You can configure simple
, oauth
, opa
, or custom
authorization for Kafka brokers. When enabled, authorization is applied to all enabled listeners.
When you configure the KafkaUser
authentication and authorization mechanisms, ensure they match the equivalent Kafka configuration:
-
KafkaUser.spec.authentication
matchesKafka.spec.kafka.listeners[*].authentication
-
KafkaUser.spec.authorization
matchesKafka.spec.kafka.authorization
You should have at least one listener supporting the authentication you want to use for the KafkaUser
.
Authentication between Kafka users and Kafka brokers depends on the authentication settings for each. For example, it is not possible to authenticate a user with mTLS if it is not also enabled in the Kafka configuration.
Streams for Apache Kafka operators automate the configuration process and create the certificates required for authentication:
- The Cluster Operator creates the listeners and sets up the cluster and client certificate authority (CA) certificates to enable authentication with the Kafka cluster.
- The User Operator creates the user representing the client and the security credentials used for client authentication, based on the chosen authentication type.
You add the certificates to your client configuration.
In this procedure, the CA certificates generated by the Cluster Operator are used, but you can replace them by installing your own certificates. You can also configure your listener to use a Kafka listener certificate managed by an external CA (certificate authority).
Certificates are available in PEM (.crt) and PKCS #12 (.p12) formats. This procedure uses PEM certificates. Use PEM certificates with clients that use certificates in X.509 format.
For internal clients in the same OpenShift cluster and namespace, you can mount the cluster CA certificate in the pod specification. For more information, see Configuring internal clients to trust the cluster CA.
Prerequisites
- The Kafka cluster is available for connection by a client running outside the OpenShift cluster
- The Cluster Operator and User Operator are running in the cluster
Procedure
Configure the Kafka cluster with a Kafka listener.
- Define the authentication required to access the Kafka broker through the listener.
Enable authorization on the Kafka broker.
Example listener configuration
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: myproject spec: kafka: # ... listeners: 1 - name: external1 2 port: 9094 3 type: <listener_type> 4 tls: true 5 authentication: type: tls 6 configuration: 7 #... authorization: 8 type: simple superUsers: - super-user-name 9 # ...
- 1
- Configuration options for enabling external listeners are described in the Generic Kafka listener schema reference.
- 2
- Name to identify the listener. Must be unique within the Kafka cluster.
- 3
- Port number used by the listener inside Kafka. The port number has to be unique within a given Kafka cluster. Allowed port numbers are 9092 and higher with the exception of ports 9404 and 9999, which are already used for Prometheus and JMX. Depending on the listener type, the port number might not be the same as the port number that connects Kafka clients.
- 4
- External listener type specified as
route
(OpenShift only),loadbalancer
,nodeport
oringress
(Kubernetes only). An internal listener is specified asinternal
orcluster-ip
. - 5
- Required. TLS encryption on the listener. For
route
andingress
type listeners it must be set totrue
. For mTLS authentication, also use theauthentication
property. - 6
- Client authentication mechanism on the listener. For server and client authentication using mTLS, you specify
tls: true
andauthentication.type: tls
. - 7
- (Optional) Depending on the requirements of the listener type, you can specify additional listener configuration.
- 8
- Authorization specified as
simple
, which uses theAclAuthorizer
andStandardAuthorizer
Kafka plugins. - 9
- (Optional) Super users can access all brokers regardless of any access restrictions defined in ACLs.
WarningAn OpenShift route address comprises the Kafka cluster name, the listener name, the project name, and the domain of the router. For example,
my-cluster-kafka-external1-bootstrap-my-project.domain.com
(<cluster_name>-kafka-<listener_name>-bootstrap-<namespace>.<domain>). Each DNS label (between periods “.”) must not exceed 63 characters, and the total length of the address must not exceed 255 characters.
Create or update the
Kafka
resource.oc apply -f <kafka_configuration_file>
The Kafka cluster is configured with a Kafka broker listener using mTLS authentication.
A service is created for each Kafka broker pod.
A service is created to serve as the bootstrap address for connection to the Kafka cluster.
A service is also created as the external bootstrap address for external connection to the Kafka cluster using
nodeport
listeners.The cluster CA certificate to verify the identity of the kafka brokers is also created in the secret
<cluster_name>-cluster-ca-cert
.NoteIf you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.
Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafka
resource.oc get kafka <kafka_cluster_name> -o=jsonpath='{.status.listeners[?(@.name=="<listener_name>")].bootstrapServers}{"\n"}'
For example:
oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'
Use the bootstrap address in your Kafka client to connect to the Kafka cluster.
Create or modify a user representing the client that requires access to the Kafka cluster.
-
Specify the same authentication type as the
Kafka
listener. Specify the authorization ACLs for
simple
authorization.Example user configuration
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaUser metadata: name: my-user labels: strimzi.io/cluster: my-cluster 1 spec: authentication: type: tls 2 authorization: type: simple acls: 3 - resource: type: topic name: my-topic patternType: literal operations: - Describe - Read - resource: type: group name: my-group patternType: literal operations: - Read
-
Specify the same authentication type as the
Create or modify the
KafkaUser
resource.oc apply -f USER-CONFIG-FILE
The user is created, as well as a secret with the same name as the
KafkaUser
resource. The secret contains a public and private key for mTLS authentication.Example secret
apiVersion: v1 kind: Secret metadata: name: my-user labels: strimzi.io/kind: KafkaUser strimzi.io/cluster: my-cluster type: Opaque data: ca.crt: <public_key> # Public key of the clients CA user.crt: <user_certificate> # Public key of the user user.key: <user_private_key> # Private key of the user user.p12: <store> # PKCS #12 store for user certificates and keys user.password: <password_for_store> # Protects the PKCS #12 store
Extract the cluster CA certificate from the
<cluster_name>-cluster-ca-cert
secret of the Kafka cluster.oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
Extract the user CA certificate from the
<user_name>
secret.oc get secret <user_name> -o jsonpath='{.data.user\.crt}' | base64 -d > user.crt
Extract the private key of the user from the
<user_name>
secret.oc get secret <user_name> -o jsonpath='{.data.user\.key}' | base64 -d > user.key
Configure your client with the bootstrap address hostname and port for connecting to the Kafka cluster:
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<hostname>:<port>");
Configure your client with the truststore credentials to verify the identity of the Kafka cluster.
Specify the public cluster CA certificate.
Example truststore configuration
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG, "<ca.crt_file_content>");
SSL is the specified security protocol for mTLS authentication. Specify
SASL_SSL
for SCRAM-SHA-512 authentication over TLS. PEM is the file format of the truststore.Configure your client with the keystore credentials to verify the user when connecting to the Kafka cluster.
Specify the public certificate and private key.
Example keystore configuration
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "PEM"); props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "<user.crt_file_content>"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "<user.key_file_content>");
Add the keystore certificate and the private key directly to the configuration. Add as a single-line format. Between the
BEGIN CERTIFICATE
andEND CERTIFICATE
delimiters, start with a newline character (\n
). End each line from the original certificate with\n
too.Example keystore configuration
props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "-----BEGIN CERTIFICATE----- \n<user_certificate_content_line_1>\n<user_certificate_content_line_n>\n-----END CERTIFICATE---"); props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "----BEGIN PRIVATE KEY-----\n<user_key_content_line_1>\n<user_key_content_line_n>\n-----END PRIVATE KEY-----");
Additional resources
- Section 15.1.1, “Listener authentication”
- Section 15.1.2, “Kafka authorization”
If you are using an authorization server, you can use token-based authentication and authorization:
14.5. Accessing Kafka using node ports
Use node ports to access a Streams for Apache Kafka cluster from an external client outside the OpenShift cluster.
To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.
The procedure shows basic nodeport
listener configuration. You can use listener properties to enable TLS encryption (tls
) and specify a client authentication mechanism (authentication
). Add additional configuration using configuration
properties. For example, you can use the following configuration properties with nodeport
listeners:
preferredNodePortAddressType
- Specifies the first address type that’s checked as the node address.
externalTrafficPolicy
- Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
nodePort
- Overrides the assigned node port numbers for the bootstrap and broker services.
For more information on listener configuration, see the GenericKafkaListener
schema reference.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster
. The name of the listener is external4
.
Procedure
Configure a
Kafka
resource with an external listener set to thenodeport
type.For example:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: labels: app: my-cluster name: my-cluster namespace: myproject spec: kafka: # ... listeners: - name: external4 port: 9094 type: nodeport tls: true authentication: type: tls # ... # ... zookeeper: # ...
Create or update the resource.
oc apply -f <kafka_configuration_file>
A cluster CA certificate to verify the identity of the kafka brokers is created in the secret
my-cluster-cluster-ca-cert
.NodePort
type services are created for each Kafka broker, as well as an external bootstrap service.Node port services created for the bootstrap and brokers
NAME TYPE CLUSTER-IP PORT(S) my-cluster-kafka-external4-0 NodePort 172.30.55.13 9094:31789/TCP my-cluster-kafka-external4-1 NodePort 172.30.250.248 9094:30028/TCP my-cluster-kafka-external4-2 NodePort 172.30.115.81 9094:32650/TCP my-cluster-kafka-external4-bootstrap NodePort 172.30.30.23 9094:32650/TCP
The bootstrap address used for client connection is propagated to the
status
of theKafka
resource.Example status for the bootstrap address
status: clusterId: Y_RJQDGKRXmNF7fEcWldJQ conditions: - lastTransitionTime: '2023-01-31T14:59:37.113630Z' status: 'True' type: Ready kafkaVersion: 3.7.0 listeners: # ... - addresses: - host: ip-10-0-224-199.us-west-2.compute.internal port: 32650 bootstrapServers: 'ip-10-0-224-199.us-west-2.compute.internal:32650' certificates: - | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- name: external4 observedGeneration: 2 operatorLastSuccessfulVersion: 2.7 # ...
Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafka
resource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external4")].bootstrapServers}{"\n"}' ip-10-0-224-199.us-west-2.compute.internal:32650
Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
Configure your client to connect to the brokers.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
ip-10-0-224-199.us-west-2.compute.internal:32650
. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
14.6. Accessing Kafka using loadbalancers
Use loadbalancers to access a Streams for Apache Kafka cluster from an external client outside the OpenShift cluster.
To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.
The procedure shows basic loadbalancer
listener configuration. You can use listener properties to enable TLS encryption (tls
) and specify a client authentication mechanism (authentication
). Add additional configuration using configuration
properties. For example, you can use the following configuration properties with loadbalancer
listeners:
loadBalancerSourceRanges
- Restricts traffic to a specified list of CIDR (Classless Inter-Domain Routing) ranges.
externalTrafficPolicy
- Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
loadBalancerIP
- Requests a specific IP address when creating a loadbalancer.
For more information on listener configuration, see the GenericKafkaListener
schema reference.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster
. The name of the listener is external3
.
Procedure
Configure a
Kafka
resource with an external listener set to theloadbalancer
type.For example:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: labels: app: my-cluster name: my-cluster namespace: myproject spec: kafka: # ... listeners: - name: external3 port: 9094 type: loadbalancer tls: true authentication: type: tls # ... # ... zookeeper: # ...
Create or update the resource.
oc apply -f <kafka_configuration_file>
A cluster CA certificate to verify the identity of the kafka brokers is also created in the secret
my-cluster-cluster-ca-cert
.loadbalancer
type services and loadbalancers are created for each Kafka broker, as well as an external bootstrap service.Loadbalancer services and loadbalancers created for the bootstraps and brokers
NAME TYPE CLUSTER-IP PORT(S) my-cluster-kafka-external3-0 LoadBalancer 172.30.204.234 9094:30011/TCP my-cluster-kafka-external3-1 LoadBalancer 172.30.164.89 9094:32544/TCP my-cluster-kafka-external3-2 LoadBalancer 172.30.73.151 9094:32504/TCP my-cluster-kafka-external3-bootstrap LoadBalancer 172.30.30.228 9094:30371/TCP NAME EXTERNAL-IP (loadbalancer) my-cluster-kafka-external3-0 a8a519e464b924000b6c0f0a05e19f0d-1132975133.us-west-2.elb.amazonaws.com my-cluster-kafka-external3-1 ab6adc22b556343afb0db5ea05d07347-611832211.us-west-2.elb.amazonaws.com my-cluster-kafka-external3-2 a9173e8ccb1914778aeb17eca98713c0-777597560.us-west-2.elb.amazonaws.com my-cluster-kafka-external3-bootstrap a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com
The bootstrap address used for client connection is propagated to the
status
of theKafka
resource.Example status for the bootstrap address
status: clusterId: Y_RJQDGKRXmNF7fEcWldJQ conditions: - lastTransitionTime: '2023-01-31T14:59:37.113630Z' status: 'True' type: Ready kafkaVersion: 3.7.0 listeners: # ... - addresses: - host: >- a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com port: 9094 bootstrapServers: >- a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094 certificates: - | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- name: external3 observedGeneration: 2 operatorLastSuccessfulVersion: 2.7 # ...
The DNS addresses used for client connection are propagated to the
status
of each loadbalancer service.Example status for the bootstrap loadbalancer
status: loadBalancer: ingress: - hostname: >- a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com # ...
Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the
Kafka
resource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external3")].bootstrapServers}{"\n"}' a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094
Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
Configure your client to connect to the brokers.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9094
. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
-
Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example,
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
14.7. Accessing Kafka using OpenShift routes
Use OpenShift routes to access a Streams for Apache Kafka cluster from clients outside the OpenShift cluster.
To be able to use routes, add configuration for a route
type listener in the Kafka
custom resource. When applied, the configuration creates a dedicated route and service for an external bootstrap and each broker in the cluster. Clients connect to the bootstrap route, which routes them through the bootstrap service to connect to a broker. Per-broker connections are then established using DNS names, which route traffic from the client to the broker through the broker-specific routes and services.
To connect to a broker, you specify a hostname for the route bootstrap address, as well as the certificate used for TLS encryption. For access using routes, the port is always 443.
An OpenShift route address comprises the Kafka cluster name, the listener name, the project name, and the domain of the router. For example, my-cluster-kafka-external1-bootstrap-my-project.domain.com
(<cluster_name>-kafka-<listener_name>-bootstrap-<namespace>.<domain>). Each DNS label (between periods “.”) must not exceed 63 characters, and the total length of the address must not exceed 255 characters.
The procedure shows basic listener configuration. TLS encryption (tls
) must be enabled. You can also specify a client authentication mechanism (authentication
). Add additional configuration using configuration
properties. For example, you can use the host
configuration property with route
listeners to specify the hostnames used by the bootstrap and per-broker services.
For more information on listener configuration, see the GenericKafkaListener
schema reference.
TLS passthrough
TLS passthrough is enabled for routes created by Streams for Apache Kafka. Kafka uses a binary protocol over TCP, but routes are designed to work with a HTTP protocol. To be able to route TCP traffic through routes, Streams for Apache Kafka uses TLS passthrough with Server Name Indication (SNI).
SNI helps with identifying and passing connection to Kafka brokers. In passthrough mode, TLS encryption is always used. Because the connection passes to the brokers, the listeners use TLS certificates signed by the internal cluster CA and not the ingress certificates. To configure listeners to use your own listener certificates, use the brokerCertChainAndKey
property.
Prerequisites
- A running Cluster Operator
In this procedure, the Kafka cluster name is my-cluster
. The name of the listener is external1
.
Procedure
Configure a
Kafka
resource with an external listener set to theroute
type.For example:
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: labels: app: my-cluster name: my-cluster namespace: myproject spec: kafka: # ... listeners: - name: external1 port: 9094 type: route tls: true 1 authentication: type: tls # ... # ... zookeeper: # ...
- 1
- For
route
type listeners, TLS encryption must be enabled (true
).
Create or update the resource.
oc apply -f <kafka_configuration_file>
A cluster CA certificate to verify the identity of the kafka brokers is created in the secret
my-cluster-cluster-ca-cert
.ClusterIP
type services are created for each Kafka broker, as well as an external bootstrap service.A
route
is also created for each service, with a DNS address (host/port) to expose them using the default OpenShift HAProxy router.The routes are preconfigured with TLS passthrough.
Routes created for the bootstraps and brokers
NAME HOST/PORT SERVICES PORT TERMINATION my-cluster-kafka-external1-0 my-cluster-kafka-external1-0-my-project.router.com my-cluster-kafka-external1-0 9094 passthrough my-cluster-kafka-external1-1 my-cluster-kafka-external1-1-my-project.router.com my-cluster-kafka-external1-1 9094 passthrough my-cluster-kafka-external1-2 my-cluster-kafka-external1-2-my-project.router.com my-cluster-kafka-external1-2 9094 passthrough my-cluster-kafka-external1-bootstrap my-cluster-kafka-external1-bootstrap-my-project.router.com my-cluster-kafka-external1-bootstrap 9094 passthrough
The DNS addresses used for client connection are propagated to the
status
of each route.Example status for the bootstrap route
status: ingress: - host: >- my-cluster-kafka-external1-bootstrap-my-project.router.com # ...
Use a target broker to check the client-server TLS connection on port 443 using the OpenSSL
s_client
.openssl s_client -connect my-cluster-kafka-external1-0-my-project.router.com:443 -servername my-cluster-kafka-external1-0-my-project.router.com -showcerts
The server name is the Server Name Indication (SNI) for passing the connection to the broker.
If the connection is successful, the certificates for the broker are returned.
Certificates for the broker
Certificate chain 0 s:O = io.strimzi, CN = my-cluster-kafka i:O = io.strimzi, CN = cluster-ca v0
Retrieve the address of the bootstrap service from the status of the
Kafka
resource.oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external1")].bootstrapServers}{"\n"}' my-cluster-kafka-external1-bootstrap-my-project.router.com:443
The address comprises the Kafka cluster name, the listener name, the project name and the domain of the router (
router.com
in this example).Extract the cluster CA certificate.
oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
Configure your client to connect to the brokers.
- Specify the address for the bootstrap service and port 443 in your Kafka client as the bootstrap address to connect to the Kafka cluster.
Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.
If you enabled a client authentication mechanism, you will also need to configure it in your client.
If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.
14.8. Returning connection details for services
Service discovery makes it easier for client applications running in the same OpenShift cluster as Streams for Apache Kafka to interact with a Kafka cluster.
A service discovery label and annotation are generated for services used to access the Kafka cluster:
- Internal Kafka bootstrap service
- Kafka Bridge service
The label helps to make the service discoverable, while the annotation provides connection details for client applications to establish connections.
The service discovery label, strimzi.io/discovery
, is set as true
for the Service
resources. The service discovery annotation has the same key, providing connection details in JSON format for each service.
Example internal Kafka bootstrap service
apiVersion: v1 kind: Service metadata: annotations: strimzi.io/discovery: |- [ { "port" : 9092, "tls" : false, "protocol" : "kafka", "auth" : "scram-sha-512" }, { "port" : 9093, "tls" : true, "protocol" : "kafka", "auth" : "tls" } ] labels: strimzi.io/cluster: my-cluster strimzi.io/discovery: "true" strimzi.io/kind: Kafka strimzi.io/name: my-cluster-kafka-bootstrap name: my-cluster-kafka-bootstrap spec: #...
Example Kafka Bridge service
apiVersion: v1 kind: Service metadata: annotations: strimzi.io/discovery: |- [ { "port" : 8080, "tls" : false, "auth" : "none", "protocol" : "http" } ] labels: strimzi.io/cluster: my-bridge strimzi.io/discovery: "true" strimzi.io/kind: KafkaBridge strimzi.io/name: my-bridge-bridge-service
Find services by specifying the discovery label when fetching services from the command line or a corresponding API call.
Returning services using the discovery label
oc get service -l strimzi.io/discovery=true
Connection details are returned when retrieving the service discovery label.