Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Setting up client access to a Kafka cluster

After you have deployed AMQ Streams, you can set up client access to your Kafka cluster. To verify the deployment, you can deploy example producer and consumer clients. Otherwise, create listeners that provide client access within or outside the OpenShift cluster.

7.1. Deploying example clients

Deploy example producer and consumer clients to send and receive messages. You can use these clients to verify a deployment of AMQ Streams.

Prerequisites

  • The Kafka cluster is available for the clients.

Procedure

  1. Deploy a Kafka producer.

    Copy to Clipboard Toggle word wrap 
    oc run kafka-producer -ti --image=registry.redhat.io/amq-streams/kafka-34-rhel8:2.4.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server cluster-name-kafka-bootstrap:9092 --topic my-topic
  2. Type a message into the console where the producer is running.
  3. Press Enter to send the message.

  4. Deploy a Kafka consumer.

    Copy to Clipboard Toggle word wrap 
    oc run kafka-consumer -ti --image=registry.redhat.io/amq-streams/kafka-34-rhel8:2.4.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server cluster-name-kafka-bootstrap:9092 --topic my-topic --from-beginning
  5. Confirm that you see the incoming messages in the consumer console.

7.2. Configuring listeners to connect to Kafka brokers

Use listeners for client connection to Kafka brokers. AMQ Streams provides a generic GenericKafkaListener schema with properties to configure listeners through the Kafka resource. The GenericKafkaListener provides a flexible approach to listener configuration. You can specify properties to configure internal listeners for connecting within the OpenShift cluster or external listeners for connecting outside the OpenShift cluster.

Specify a connection type to expose Kafka in the listener configuration. The type chosen depends on your requirements, and your environment and infrastructure. The following listener types are supported:

Internal listeners
  • internal to connect within the same OpenShift cluster
  • cluster-ip to expose Kafka using per-broker ClusterIP services
External listeners
  • nodeport to use ports on OpenShift nodes
  • loadbalancer to use loadbalancer services
  • ingress to use Kubernetes Ingress and the Ingress NGINX Controller for Kubernetes (Kubernetes only)
  • route to use OpenShift Route and the default HAProxy router (OpenShift only)
Important

Do not use ingress on OpenShift, use the route type instead. The Ingress NGINX Controller is only intended for use on Kubernetes. The route type is only supported on OpenShift.

An internal type listener configuration uses a headless service and the DNS names given to the broker pods. You might want to join your OpenShift network to an outside network. In which case, you can configure an internal type listener (using the useServiceDnsDomain property) so that the OpenShift service DNS domain (typically .cluster.local) is not used. You can also configure a cluster-ip type of listener that exposes a Kafka cluster based on per-broker ClusterIP services. This is a useful option when you can’t route through the headless service or you wish to incorporate a custom access mechanism. For example, you might use this listener when building your own type of external listener for a specific Ingress controller or the OpenShift Gateway API.

External listeners handle access to a Kafka cluster from networks that require different authentication mechanisms. You can configure external listeners for client access outside an OpenShift environment using a specified connection mechanism, such as a loadbalancer or route. For example, loadbalancers might not be suitable for certain infrastructure, such as bare metal, where node ports provide a better option.

Each listener is defined as an array in the Kafka resource.

Example listener configuration

Copy to Clipboard Toggle word wrap 
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        configuration:
          useServiceDnsDomain: true
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: route
        tls: true
        configuration:
          brokerCertChainAndKey:
            secretName: my-secret
            certificate: my-certificate.crt
            key: my-key.key
    # ...

You can configure as many listeners as required, as long as their names and ports are unique. You can also configure listeners for secure connection using authentication.

If you want to know more about the pros and cons of each connection type, refer to Accessing Apache Kafka in Strimzi.

Note

If you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.

Additional resources

7.3. Setting up client access to a Kafka cluster using listeners

Using the address of the Kafka cluster, you can provide access to a client in the same OpenShift cluster; or provide external access to a client on a different OpenShift namespace or outside OpenShift entirely. This procedure shows how to configure client access to a Kafka cluster from outside OpenShift or from another OpenShift cluster.

A Kafka listener provides access to the Kafka cluster. Client access is secured using the following configuration:

  1. An external listener is configured for the Kafka cluster, with TLS encryption and mTLS authentication, and Kafka simple authorization enabled.
  2. A KafkaUser is created for the client, with mTLS authentication, and Access Control Lists (ACLs) defined for simple authorization.

You can configure your listener to use mutual tls, scram-sha-512, or oauth authentication. mTLS always uses encryption, but encryption is also recommended when using SCRAM-SHA-512 and OAuth 2.0 authentication.

You can configure simple, oauth, opa, or custom authorization for Kafka brokers. When enabled, authorization is applied to all enabled listeners.

When you configure the KafkaUser authentication and authorization mechanisms, ensure they match the equivalent Kafka configuration:

  • KafkaUser.spec.authentication matches Kafka.spec.kafka.listeners[*].authentication
  • KafkaUser.spec.authorization matches Kafka.spec.kafka.authorization

You should have at least one listener supporting the authentication you want to use for the KafkaUser.

Note

Authentication between Kafka users and Kafka brokers depends on the authentication settings for each. For example, it is not possible to authenticate a user with mTLS if it is not also enabled in the Kafka configuration.

AMQ Streams operators automate the configuration process and create the certificates required for authentication:

  • The Cluster Operator creates the listeners and sets up the cluster and client certificate authority (CA) certificates to enable authentication with the Kafka cluster.
  • The User Operator creates the user representing the client and the security credentials used for client authentication, based on the chosen authentication type.

You add the certificates to your client configuration.

In this procedure, the CA certificates generated by the Cluster Operator are used, but you can replace them by installing your own certificates. You can also configure your listener to use a Kafka listener certificate managed by an external CA (certificate authority).

Certificates are available in PEM (.crt) and PKCS #12 (.p12) formats. This procedure uses PEM certificates. Use PEM certificates with clients that use certificates in X.509 format.

Note

For internal clients in the same OpenShift cluster and namespace, you can mount the cluster CA certificate in the pod specification. For more information, see Configuring internal clients to trust the cluster CA.

Prerequisites

  • The Kafka cluster is available for connection by a client running outside the OpenShift cluster
  • The Cluster Operator and User Operator are running in the cluster

Procedure

  1. Configure the Kafka cluster with a Kafka listener.

    • Define the authentication required to access the Kafka broker through the listener.

    • Enable authorization on the Kafka broker.

      Example listener configuration

      Copy to Clipboard Toggle word wrap 
      apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: myproject
spec:
  kafka:
    # ...
    listeners:
      1 
      
    - name: external
      2 
      
      port: 9094
      3 
      
      type: <listener_type>
      4 
      
      tls: true
      5 
      
      authentication:
        type: tls
      6 
      
      configuration:
      7 
      
        #...
    authorization:
      8 
      
      type: simple
      superUsers:
        - super-user-name
      9 
      
  # ...

      1
      Configuration options for enabling external listeners are described in the Generic Kafka listener schema reference.
      2
      Name to identify the listener. Must be unique within the Kafka cluster.
      3
      Port number used by the listener inside Kafka. The port number has to be unique within a given Kafka cluster. Allowed port numbers are 9092 and higher with the exception of ports 9404 and 9999, which are already used for Prometheus and JMX. Depending on the listener type, the port number might not be the same as the port number that connects Kafka clients.
      4
      External listener type specified as route (OpenShift only), loadbalancer, nodeport or ingress (Kubernetes only). An internal listener is specified as internal or cluster-ip.
      5
      Required. TLS encryption on the listener. For route and ingress type listeners it must be set to true. For mTLS authentication, also use the authentication property.
      6
      Client authentication mechanism on the listener. For server and client authentication using mTLS, you specify tls: true and authentication.type: tls.
      7
      (Optional) Depending on the requirements of the listener type, you can specify additional listener configuration.
      8
      Authorization specified as simple, which uses the AclAuthorizer Kafka plugin.
      9
      (Optional) Super users can access all brokers regardless of any access restrictions defined in ACLs.
      Warning

      An OpenShift Route address comprises the name of the Kafka cluster, the name of the listener, and the name of the namespace it is created in. For example, my-cluster-kafka-listener1-bootstrap-myproject (CLUSTER-NAME-kafka-LISTENER-NAME-bootstrap-NAMESPACE). If you are using a route listener type, be careful that the whole length of the address does not exceed a maximum limit of 63 characters.

  2. Create or update the Kafka resource.

    Copy to Clipboard Toggle word wrap 
    oc apply -f <kafka_configuration_file>

    The Kafka cluster is configured with a Kafka broker listener using mTLS authentication.

    A service is created for each Kafka broker pod.

    A service is created to serve as the bootstrap address for connection to the Kafka cluster.

    A service is also created as the external bootstrap address for external connection to the Kafka cluster using nodeport listeners.

    The cluster CA certificate to verify the identity of the kafka brokers is also created in the secret <cluster_name>-cluster-ca-cert.

    Note

    If you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.

  3. Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the Kafka resource.

    Copy to Clipboard Toggle word wrap 
    oc get kafka <kafka_cluster_name> -o=jsonpath='{.status.listeners[?(@.name=="<listener_name>")].bootstrapServers}{"\n"}'

    For example:

    Copy to Clipboard Toggle word wrap 
    oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'

    Use the bootstrap address in your Kafka client to connect to the Kafka cluster.

  4. Create or modify a user representing the client that requires access to the Kafka cluster.

    • Specify the same authentication type as the Kafka listener.

    • Specify the authorization ACLs for simple authorization.

      Example user configuration

      Copy to Clipboard Toggle word wrap 
      apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: my-user
  labels:
    strimzi.io/cluster: my-cluster
      1 
      
spec:
  authentication:
    type: tls
      2 
      
  authorization:
    type: simple
    acls:
      3 
      
      - resource:
          type: topic
          name: my-topic
          patternType: literal
        operations:
          - Describe
          - Read
      - resource:
          type: group
          name: my-group
          patternType: literal
        operations:
          - Read

      1
      The label must match the label of the Kafka cluster.
      2
      Authentication specified as mutual tls.
      3
      Simple authorization requires an accompanying list of ACL rules to apply to the user. The rules define the operations allowed on Kafka resources based on the username (my-user).

  5. Create or modify the KafkaUser resource.

    Copy to Clipboard Toggle word wrap 
    oc apply -f USER-CONFIG-FILE

    The user is created, as well as a secret with the same name as the KafkaUser resource. The secret contains a public and private key for mTLS authentication.

    Example secret

    Copy to Clipboard Toggle word wrap 
    apiVersion: v1
kind: Secret
metadata:
  name: my-user
  labels:
    strimzi.io/kind: KafkaUser
    strimzi.io/cluster: my-cluster
type: Opaque
data:
  ca.crt: <public_key> # Public key of the clients CA
  user.crt: <user_certificate> # Public key of the user
  user.key: <user_private_key> # Private key of the user
  user.p12: <store> # PKCS #12 store for user certificates and keys
  user.password: <password_for_store> # Protects the PKCS #12 store

  6. Extract the cluster CA certificate from the <cluster_name>-cluster-ca-cert secret of the Kafka cluster.

    Copy to Clipboard Toggle word wrap 
    oc get secret <cluster_name>-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

  7. Extract the user CA certificate from the <user_name> secret.

    Copy to Clipboard Toggle word wrap 
    oc get secret <user_name> -o jsonpath='{.data.user\.crt}' | base64 -d > user.crt

  8. Extract the private key of the user from the <user_name> secret.

    Copy to Clipboard Toggle word wrap 
    oc get secret <user_name> -o jsonpath='{.data.user\.key}' | base64 -d > user.key

  9. Configure your client with the bootstrap address hostname and port for connecting to the Kafka cluster:

    Copy to Clipboard Toggle word wrap 
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<hostname>:<port>");

  10. Configure your client with the truststore credentials to verify the identity of the Kafka cluster.

    Specify the public cluster CA certificate.

    Example truststore configuration

    Copy to Clipboard Toggle word wrap 
    props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "PEM");
props.put(SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG, "<ca.crt_file_content>");

    SSL is the specified security protocol for mTLS authentication. Specify SASL_SSL for SCRAM-SHA-512 authentication over TLS. PEM is the file format of the truststore.

  11. Configure your client with the keystore credentials to verify the user when connecting to the Kafka cluster.

    Specify the public certificate and private key.

    Example keystore configuration

    Copy to Clipboard Toggle word wrap 
    props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
props.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "PEM");
props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "<user.crt_file_content>");
props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "<user.key_file_content>");

    Add the keystore certificate and the private key directly to the configuration. Add as a single-line format. Between the BEGIN CERTIFICATE and END CERTIFICATE delimiters, start with a newline character (\n). End each line from the original certificate with \n too.

    Example keystore configuration

    Copy to Clipboard Toggle word wrap 
    props.put(SslConfigs.SSL_KEYSTORE_CERTIFICATE_CHAIN_CONFIG, "-----BEGIN CERTIFICATE----- \n<user_certificate_content_line_1>\n<user_certificate_content_line_n>\n-----END CERTIFICATE---");
props.put(SslConfigs.SSL_KEYSTORE_KEY_CONFIG, "----BEGIN PRIVATE KEY-----\n<user_key_content_line_1>\n<user_key_content_line_n>\n-----END PRIVATE KEY-----");

    Additional resources

7.4. Accessing Kafka using node ports

Use node ports to access an AMQ Streams Kafka cluster from an external client outside the OpenShift cluster.

To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.

The procedure shows basic nodeport listener configuration. You can use listener properties to enable TLS encryption (tls) and specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the following configuration properties with nodeport listeners:

preferredNodePortAddressType
Specifies the first address type that’s checked as the node address.
externalTrafficPolicy
Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
nodePort
Overrides the assigned node port numbers for the bootstrap and broker services.

For more information on listener configuration, see the GenericKafkaListener schema reference.

Prerequisites

  • A running Cluster Operator

In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external.

Procedure

  1. Configure a Kafka resource with an external listener set to the nodeport type.

    For example:

    Copy to Clipboard Toggle word wrap 
    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  labels:
    app: my-cluster
  name: my-cluster
  namespace: myproject
spec:
  kafka:
    # ...
    listeners:
      - name: external
        port: 9094
        type: nodeport
        tls: true
        authentication:
          type: tls
        # ...
    # ...
  zookeeper:
    # ...

  2. Create or update the resource.

    Copy to Clipboard Toggle word wrap 
    oc apply -f <kafka_configuration_file>

    A cluster CA certificate to verify the identity of the kafka brokers is created in the secret my-cluster-cluster-ca-cert.

    NodePort type services are created for each Kafka broker, as well as an external bootstrap service.

    Node port services created for the bootstrap and brokers

    Copy to Clipboard Toggle word wrap 
    NAME                                 TYPE      CLUSTER-IP      PORT(S)
my-cluster-kafka-external-0          NodePort  172.30.55.13    9094:31789/TCP
my-cluster-kafka-external-1          NodePort  172.30.250.248  9094:30028/TCP
my-cluster-kafka-external-2          NodePort  172.30.115.81   9094:32650/TCP
my-cluster-kafka-external-bootstrap  NodePort  172.30.30.23    9094:32650/TCP

    The bootstrap address used for client connection is propagated to the status of the Kafka resource.

    Example status for the bootstrap address

    Copy to Clipboard Toggle word wrap 
    status:
  clusterId: Y_RJQDGKRXmNF7fEcWldJQ
  conditions:
    - lastTransitionTime: '2023-01-31T14:59:37.113630Z'
      status: 'True'
      type: Ready
  listeners:
    # ...
    - addresses:
        - host: ip-10-0-224-199.us-west-2.compute.internal
          port: 32650
      bootstrapServers: 'ip-10-0-224-199.us-west-2.compute.internal:32650'
      certificates:
        - |
          -----BEGIN CERTIFICATE-----

          -----END CERTIFICATE-----
      name: external
      type: external
  observedGeneration: 2
 # ...

  3. Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the Kafka resource.

    Copy to Clipboard Toggle word wrap 
    oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'

ip-10-0-224-199.us-west-2.compute.internal:32650

  4. Extract the cluster CA certificate.

    Copy to Clipboard Toggle word wrap 
    oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

  5. Configure your client to connect to the brokers.

    1. Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example, ip-10-0-224-199.us-west-2.compute.internal:32650.

    2. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.

      If you enabled a client authentication mechanism, you will also need to configure it in your client.

Note

If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.

7.5. Accessing Kafka using loadbalancers

Use loadbalancers to access an AMQ Streams Kafka cluster from an external client outside the OpenShift cluster.

To connect to a broker, you specify a hostname and port number for the Kafka bootstrap address, as well as the certificate used for TLS encryption.

The procedure shows basic loadbalancer listener configuration. You can use listener properties to enable TLS encryption (tls) and specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the following configuration properties with loadbalancer listeners:

loadBalancerSourceRanges
Restricts traffic to a specified list of CIDR (Classless Inter-Domain Routing) ranges.
externalTrafficPolicy
Specifies whether the service routes external traffic to node-local or cluster-wide endpoints.
loadBalancerIP
Requests a specific IP address when creating a loadbalancer.

For more information on listener configuration, see the GenericKafkaListener schema reference.

Prerequisites

  • A running Cluster Operator

In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external.

Procedure

  1. Configure a Kafka resource with an external listener set to the loadbalancer type.

    For example:

    Copy to Clipboard Toggle word wrap 
    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  labels:
    app: my-cluster
  name: my-cluster
  namespace: myproject
spec:
  kafka:
    # ...
    listeners:
      - name: external
        port: 9095
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        # ...
    # ...
  zookeeper:
    # ...

  2. Create or update the resource.

    Copy to Clipboard Toggle word wrap 
    oc apply -f <kafka_configuration_file>

    A cluster CA certificate to verify the identity of the kafka brokers is also created in the secret my-cluster-cluster-ca-cert.

    loadbalancer type services and loadbalancers are created for each Kafka broker, as well as an external bootstrap service.

    Loadbalancer services and loadbalancers created for the bootstraps and brokers

    Copy to Clipboard Toggle word wrap 
    NAME                                  TYPE            CLUSTER-IP      PORT(S)
my-cluster-kafka-external-0          LoadBalancer     172.30.204.234  9095:30011/TCP
my-cluster-kafka-external-1          LoadBalancer     172.30.164.89   9095:32544/TCP
my-cluster-kafka-external-2          LoadBalancer     172.30.73.151   9095:32504/TCP
my-cluster-kafka-external-bootstrap  LoadBalancer     172.30.30.228   9095:30371/TCP

NAME                                 EXTERNAL-IP (loadbalancer)
my-cluster-kafka-external-0          a8a519e464b924000b6c0f0a05e19f0d-1132975133.us-west-2.elb.amazonaws.com
my-cluster-kafka-external-1          ab6adc22b556343afb0db5ea05d07347-611832211.us-west-2.elb.amazonaws.com
my-cluster-kafka-external-2          a9173e8ccb1914778aeb17eca98713c0-777597560.us-west-2.elb.amazonaws.com
my-cluster-kafka-external-bootstrap  a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com

    The bootstrap address used for client connection is propagated to the status of the Kafka resource.

    Example status for the bootstrap address

    Copy to Clipboard Toggle word wrap 
    status:
  clusterId: Y_RJQDGKRXmNF7fEcWldJQ
  conditions:
    - lastTransitionTime: '2023-01-31T14:59:37.113630Z'
      status: 'True'
      type: Ready
  listeners:
    # ...
    - addresses:
        - host: >-
            a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com
          port: 9095
      bootstrapServers: >-
        a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9095
      certificates:
        - |
          -----BEGIN CERTIFICATE-----

          -----END CERTIFICATE-----
      name: external
      type: external
  observedGeneration: 2
 # ...

    The DNS addresses used for client connection are propagated to the status of each loadbalancer service.

    Example status for the bootstrap loadbalancer

    Copy to Clipboard Toggle word wrap 
    status:
  loadBalancer:
    ingress:
      - hostname: >-
          a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com
 # ...

  3. Retrieve the bootstrap address you can use to access the Kafka cluster from the status of the Kafka resource.

    Copy to Clipboard Toggle word wrap 
    oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'

a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9095

  4. Extract the cluster CA certificate.

    Copy to Clipboard Toggle word wrap 
    oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

  5. Configure your client to connect to the brokers.

    1. Specify the bootstrap host and port in your Kafka client as the bootstrap address to connect to the Kafka cluster. For example, a8d4a6fb363bf447fb6e475fc3040176-36312313.us-west-2.elb.amazonaws.com:9095.

    2. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.

      If you enabled a client authentication mechanism, you will also need to configure it in your client.

Note

If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.

7.6. Accessing Kafka using OpenShift routes

Use OpenShift routes to access an AMQ Streams Kafka cluster from clients outside the OpenShift cluster.

To be able to use routes, add configuration for a route type listener in the Kafka custom resource. When applied, the configuration creates a dedicated route and service for an external bootstrap and each broker in the cluster. Clients connect to the bootstrap route, which routes them through the bootstrap service to connect to a broker. Per-broker connections are then established using DNS names, which route traffic from the client to the broker through the broker-specific routes and services.

To connect to a broker, you specify a hostname for the route bootstrap address, as well as the certificate used for TLS encryption. For access using routes, the port is always 443.

Warning

An OpenShift route address comprises the name of the Kafka cluster, the name of the listener, and the name of the project it is created in. For example, my-cluster-kafka-external-bootstrap-myproject (<cluster_name>-kafka-<listener_name>-bootstrap-<namespace>). Be careful that the whole length of the address does not exceed a maximum limit of 63 characters.

The procedure shows basic listener configuration. TLS encryption (tls) must be enabled. You can also specify a client authentication mechanism (authentication). Add additional configuration using configuration properties. For example, you can use the host configuration property with route listeners to specify the hostnames used by the bootstrap and per-broker services.

For more information on listener configuration, see the GenericKafkaListener schema reference.

TLS passthrough

TLS passthrough is enabled for routes created by AMQ Streams. Kafka uses a binary protocol over TCP, but routes are designed to work with a HTTP protocol. To be able to route TCP traffic through routes, AMQ Streams uses TLS passthrough with Server Name Indication (SNI).

SNI helps with identifying and passing connection to Kafka brokers. In passthrough mode, TLS encryption is always used. Because the connection passes to the brokers, the listeners use TLS certificates signed by the internal cluster CA and not the ingress certificates. To configure listeners to use your own listener certificates, use the brokerCertChainAndKey property.

Prerequisites

  • A running Cluster Operator

In this procedure, the Kafka cluster name is my-cluster. The name of the listener is external.

Procedure

  1. Configure a Kafka resource with an external listener set to the route type.

    For example:

    Copy to Clipboard Toggle word wrap 
    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  labels:
    app: my-cluster
  name: my-cluster
  namespace: myproject
spec:
  kafka:
    # ...
    listeners:
      - name: external
        port: 9094
        type: route
        tls: true
    1 
    
        authentication:
          type: tls
        # ...
    # ...
  zookeeper:
    # ...
    1
    For route type listeners, TLS encryption must be enabled (true).

  2. Create or update the resource.

    Copy to Clipboard Toggle word wrap 
    oc apply -f <kafka_configuration_file>

    A cluster CA certificate to verify the identity of the kafka brokers is created in the secret my-cluster-cluster-ca-cert.

    ClusterIP type services are created for each Kafka broker, as well as an external bootstrap service.

    A route is also created for each service, with a DNS address (host/port) to expose them using the default OpenShift HAProxy router.

    The routes are preconfigured with TLS passthrough.

    Routes created for the bootstraps and brokers

    Copy to Clipboard Toggle word wrap 
    NAME                                  HOST/PORT                                                   SERVICES                              PORT  TERMINATION
my-cluster-kafka-external-0          my-cluster-kafka-external-0-my-project.router.com          my-cluster-kafka-external-0          9094  passthrough
my-cluster-kafka-external-1          my-cluster-kafka-external-1-my-project.router.com          my-cluster-kafka-external-1          9094  passthrough
my-cluster-kafka-external-2          my-cluster-kafka-external-2-my-project.router.com          my-cluster-kafka-external-2          9094  passthrough
my-cluster-kafka-external-bootstrap  my-cluster-kafka-external-bootstrap-my-project.router.com  my-cluster-kafka-external-bootstrap  9094  passthrough

    The DNS addresses used for client connection are propagated to the status of each route.

    Example status for the bootstrap route

    Copy to Clipboard Toggle word wrap 
    status:
  ingress:
    - host: >-
        my-cluster-kafka-external-bootstrap-my-project.router.com
 # ...

  3. Use a target broker to check the client-server TLS connection on port 443 using the OpenSSL s_client.

    Copy to Clipboard Toggle word wrap 
    openssl s_client -connect my-cluster-kafka-external-0-my-project.router.com:443 -servername my-cluster-kafka-external-0-my-project.router.com -showcerts

    The server name is the SNI for passing the connection to the broker.

    If the connection is successful, the certificates for the broker are returned.

    Certificates for the broker

    Copy to Clipboard Toggle word wrap 
    Certificate chain
 0 s:O = io.strimzi, CN = my-cluster-kafka
   i:O = io.strimzi, CN = cluster-ca v0

  4. Retrieve the address of the bootstrap service from the status of the Kafka resource.

    Copy to Clipboard Toggle word wrap 
    oc get kafka my-cluster -o=jsonpath='{.status.listeners[?(@.name=="external")].bootstrapServers}{"\n"}'

my-cluster-kafka-external-bootstrap-my-project.router.com:443

    The address comprises the cluster name, the listener name, the project name and the domain of the router (router.com in this example).

  5. Extract the cluster CA certificate.

    Copy to Clipboard Toggle word wrap 
    oc get secret my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

  6. Configure your client to connect to the brokers.

    1. Specify the address for the bootstrap service and port 443 in your Kafka client as the bootstrap address to connect to the Kafka cluster.

    2. Add the extracted certificate to the truststore of your Kafka client to configure a TLS connection.

      If you enabled a client authentication mechanism, you will also need to configure it in your client.

Note

If you are using your own listener certificates, check whether you need to add the CA certificate to the client’s truststore configuration. If it is a public (external) CA, you usually won’t need to add it.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.