Getting Started with Streams for Apache Kafka on OpenShift

Procedure

1. Navigate in the OpenShift web console to the Home > Projects page and create a project (namespace) for the installation.

   We use a project named streams-kafka in this example.

2. Navigate to the Operators > OperatorHub page.

3. Scroll or type a keyword into the Filter by keyword box to find the Streams for Apache Kafka operator.

   The operator is located in the Streaming & Messaging category.

4. Click Streams for Apache Kafka to display the operator information.
5. Read the information about the operator and click Install.

6. On the Install Operator page, choose from the following installation and update options:

   • Update Channel: Choose the update channel for the operator.

     • The (default) stable channel contains all the latest updates and releases, including major, minor, and micro releases, which are assumed to be well tested and stable.
     • An amq-streams-X.x channel contains the minor and micro release updates for a major release, where X is the major release version number.
     • An amq-streams-X.Y.x channel contains the micro release updates for a minor release, where X is the major release version number and Y is the minor release version number.

   • Installation Mode: Choose the project you created to install the operator on a specific namespace.

     You can install the Streams for Apache Kafka operator to all namespaces in the cluster (the default option) or a specific namespace. We recommend that you dedicate a specific namespace to the Kafka cluster and other Streams for Apache Kafka components.

   • Update approval: By default, the Streams for Apache Kafka operator is automatically upgraded to the latest Streams for Apache Kafka version by the Operator Lifecycle Manager (OLM). Optionally, select Manual if you want to manually approve future upgrades. For more information on operators, see the OpenShift documentation.

7. Click Install to install the operator to your selected namespace.

   The Streams for Apache Kafka operator deploys the Cluster Operator, CRDs, and role-based access control (RBAC) resources to the selected namespace.

8. After the operator is ready for use, navigate to Operators > Installed Operators to verify that the operator has installed to the selected namespace.

   The status will show as Succeeded.

   You can now use the Streams for Apache Kafka operator to deploy Kafka components, starting with a Kafka cluster and node pools.

Procedure

1. Navigate in the web console to the Operators > Installed Operators page and click Streams for Apache Kafka to display the operator details.

   From Provided APIs, you can create instances of Kafka components.

2. Click Create instance under Kafka to create a Kafka instance.

   By default, you’ll create a Kafka cluster called my-cluster:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     annotations:
       strimzi.io/node-pools: enabled
       strimzi.io/kraft: enabled
   spec:
     kafka:
       config:
         offsets.topic.replication.factor: 3
         transaction.state.log.replication.factor: 3
         transaction.state.log.min.isr: 2
         default.replication.factor: 3
         min.insync.replicas: 2
       listeners:
         - name: plain
           port: 9092
           type: internal
           tls: false
         - name: tls
           port: 9093
           type: internal
           tls: true
       version: 4.1.0
       metadataVersion: 4.1
     entityOperator:
       topicOperator: {}
       userOperator: {}

   Copy to Clipboard Toggle word wrap

3. Click Create to start the installation of Kafka.

   The Kafka resource remains in a pending state until at least one node pool is created.

4. Click Create instance under KafkaNodePool to create a node pool instance.

   Switch to YAML view and paste a minimal broker pool configuration with ephemeral storage:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: broker
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - broker
     storage:
       type: jbod
       volumes:
         - id: 0
           type: ephemeral

   Copy to Clipboard Toggle word wrap

5. Click Create instance under KafkaNodePool to create a second node pool instance.

   Switch to YAML view and paste a minimal controller pool configuration with ephemeral storage:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaNodePool
   metadata:
     name: controller
     labels:
       strimzi.io/cluster: my-cluster
   spec:
     replicas: 3
     roles:
       - controller
     storage:
       type: jbod
       volumes:
         - id: 0
           type: ephemeral
           kraftMetadata: shared

   Copy to Clipboard Toggle word wrap
6. Select the Kafka page to show the installed Kafka clusters. Wait until the status of the Kafka cluster changes to Ready.

Procedure

1. Navigate in the web console to the Operators > Installed Operators page and select Streams for Apache Kafka to display the operator details.
2. Select the Kafka page to show the installed Kafka clusters.

3. Click the name of the Kafka cluster you are configuring to view its details.

   A Kafka cluster named my-cluster is used in this example.

4. Switch to YAML view for the Kafka cluster my-cluster.

5. Add route listener configuration to create an OpenShift route named listener1.

   The listener configuration must be set to the route type. You add the listener configuration under listeners in the Kafka configuration.

   External route listener configuration

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
     annotations:
       strimzi.io/node-pools: enabled
       strimzi.io/kraft: enabled
     namespace: streams-kafka
   spec:
     kafka:
       # ...
       listeners:
         # ...
         - name: listener1
           port: 9094
           type: route
           tls: true
   # ...

   Copy to Clipboard Toggle word wrap

   The client connects on port 443, the default router port, but traffic is then routed to the port you configure, which is 9094 in this example.

6. Save the updated configuration.

7. Select the Resources page for the Kafka cluster my-cluster to locate the connection information you will need for your client.

   From the Resources page, you’ll find details for the route listener and the public cluster certificate you need to connect to the Kafka cluster.

8. Click the name of the my-cluster-kafka-listener1-bootstrap route created for the Kafka cluster to show the route details.

9. Make a note of the hostname.

   The hostname is specified with port 443 in a Kafka client as the bootstrap address for connecting to the Kafka cluster.

   You can also locate the bootstrap address by navigating to Networking > Routes and selecting the streams-kafka project to display the routes created in the namespace.

   Or you can use the oc tool to extract the bootstrap details.

   Extracting bootstrap information

   oc get routes my-cluster-kafka-listener1-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'

   Copy to Clipboard Toggle word wrap

10. Navigate back to the Resources page and click the name of the my-cluster-cluster-ca-cert to show the secret details for accessing the Kafka cluster.

    The ca.crt certificate file contains the public certificate of the Kafka cluster.

    You will need the certificate to access the Kafka broker.

11. Make a local copy of the ca.crt public certificate file.

    You can copy the details of the certificate or use the OpenShift oc tool to extract them.

    Extracting the public certificate

    oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt

    Copy to Clipboard Toggle word wrap

12. Create a local truststore for the public cluster certificate using keytool.

    Creating a local truststore

    keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt

    Copy to Clipboard Toggle word wrap

    When prompted, create a password for accessing the truststore.

    The truststore is specified in a Kafka client for authenticating access to the Kafka cluster.

    You are now ready to start sending and receiving messages.

1. Use the oc command-line interface to deploy a Kafka producer.

   This example deploys a Kafka producer that connects to the Kafka cluster my-cluster

   A topic named my-topic is created.

   Deploying a Kafka producer to OpenShift

   oc run kafka-producer -ti \
   --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 \
   --rm=true \
   --restart=Never \
   -- bin/kafka-console-producer.sh \
   --bootstrap-server my-cluster-kafka-bootstrap:9092 \
   --topic my-topic

   Copy to Clipboard Toggle word wrap

   Note

   If the connection fails, check that the Kafka cluster is running and the correct cluster name is specified as the bootstrap-server.

2. From the command prompt, enter a number of messages.
3. Navigate in the OpenShift web console to the Home > Projects page and select the streams-kafka project you created.
4. From the list of pods, click kafka-producer to view the producer pod details.
5. Select Logs page to check the messages you entered are present.

6. Use the oc command-line interface to deploy a Kafka consumer.

   Deploying a Kafka consumer to OpenShift

   oc run kafka-consumer -ti \
   --image=registry.redhat.io/amq-streams/kafka-41-rhel9:3.1.0 \
   --rm=true \
   --restart=Never \
   -- bin/kafka-console-consumer.sh \
   --bootstrap-server my-cluster-kafka-bootstrap:9092 \
   --topic my-topic \
   --from-beginning

   Copy to Clipboard Toggle word wrap

   The consumer consumed messages produced to my-topic.

7. From the command prompt, confirm that you see the incoming messages in the consumer console.
8. Navigate in the OpenShift web console to the Home > Projects page and select the streams-kafka project you created.
9. From the list of pods, click kafka-consumer to view the consumer pod details.
10. Select the Logs page to check the messages you consumed are present.

1. Download and extract the Streams for Apache Kafka <version> binaries from the Streams for Apache Kafka software downloads page.

   Unzip the amq-streams-<version>-bin.zip file to any destination.

2. Open a command-line interface, and start the Kafka console producer with the topic my-topic and the authentication properties for TLS.

   Add the properties that are required for accessing the Kafka broker with an OpenShift route.

   • Use the hostname and port 443 for the OpenShift route you are using.

   • Use the password and reference to the truststore you created for the broker certificate.

     Starting a local Kafka producer

     kafka-console-producer.sh \
     --bootstrap-server my-cluster-kafka-listener1-bootstrap-streams-kafka.apps.ci-ln-50kcyvt-72292.origin-ci-int-gce.dev.rhcloud.com:443 \
     --producer-property security.protocol=SSL \
     --producer-property ssl.truststore.password=password \
     --producer-property ssl.truststore.location=client.truststore.jks \
     --topic my-topic

     Copy to Clipboard Toggle word wrap

3. Type your message into the command-line interface where the producer is running.
4. Press enter to send the message.

5. Open a new command-line interface tab or window, and start the Kafka console consumer to receive the messages.

   Use the same connection details as the producer.

   Starting a local Kafka consumer

   kafka-console-consumer.sh \
   --bootstrap-server my-cluster-kafka-listener1-bootstrap-streams-kafka.apps.ci-ln-50kcyvt-72292.origin-ci-int-gce.dev.rhcloud.com:443 \
   --consumer-property security.protocol=SSL \
   --consumer-property ssl.truststore.password=password \
   --consumer-property ssl.truststore.location=client.truststore.jks \
   --topic my-topic --from-beginning

   Copy to Clipboard Toggle word wrap

6. Confirm that you see the incoming messages in the consumer console.
7. Press Crtl+C to exit the Kafka console producer and consumer.