Chapter 2. Try AMQ Streams


Install AMQ Streams and start sending and receiving messages from a topic in minutes.

You will:

  • Install AMQ Streams
  • Create a Kafka cluster
  • Access the Kafka cluster to send and receive messages

Ensure you have the prerequisites and then follow the tasks in the order provided in this chapter.

2.1. Prerequisites

  • OpenShift Container Platform cluster (3.11 and later) running on which to deploy AMQ Streams

2.2. Downloading AMQ Streams

Download a zip file that contains the resources required for installation along with examples for configuration.

Prerequisites

Procedure

  1. Download the amq-streams-x.y.z-ocp-install-examples.zip file from the AMQ Streams download site.
  2. Unzip the file to any destination.

    • On Windows or Mac, you can extract the contents of the ZIP archive by double clicking on the ZIP file.
    • On Red Hat Enterprise Linux, open a terminal window in the target machine and navigate to where the ZIP file was downloaded.

      Extract the ZIP file by executing the following command:

      unzip amq-streams-x.y.z-ocp-install-examples.zip

2.3. Installing AMQ Streams

Install AMQ Streams with the Custom Resource Definitions (CRDs) required for deployment.

In this task you create namespaces in the cluster for your deployment. It is good practice to use namespaces to separate functions.

Prerequisites

  • Installation requires a user with cluster-admin role, such as system:admin

Procedure

  1. Login in to the OpenShift cluster with cluster admin privileges.

    For example:

    oc login -u system:admin
  2. Create a new kafka (project) namespace for the AMQ Streams Kafka Cluster Operator.

    oc new-project kafka
  3. Modify the installation files to reference the new kafka namespace where you will install the AMQ Streams Kafka Cluster Operator.

    Note

    By default, the files work in the myproject namespace.

    • On Linux, use:

      sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml
    • On Mac, use:

      sed -i '' 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml
  4. Deploy the CRDs and role-based access control (RBAC) resources to manage the CRDs.

    oc new-project kafka
    oc apply -f install/cluster-operator/
  5. Create a new my-kafka-project namespace where you will deploy your Kafka cluster.

    oc new-project my-kafka-project
  6. Give access to my-kafka-project to a non-admin user developer.

    For example:

    oc adm policy add-role-to-user admin developer -n my-kafka-project
  7. Give permission to the Cluster Operator to watch the my-kafka-project namespace.

    oc set env deploy/strimzi-cluster-operator STRIMZI_NAMESPACE=kafka,my-kafka-project -n kafka
    oc apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n my-kafka-project
    oc apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n my-kafka-project
    oc apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n my-kafka-project

    The commands create role bindings that grant permission for the Cluster Operator to access the Kafka cluster.

  8. Create a new cluster role strimzi-admin.

    oc apply -f install/strimzi-admin
  9. Add the role to the non-admin user developer.

    oc adm policy add-cluster-role-to-user strimzi-admin developer

2.4. Creating a cluster

Create a Kafka cluster, then a topic within the cluster.

When you create a cluster, the Cluster Operator you deployed watches for new Kafka resources.

Prerequisites

  • For the Kafka cluster, a Cluster Operator is deployed
  • For the topic, a running Kafka cluster

Procedure

  1. Log in to the my-kafka-project namespace as user developer.

    For example:

    oc login -u developer
    oc project my-kafka-project

    After new users log in to OpenShift Container Platform, an account is created for that user.

  2. Create a new my-cluster Kafka cluster with 3 Zookeeper and 3 broker nodes.

    • Use ephemeral storage
    • Expose the Kafka cluster outside of the OpenShift cluster using an external listener configured to use route.

      cat << EOF | oc create -f -
      apiVersion: kafka.strimzi.io/v1beta1
      kind: Kafka
      metadata:
        name: my-cluster
      spec:
        kafka:
          replicas: 3
          listeners:
            plain: {}
            tls: {}
            external:
              type: route
          storage:
            type: ephemeral
        zookeeper:
          replicas: 3
          storage:
            type: ephemeral
        entityOperator:
          topicOperator: {}
      EOF
  3. Wait for the cluster to be deployed:

    oc wait kafka/my-cluster --for=condition=Ready --timeout=300s -n kafka
  4. Now that your cluster is running, create a topic to publish and subscribe from your external client.

    Create the following my-topic custom resource definition with 3 replicas and 3 partitions in the my-cluster Kafka cluster:

    cat << EOF | oc create -f -
    apiVersion: kafka.strimzi.io/v1beta1
    kind: KafkaTopic
    metadata:
      name: my-topic
      labels:
        strimzi.io/cluster: "my-cluster"
    spec:
      partitions: 3
      replicas: 3
    EOF

2.5. Accessing the cluster

As route is used for external access to the cluster, a cluster CA certificate is required to enable TLS (Transport Layer Security) encryption between the broker and the client.

Prerequisites

  • A Kafka cluster running within the OpenShift cluster
  • A running Cluster Operator

Procedure

  1. Find the address of the bootstrap route:

    oc get routes my-cluster-kafka-bootstrap -o=jsonpath='{.status.ingress[0].host}{"\n"}'

    Use the address together with port 443 in your Kafka client as the bootstrap address.

  2. Extract the public certificate of the broker certification authority:

    oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt
  3. Import the trusted certificate to a truststore:

    keytool -keystore client.truststore.jks -alias CARoot -import -file ca.crt

    You are now ready to start sending and receiving messages.

2.6. Sending and receiving messages from a topic

Test your AMQ Streams installation by sending and receiving messages outside the cluster from my-topic.

Use a terminal to run a Kafka producer and consumer on a local machine.

Prerequisites

Procedure

  1. Download the latest version of the AMQ Streams archive (amq-streams-x.y.z-bin.zip) from the AMQ Streams download site.

    Unzip the file to any destination.

  2. Open a terminal and start the Kafka console producer with the topic my-topic and the authentication properties for TLS:

    bin/kafka-console-producer.sh --broker-list <route-address>:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./client.truststore.jks --topic my-topic
  3. Type your message into the console where the producer is running.
  4. Press Enter to send the message.
  5. Open a new terminal tab or window and start the Kafka console consumer to receive the messages:

    bin/kafka-console-consumer.sh --bootstrap-server <route-address>:443 --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=./client.truststore.jks --topic my-topic --from-beginning
  6. Confirm that you see the incoming messages in the consumer console.
  7. Press Crtl+C to exit the Kafka console producer and consumer.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.