Chapter 2. Connecting the Streams for Apache Kafka Console to a Kafka cluster

Procedure

1. Download and extract the Streams for Apache Kafka Console installation artifacts.

   The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page.

   The files contain the deployment configuration required for the console, the Kafka cluster, and Prometheus.

   The example Kafka configuration creates a route listener that the console uses to connect to the Kafka cluster. As the console and the Kafka cluster are deployed on the same OpenShift cluster, you can use the internal bootstrap address of the Kafka cluster instead of a route.

2. Create a Prometheus instance with the configuration required by the console by applying the Prometheus installation files:

   1. Edit ${NAMESPACE} in the console-prometheus-server.clusterrolebinding.yaml file to use the namespace the Prometheus instance is going to be installed into:

      sed -i 's/${NAMESPACE}/'"my-project"'/g' <resource_path>/console-prometheus-server.clusterrolebinding.yaml

      For example, in this procedure we are installing to the my-project namespace. The configuration binds the role for Prometheus with its service account.

   2. Create the Prometheus instance by applying the installation files in this order:

      # Prometheus security resources
      oc apply -n my-project -f <resource_path>/prometheus/console-prometheus-server.clusterrole.yaml
      oc apply -n my-project -f <resource_path>/prometheus/console-prometheus-server.serviceaccount.yaml
      oc apply -n my-project -f <resource_path>/prometheus/console-prometheus-server.clusterrolebinding.yaml
      
      # Prometheus PodMonitor and Kubernetes scrape configurations
      oc apply -n my-project -f <resource_path>/prometheus/kafka-resources.podmonitor.yaml
      oc apply -n my-project -f <resource_path>/prometheus/kubernetes-scrape-configs.secret.yaml
      
      # Prometheus instance
      oc apply -n my-project -f <resource_path>/prometheus/console-prometheus.prometheus.yaml

      The instance is named console-prometheus and the URL of the service for connecting the console is http://prometheus-operated.my-project.svc.cluster.local:9090, with my-project taken from the namespace name.

      Note

      No route is deployed for the console-prometheus instance as it does not need to be accessible from outside the OpenShift cluster.

3. Create and deploy a Kafka cluster.

   1. If you are using the console with a Kafka cluster operating in KRaft mode, update the metrics configuration for the cluster in the console-kafka-metrics.configmap.yaml file:

      • Uncomment the KRaft-related metrics configuration.
      • Comment out the ZooKeeper related metrics.

      This file contains the metrics configuration required by the console.

   2. Edit the KafkaUser custom resource in the console-kafka-user1.kafkauser.yaml file by adding ACL types to provide authorized access for the console to the Kafka cluster.

      At a minimum, the Kafka user requires the following ACL rules:

      • Describe, DescribeConfigs permissions for the cluster resource
      • Read, Describe, DescribeConfigs permissions for all topic resources

      • Read, Describe permissions for all group resources

        Example user authorization settings

        apiVersion: kafka.strimzi.io/v1beta2
        kind: KafkaUser
        metadata:
          name: console-cluster-user1
          labels:
            strimzi.io/cluster: console-kafka
        spec:
          authentication:
            type: scram-sha-512
          authorization:
            type: simple
            acls:
              - resource:
                  type: cluster
                  name: ""
                  patternType: literal
                operations:
                  - Describe
                  - DescribeConfigs
              - resource:
                  type: topic
                  name: "*"
                  patternType: literal
                operations:
                  - Read
                  - Describe
                  - DescribeConfigs
              - resource:
                  type: group
                  name: "*"
                  patternType: literal
                operations:
                  - Read
                  - Describe

   3. Edit the console-kafka.kafka.yaml file to replace the placeholders:

      sed -i 's/type: ${LISTENER_TYPE}/type: route/g' console-kafka.kafka.yaml
      sed -i 's/\${CLUSTER_DOMAIN}/'"<my_router_base_domain>"'/g' console-kafka.kafka.yaml

      This file contains the Kafka custom resource configuration to create the Kafka cluster.

      These commands do the following:

      • Replace type: ${LISTENER_TYPE} with type: route. While this example uses a route type, you can replace ${LISTENER_TYPE} with any valid listener type for your deployment.
      • Replace ${CLUSTER_DOMAIN} with the value of the base domain required to specify the route listener hosts used by the bootstrap and per-broker services. By default, route listener hosts are automatically assigned by OpenShift. However, you can override the assigned route hosts by specifying hosts.

      Alternatively, you can copy the example configuration to your own Kafka deployment.

   4. Create the Kafka cluster by applying the installation files in this order:

      # Metrics configuration
      oc apply -n my-project -f <resource_path>/console-kafka-metrics.configmap.yaml
      
      # Create the cluster
      oc apply -n my-project -f <resource_path>/console-kafka.kafka.yaml
      
      # Create a user for the cluster
      oc apply -n my-project -f <resource_path>/console-kafka-user1.kafkauser.yaml

      If you are using your own Kafka cluster, apply the updated Kafka resource configuration instead of console-kafka.kafka.yaml.

      The installation files create a Kafka cluster as well as a Kafka user and the metrics configuration required by the console for connecting to the cluster A Kafka user and metrics configuration are required for each Kafka cluster you want to monitor through the console. Each Kafka user requires a unique name.

   5. If the Kafka cluster is in a different namespace from your Prometheus instance, modify the kafka-resources.podmonitor.yaml file to include a namespaceSelector:

      apiVersion: monitoring.coreos.com/v1
      kind: PodMonitor
      metadata:
        name: kafka-resources
        labels:
          app: console-kafka-monitor
      spec:
        namespaceSelector:
          matchNames:
            - <kafka_namespace>
        # ...

      This ensures that Prometheus can monitor the Kafka pods. Replace <kafka_namespace> with the actual namespace where your Kafka cluster is deployed.

4. Check the status of the deployment:

   oc get pods -n <my_console_namespace>

   Output shows the operators and cluster readiness

   NAME                        READY   STATUS   RESTARTS
   strimzi-cluster-operator    1/1     Running  0
   console-kafka-kafka-0       1/1     Running  0
   console-kafka-kafka-1       1/1     Running  0
   console-kafka-kafka-2       1/1     Running  0
   prometheus-operator-...     1/1     Running  0
   console-prometheus          1/1     Running  0

   Here, console-kafka is the name of the cluster.

   A pod ID identifies the pods created.

   With the default deployment, you install three pods.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running.

5. Install the Streams for Apache Kafka Console.

   1. Edit the console-server.clusterrolebinding.yaml file to use the namespace the console instance is going to be installed into:

      sed -i 's/${NAMESPACE}/'"my-project"'/g' /<resource_path>console-server.clusterrolebinding.yaml

      The configuration binds the role for the console with its service account.

   2. Install the console user interface and route to the interface by applying the installation files in this order:

      # Console security resources
      oc apply -n my-project -f <resource_path>/console-server.clusterrole.yaml
      oc apply -n my-project -f <resource_path>/console-server.serviceaccount.yaml
      oc apply -n my-project -f <resource_path>/console-server.clusterrolebinding.yaml
      
      # Console user interface service
      oc apply -n my-project -f <resource_path>/console-ui.service.yaml
      
      # Console route
      oc apply -n my-project -f <resource_path>/console-ui.route.yaml

      The install creates the role, role binding, service account, services, and route necessary to run the console user interface.

   3. Create a Secret called console-ui-secrets containing two secret values (as described in the prerequisites) for session management and authentication within the console:

      oc create secret generic console-ui-secrets -n my-project \
          --from-literal=SESSION_SECRET="<session_secret_value>" \
          --from-literal=NEXTAUTH_SECRET="<next_secret_value>"

      The secrets are mounted as environment variables when the console is deployed.

   4. Get the hostname for the route created for the console user interface:

      oc get route console-ui-route -n my-project -o jsonpath='{.spec.host}'

      The hostname is required for access to the console user interface.

   5. Edit the console.deployment.yaml file to replace the placeholders:

      sed -i 's/${CONSOLE_HOSTNAME}/'"<route_hostname>"'/g' console.deployment.yaml
      sed -i 's/${NAMESPACE}/'"my-project"'/g' console.deployment.yaml

      These commands do the following:

      • Replace https://${CONSOLE_HOSTNAME} with https://<route_hostname>, which is the route used to access the console user interface.
      • Replace ${NAMESPACE} with the my-project namespace name in http://prometheus-operated.${NAMESPACE}.svc.cluster.local:9090, which is the URL of the Prometheus instance used by the console.

      If you are using your own Kafka cluster, ensure that the environment variables are configured with the correct values. The values enable the console to connect with the cluster and retrieve metrics.

   6. Install the console:

      oc apply -n my-project -f <resource_path>/console.deployment.yaml

      Output shows the console readiness

      NAME                        READY   STATUS   RESTARTS
      strimzi-cluster-operator    1/1     Running  0
      console-kafka-kafka-0       1/1     Running  0
      console-kafka-kafka-0       1/1     Running  0
      console-kafka-kafka-0       1/1     Running  0
      prometheus-operator-...     1/1     Running  0
      console-prometheus          1/1     Running  0
      console-...                 2/2     Running  0