Chapter 1. Features

AMQ Streams now supports Apache Kafka version 2.4.0.

AMQ Streams uses Kafka 2.4.0. Only Kafka distributions built by Red Hat are supported.

You must upgrade the Cluster Operator to AMQ Streams version 1.4 before you can upgrade brokers and client applications to Kafka 2.4.0. For upgrade instructions, see AMQ Streams and Kafka upgrades.

Refer to the Kafka 2.3.0 and Kafka 2.4.0 Release Notes for additional information.

For more information on supported versions, see the Red Hat AMQ 7 Component Details Page on the Customer Portal.

To take advantage of incremental cooperative rebalancing, you must upgrade consumers and Kafka Streams applications to use the new protocol instead of the old eager rebalance protocol.

See Upgrading consumers and Kafka Streams applications to cooperative rebalancing and Notable changes in 2.4.0 in the Apache Kafka documentation.

AMQ Streams now provides Kubernetes-native management of connectors in a Kafka Connect cluster using a new custom resource, named KafkaConnector, and an internal operator.

A KafkaConnector YAML file describes the configuration of a source or sink connector that you deploy to your Kubernetes cluster to either create a new connector instance or manage a running one. Like other Kafka resources, the Cluster Operator updates running connector instances to match the configurations defined in their KafkaConnectors.

The Installation and Example Files now include an example KafkaConnector resource in examples/connector/source-connector.yaml. Deploy the example YAML file to create a FileStreamSourceConnector that sends each line of the license file to Kafka as a message in a topic named my-topic.

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  Config:
    file: "/opt/kafka/LICENSE"
    topic: my-topic
    # ...

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  Config:
    file: "/opt/kafka/LICENSE"
    topic: my-topic
    # ...

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...

You can now provide your own server certificates and private keys for the following types of listeners:

These user-provided certificates are called Kafka listener certificates.

You can use your organization’s private Certificate Authority (CA) or a public CA to generate and sign your own Kafka listener certificates.

# ...
listeners:
  plain: {}
  external:
    type: loadbalancer
    configuration:
      brokerCertChainAndKey:
        secretName: my-secret
        certificate: my-listener-certificate.crt
        key: my-listener-key.key
    tls: true
    authentication:
      type: tls
# ...

# ...
listeners:
  plain: {}
  external:
    type: loadbalancer
    configuration:
      brokerCertChainAndKey:
        secretName: my-secret
        certificate: my-listener-certificate.crt
        key: my-listener-key.key
    tls: true
    authentication:
      type: tls
# ...

See Kafka listener certificates and Providing your own Kafka listener certificates.

Support for OAuth 2.0 authentication moves from a Technology Preview to a generally available component of AMQ Streams.

AMQ Streams supports the use of OAuth 2.0 authentication using the SASL OAUTHBEARER mechanism. Using OAuth 2.0 token based authentication, application clients can access resources on application servers (called ‘resource servers’) without exposing account credentials. The client presents an access token as a means of authenticating, which application servers can also use to find more information about the level of access granted. The authorization server handles the granting of access and inquiries about access.

In the context of AMQ Streams:

The brokers and clients communicate with the OAuth 2.0 authorization server, as necessary, to obtain or validate access tokens.

For a deployment of AMQ Streams, OAuth 2.0 integration provides:

You can use Red Hat Single Sign-On to:

See Using OAuth 2.0 token-based authentication.

Debezium for Change Data Capture is a distributed platform that monitors databases and creates change event streams. Debezium is built on Apache Kafka and can be deployed and integrated with AMQ Streams. Following a deployment of AMQ Streams, you deploy Debezium as a connector configuration through Kafka Connect. Debezium captures row-level changes to a database table and passes corresponding change events to AMQ Streams on OpenShift. Applications can read these change event streams and access the change events in the order in which they occurred.

Debezium has multiple uses, including:

Debezium provides connectors (based on Kafka Connect) for the following common databases:

For more information on deploying Debezium with AMQ Streams, refer to the product documentation.