Release Notes for AMQ Streams 2.5 on OpenShift

AMQ Streams 2.5.x is the Long Term Support (LTS) offering for AMQ Streams.

The latest patch release is AMQ Streams 2.5.2. The AMQ Streams product images have changed to version 2.5.2. Although the supported Kafka version is listed as 3.5.0, it incorporates updates and improvements from Kafka 3.5.2.

For information on the LTS terms and dates, see the AMQ Streams LTS Support Policy.

AMQ Streams 2.5 is supported on OpenShift Container Platform 4.12 and later.

For more information, see Chapter 11, Supported Configurations.

AMQ Streams supports and uses Apache Kafka version 3.5.0. Updates for Kafka 3.5.2 are incorporated with the 2.5.2 patch release. Only Kafka distributions built by Red Hat are supported.

You must upgrade the Cluster Operator to AMQ Streams version 2.5 before you can upgrade brokers and client applications to Kafka 3.5.0. For upgrade instructions, see Upgrading AMQ Streams.

Refer to the Kafka 3.5.0, Kafka 3.5.1, and Kafka 3.5.2 Release Notes for additional information.

Kafka 3.4.x is supported only for the purpose of upgrading to AMQ Streams 2.5.

The v1beta2 API version for all custom resources was introduced with AMQ Streams 1.7. For AMQ Streams 1.8, v1alpha1 and v1beta1 API versions were removed from all AMQ Streams custom resources apart from KafkaTopic and KafkaUser.

Upgrade of the custom resources to v1beta2 prepares AMQ Streams for a move to Kubernetes CRD v1, which is required for Kubernetes 1.22.

If you are upgrading from an AMQ Streams version prior to version 1.7:

This release introduces the KafkaNodePools feature gate and a new KafkaNodePool custom resource that enables the configuration of different pools of Apache Kafka nodes. This feature gate is at an alpha level of maturity, which means that it is disabled by default, and should be treated as a developer preview.

A node pool refers to a distinct group of Kafka nodes within a Kafka cluster. The KafkaNodePool custom resource represents the configuration for nodes only in the node pool. Each pool has its own unique configuration, which includes mandatory settings such as the number of replicas, storage configuration, and a list of assigned roles. As you can assign roles to the nodes in a node pool, you can try the feature with a Kafka cluster that uses ZooKeeper for cluster management or KRaft mode.

To enable the KafkaNodePools feature gate, specify +KafkaNodePools in the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +KafkaNodePools

See Configuring node pools.

This release also incorporates the UnidirectionalTopicOperator feature gate, introducing a unidirectional topic management mode. With unidirectional mode, you create Kafka topics using the KafkaTopic resource, which are then managed by the Topic Operator. This feature gate is at an alpha level of maturity, and should be treated as a developer preview.

To enable the UnidirectionalTopicOperator feature gate, specify +UnidirectionalTopicOperator in the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UnidirectionalTopicOperator

Up to this release, the only way to use the Topic Operator to manage topics was in bidirectional mode, which is compatible with using ZooKeeper for cluster management. Unidirectional mode does not require ZooKeeper for cluster management, which is an important development as Kafka moves to using KRaft mode for managing clusters.

See Using the Topic Operator.

The report.sh diagnostics tool is a script provided by Red Hat to gather essential data for troubleshooting AMQ Streams deployments on OpenShift. It collects relevant logs, configuration files, and other diagnostic data to assist in identifying and resolving issues. When you run the script, you can use additional parameters to retrieve specific data.

The tool requires the OpenShift oc command-line tool to establish a connection to the running cluster. After which you can open a terminal and run the tool to retrieve data on components.

From the following request, data is collected on a Kafka cluster, a Kafka Bridge cluster, and on secret keys and data values:

./report.sh --namespace=my-amq-streams-namespace --cluster=my-kafka-cluster --bridge=my-bridge-component --secrets=all --out-dir=~/reports

The data is output to a specified directory.

See Retrieving diagnostic and troubleshooting data.

OpenTelemetry for distributed tracing has moved to GA. You can use OpenTelemetry with a specified tracing system. OpenTelemetry has replaced OpenTracing for distributed tracing. Support for OpenTracing is deprecated.

By Default, OpenTelemetry uses the OTLP (OpenTelemetry Protocol) exporter for tracing. AMQ Streams with OpenTelemetry is distributed for use with the Jaeger exporter, but you can specify other tracing systems supported by OpenTelemetry. AMQ Streams plans to migrate to using OpenTelemetry with the OTLP exporter by default, and is phasing out support for the Jaeger exporter.

See Introducing distributed tracing.

The AMQ Streams 2.5.x release supports Kafka 3.5.0. Upgrading to the 2.5.2 patch release incorporates the updates and improvements from Kafka 3.5.2.

For an overview of the enhancements introduced with Kafka 3.5.x, refer to the Kafka 3.5.0, Kafka 3.5.1, and Kafka 3.5.2 Release Notes.

The UseStrimziPodSets feature gate has moved to GA, which means it is now permanently enabled and cannot be disabled.

StrimziPodSet resources are now used to manage pods instead of StatefulSet resources. This means that AMQ Streams handles the creation and management of pods instead of OpenShift, providing more control over the functionality.

See UseStrimziPodSets feature gate and Feature gate releases.

To deploy a Kafka cluster in KRaft mode, you must now enable the UseStrimziPodSets and KafkaNodePools feature gates. KRaft mode is supported only by using KafkaNodePool resources to manage the configuration of Kafka nodes.

For more information, see the following:

KeycloakRBACAuthorizer, the Red Hat Single Sign-On authorizer provided with AMQ Streams, has been replaced with the KeycloakAuthorizer. The new authorizer is compatible with using AMQ Streams with ZooKeeper cluster management or in KRaft mode. As with the previous authorizer, to be able to use the Red Hat Single Sign-On REST endpoints for Authorization Services provided by Red Hat Single Sign-On, you configure KeycloakAuthorizer on the Kafka broker. KeycloakRBACAuthorizer can still be used when using AMQ Streams with ZooKeeper cluster management, but you should migrate to the new authorizer.

You can now use additional configuration to manage OAuth 2.0 grants from the authorization server.

If you are using Red Hat Single Sign-On for OAuth 2.0 authorization, you can add the following properties to the authorization configuration of your Kafka brokers:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    authorization:
      type: keycloak
      tokenEndpointUri: <https://<auth_server_-_address>/auth/realms/external/protocol/openid-connect/token>
      clientId: kafka
      # ...
      grantsMaxIdleSeconds: 300
      grantsGcPeriodSeconds: 300
      grantsAlwaysLatest: false
    #...

See Configuring OAuth 2.0 authorization support.

To use OAuth 2.0 authentication in a Kafka cluster, you specify listener configuration in the Kafka custom resource with the authentication method oauth. When configuring the listener properties, it is now possible to use a JsonPath query to extract a username from the authorization server being used. You can use a JsonPath query to specify username extraction options in your listener for the userNameClaim and fallbackUserNameClaim properties. This allows you to extract a username from a token by accessing a specific value within a nested data structure. For example, you might have a username that is contained within a user info data structure within a JSON token data structure.

The following example shows how JsonPath queries are used with the properties when configuring token validation using an introspection endpoint.

- name: external
  port: 9094
  type: loadbalancer
  tls: true
  authentication:
    type: oauth
    validIssuerUri: <https://<auth-server-address>/auth/realms/external>
    introspectionEndpointUri: <https://<auth-server-address>/auth/realms/external/protocol/openid-connect/token/introspect>
    clientId: kafka-broker
    clientSecret:
      secretName: my-cluster-oauth
      key: clientSecret
    userNameClaim: "['user.info'].['user.id']" 

    maxSecondsWithoutReauthentication: 3600
    fallbackUserNameClaim: "['client.info'].['client.id']" 

    fallbackUserNamePrefix: client-account-
    # ...

See Configuring OAuth 2.0 support for Kafka brokers.

Support for Kafka Exporter deployment configuration introduces new properties to exclude specified topics and consumer groups from the metrics extracted from Kafka brokers.

You can use the following properties in the Kafka Exporter specification:

In the following example configuration, the two properties exclude topics and consumer groups that start with the prefix excluded-.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  # ...
  kafkaExporter:
    image: my-registry.io/my-org/my-exporter-cluster:latest
    groupRegex: ".*"
    topicRegex: ".*"
    groupExcludeRegex: "^excluded-.*"
    topicExcludeRegex: "^excluded-.*"
# ...

See KafkaExporterSpec schema reference.

The latest release of the Kafka Bridge introduces the following changes:

See Using the AMQ Streams Kafka Bridge.

Use the technology preview of the Kafka Static Quota plugin to set throughput and storage limits on brokers in your Kafka cluster. You enable the plugin and set limits by configuring the Kafka resource. You can set a byte-rate threshold and storage quotas to put limits on the clients interacting with your brokers.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    config:
      client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback
      client.quota.callback.static.produce: 1000000
      client.quota.callback.static.fetch: 1000000
      client.quota.callback.static.storage.soft: 400000000000
      client.quota.callback.static.storage.hard: 500000000000
      client.quota.callback.static.storage.check-interval: 5

See Setting limits on brokers using the Kafka Static Quota plugin.

To use KafkaNodePool resources to manage the configuration of pools of Kafka nodes, try the KafkaNodePools feature gate.

For more information, see Section 3.5, “(Preview) Node pools for managing nodes in a Kafka cluster”.

To set up the Topic Operator so that it only manages Kafka topics associated with KafkaTopic resources, try the UnidirectionalTopicOperator feature gate.

For more information, see Section 3.6, “(Preview) Unidirectional topic management using the Topic Operator”.

To use StrimziPodSet resources to manage Kafka Connect and Kafka MirrorMaker 2 pods, try the StableConnectIdentities feature gate.

The StableConnectIdentities feature gate controls the use of StrimziPodSet resources to manage Kafka Connect and Kafka MirrorMaker 2 pods instead of using OpenShift Deployment resources. This helps to minimize the number of rebalances of connector tasks.

To enable the StableConnectIdentities feature gate, specify +StableConnectIdentities as a value for the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +StableConnectIdentities

See StableConnectIdentities feature gate.

Apache Kafka is in the process of phasing out the need for ZooKeeper. With the new UseKRaft feature gate enabled, you can try deploying a Kafka cluster in KRaft (Kafka Raft metadata) mode without ZooKeeper.

To use KRaft mode, you must also use KafkaNodePool resources to manage the configuration of groups of nodes. To enable the UseKRaft feature gate, specify +UseKRaft,+KafkaNodePools as values for the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UseKRaft,+KafkaNodePools

Currently, the KRaft mode in AMQ Streams has the following major limitations:

See the following:

Kafka no longer includes the example file connectors FileStreamSourceConnector and FileStreamSinkConnector in its CLASSPATH and plugin.path by default. AMQ Streams has been updated so that you can still use these example connectors. The examples now have to be added to the plugin path like any connector.

Two example connector configuration files are provided:

See the following:

Support for RHEL 7 is deprecated in AMQ Streams 2.5.x. AMQ Streams 2.5.x (LTS) is the last LTS version to support RHEL 7.

In this release, the UseStrimziPodSets feature gate moved to GA, which means it is now permanently enabled and cannot be disabled. For this reason, support for StatefulSet resources to manage pods is no longer available.

The StatefulSet template properties in the Kafka custom resource (.spec.zookeeper.template.statefulSet and .spec.kafka.template.statefulSet) are deprecated and ignored. You should remove them from your custom resources.

Support for Java 8 was deprecated in Kafka 3.0.0 and AMQ Streams 2.0. Support for Java 8 was removed in AMQ Streams 2.4.0. This applies to all AMQ Streams components, including clients.

AMQ Streams supports Java 11 and Java 17. Use Java 11 or 17 when developing new applications. Plan to migrate any applications that currently use Java 8 to Java 11 or 17.

If you want to continue using Java 8 for the time being, AMQ Streams 2.2 provides Long Term Support (LTS). For information on the LTS terms and dates, see the AMQ Streams LTS Support Policy.

Support for type: jaeger tracing is deprecated.

The Jaeger clients are now retired and the OpenTracing project archived. As such, we cannot guarantee their support for future Kafka versions. We are introducing a new tracing implementation based on the OpenTelemetry project.

The operation property for configuring operations for ACL rules is deprecated. A new, more-streamlined configuration format using the operations property is now available.

authorization:
  type: simple
  acls:
    - resource:
        type: topic
        name: my-topic
      operations:
        - Read
        - Describe
        - Create
        - Write

The operation property for the old configuration format is deprecated, but still supported.

Identity replication policy is a feature used with MirrorMaker 2 to override the automatic renaming of remote topics. Instead of prepending the name with the source cluster’s name, the topic retains its original name. This setting is particularly useful for active/passive backups and data migration scenarios.

To implement an identity replication policy, you must specify a replication policy class (replication.policy.class) in the MirrorMaker 2 configuration. Previously, you could specify the io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy class included with the AMQ Streams mirror-maker-2-extensions component. However, this component is now deprecated and will be removed in the future. Therefore, it is recommended to update your implementation to use Kafka’s own replication policy class (org.apache.kafka.connect.mirror.IdentityReplicationPolicy).

See Configuring Kafka MirrorMaker 2.

Kafka MirrorMaker replicates data between two or more active Kafka clusters, within or across data centers. Kafka MirrorMaker 1 was deprecated in Kafka 3.0.0 and will be removed in Kafka 4.0.0. MirrorMaker 2 will be the only version available. MirrorMaker 2 is based on the Kafka Connect framework, connectors managing the transfer of data between clusters.

As a consequence, the AMQ Streams KafkaMirrorMaker custom resource which is used to deploy Kafka MirrorMaker 1 has been deprecated. The KafkaMirrorMaker resource will be removed from AMQ Streams when Kafka 4.0.0 is adopted.

If you are using MirrorMaker 1 (referred to as just MirrorMaker in the AMQ Streams documentation), use the KafkaMirrorMaker2 custom resource with the IdentityReplicationPolicy class. MirrorMaker 2 renames topics replicated to a target cluster. IdentityReplicationPolicy configuration overrides the automatic renaming. Use it to produce the same active/passive unidirectional replication as MirrorMaker 1.

See Configuring Kafka MirrorMaker 2.

The type property of ListenerStatus has been deprecated and will be removed in the future. ListenerStatus is used to specify the addresses of internal and external listeners. Instead of using the type, the addresses are now specified by name.

See ListenerStatus schema reference.

The Cruise Control TLS sidecar has been removed. As a result, the .spec.cruiseControl.tlsSidecar and .spec.cruiseControl.template.tlsSidecar properties are now deprecated. The properties are ignored and will be removed in the future.

The disk and cpuUtilization capacity configuration properties have been deprecated, are ignored, and will be removed in the future. The properties were used in setting capacity limits in optimization proposals to determine if resource-based optimization goals are being broken. Disk and CPU capacity limits are now automatically generated by AMQ Streams.

See Configuring and deploying Cruise Control with Kafka.

AMQ Streams 2.5.2 (Long Term Support) is the latest patch release. The patch release incorporates Kafka 3.5.2 updates.

For details of the issues fixed in Kafka 3.5.1 and 3.5.2, refer to the Kafka 3.5.1 and Kafka 3.5.2 Release Notes.

For additional details about the issues resolved in AMQ Streams 2.5.2, see AMQ Streams 2.5.x Resolved Issues.

For more information on the issue, see the Kafka 3.5.2 Release Notes.

For additional details about the issues resolved in AMQ Streams 2.5.1, see AMQ Streams 2.5.x Resolved Issues.

Deploying a Kafka instance on OpenShift Container Platform (OCP) version 4.16 triggers a continuous creation of dockercfg secrets within the Kafka namespace.

This issue is caused by the openshift.io/internal-registry-pull-secret-ref annotation being added to service accounts, which leads to a reconciliation loop as Streams for Apache Kafka and OpenShift repeatedly rewrite the annotation. Over time, this can result in the accumulation of thousands of unnecessary secrets.

If upgrading is not immediately possible, a temporary workaround is to manually configure the openshift.io/internal-registry-pull-secret-ref annotation for each service account to prevent the reconciliation loop.

template:
  serviceAccount:
    metadata:
      annotations:
        openshift.io/internal-registry-pull-secret-ref: my-cluster-entity-operator-dockercfg-qxwxd

If Cross-Origin Resource Sharing (CORS) is enabled for the Kafka Bridge, a 400 bad request error is returned when sending a HTTP request to produce messages.

To use CORS, you can deploy Red Hat 3scale for the Kafka Bridge.

The AMQ Streams Cluster Operator does not start on Internet Protocol version 6 (IPv6) clusters.

Cruise Control for AMQ Streams has a known issue that relates to the calculation of CPU utilization estimation. CPU utilization is calculated as a percentage of the defined capacity of a broker pod. The issue occurs when running Kafka brokers across nodes with varying CPU cores. For example, node1 might have 2 CPU cores and node2 might have 4 CPU cores. In this situation, Cruise Control can underestimate and overestimate CPU load of brokers The issue can prevent cluster rebalances when the pod is under heavy load.

There are two workarounds for this issue.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
  zookeeper:
    # ...
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 10000KB/s
      outboundNetwork: 10000KB/s
    config:
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      default.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

For more information, see Insufficient CPU capacity.

When running AMQ Streams in FIPS mode with JMX authentication enabled, clients may fail authentication. To work around this issue, do not enable JMX authentication while running in FIPS mode. We are investigating the issue and working to resolve it in a future release.

The following platforms are tested for AMQ Streams 2.5 running with Kafka on the version of OpenShift stated.

In AMQ Streams, only the following components released directly from the Apache Software Foundation are supported:

See also, Chapter 13, Supported integration with Red Hat products.

Kafka requires block storage; file storage options like NFS are not compatible.

AMQ Streams supports the use of OAuth 2.0 token-based authorization through Red Hat Single Sign-On Authorization Services, which allows you to manage security policies and permissions centrally.

If you deployed the Kafka Bridge on OpenShift Container Platform, you can use it with 3scale. 3scale API Management can secure the Kafka Bridge with TLS, and provide authentication and authorization. Integration with 3scale also means that additional features like metrics, rate limiting and billing are available.

For information on deploying 3scale, see Using 3scale API Management with the AMQ Streams Kafka Bridge.

The Red Hat build of Debezium is a distributed change data capture platform. It captures row-level changes in databases, creates change event records, and streams the records to Kafka topics. Debezium is built on Apache Kafka. You can deploy and integrate the Red Hat build of Debezium with AMQ Streams. Following a deployment of AMQ Streams, you deploy Debezium as a connector configuration through Kafka Connect. Debezium passes change event records to AMQ Streams on OpenShift. Applications can read these change event streams and access the change events in the order in which they occurred.

Debezium has multiple uses, including:

Debezium provides connectors (based on Kafka Connect) for the following common databases:

For more information on deploying Debezium with AMQ Streams, refer to the product documentation for the Red Hat build of Debezium.

You can use the Red Hat build of Apicurio Registry as a centralized store of service schemas for data streaming. For Kafka, you can use the Red Hat build of Apicurio Registry to store Apache Avro or JSON schema.

Apicurio Registry provides a REST API and a Java REST client to register and query the schemas from client applications through server-side endpoints.

Using Apicurio Registry decouples the process of managing schemas from the configuration of client applications. You enable an application to use a schema from the registry by specifying its URL in the client code.

For example, the schemas to serialize and deserialize messages can be stored in the registry, which are then referenced from the applications that use them to ensure that the messages that they send and receive are compatible with those schemas.

Kafka client applications can push or pull their schemas from Apicurio Registry at runtime.

For more information on using the Red Hat build of Apicurio Registry with AMQ Streams, refer to the product documentation for the Red Hat build of Apicurio Registry.

The Red Hat build of Apache Camel K is a lightweight integration framework built from Apache Camel K that runs natively in the cloud on OpenShift. Camel K supports serverless integration, which allows for development and deployment of integration tasks without the need to manage the underlying infrastructure. You can use Camel K to build and integrate event-driven applications with your AMQ Streams environment. For scenarios requiring real-time data synchronization between different systems or databases, Camel K can be used to capture and transform change in events and send them to AMQ Streams for distribution to other systems.

For more information on using the Camel K with AMQ Streams, refer to the product documentation for the Red Hat build of Apache Camel K.

Revised on 2024-08-22 09:41:45 UTC