Release Notes for AMQ Streams 2.6 on OpenShift

AMQ Streams 2.6 is supported on OpenShift Container Platform 4.11 to 4.14.

For more information, see Chapter 9, Supported Configurations.

AMQ Streams now supports and uses Apache Kafka version 3.6.0. Only Kafka distributions built by Red Hat are supported.

You must upgrade the Cluster Operator to AMQ Streams version 2.6 before you can upgrade brokers and client applications to Kafka 3.6.0. For upgrade instructions, see Upgrading AMQ Streams.

Refer to the Kafka 3.6.0 Release Notes for additional information.

Kafka 3.5.x is supported only for the purpose of upgrading to AMQ Streams 2.6.

The v1beta2 API version for all custom resources was introduced with AMQ Streams 1.7. For AMQ Streams 1.8, v1alpha1 and v1beta1 API versions were removed from all AMQ Streams custom resources apart from KafkaTopic and KafkaUser.

Upgrade of the custom resources to v1beta2 prepares AMQ Streams for a move to Kubernetes CRD v1, which is required for Kubernetes 1.22.

If you are upgrading from an AMQ Streams version prior to version 1.7:

The StableConnectIdentities feature gate moves to a beta level of maturity and is now enabled by default.

The feature allows you to use StrimziPodSet resources to manage Kafka Connect and Kafka MirrorMaker 2 pods instead of using Deployment resources. This helps to minimize the number of rebalances of connector tasks.

To disable the StableConnectIdentities feature gate, specify -StableConnectIdentities as a value for the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: -StableConnectIdentities

env:
  - name: STRIMZI_FEATURE_GATES
    value: -StableConnectIdentities

See StableConnectIdentities feature gate.

AMQ Streams now prevents scale-down of a cluster if brokers are still in use. By default, AMQ Streams performs a check to ensure that there are no partition replicas on a broker before initiating a scale-down operation on a Kafka cluster. If scale-down is blocked, you must either revert the operation or move the residual partitions as quickly as possible so that the Cluster Operator can continue to manage the cluster.

However, there may be scenarios where you want to bypass this blocking mechanism. For example, it might be useful to disable the check on busy clusters. To do this, you annotate the Kafka resource by setting strimzi.io/skip-broker-scaledown-check="true".

See Skipping checks on scale-down operations.

You can now stop a Kafka Connect or Kafka MirrorMaker 2 connector from running. Compared to the paused state, where the connector and tasks stay instantiated, when stopped only the configuration of the connector is kept but nothing is actually running. Stopping a connector from running might be better suited for longer durations than just pausing. A paused connector is quicker to resume, but a stopped connector can free up memory and resources.

The pause property of the KafkaConnectorSpec schema and KafkaMirrorMaker2ConnectorSpec schema has been deprecated. Instead, both schemas now include a new state property. The state property allows you to configure one of the following values: running, paused, and stopped.

For example, if you want to stop a Kafka Connect connector resource, you change state to stopped in the configuration.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  config:
    file: "/opt/kafka/LICENSE"
    topic: my-topic
  state: stopped
    # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  config:
    file: "/opt/kafka/LICENSE"
    topic: my-topic
  state: stopped
    # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker2
spec:
  version: 3.6.0
  replicas: 3
  connectCluster: "my-cluster-target"
  clusters:
    # ...
  mirrors:
  - sourceCluster: "my-cluster-source"
    targetCluster: "my-cluster-target"
    sourceConnector:
      tasksMax: 10
      autoRestart:
        enabled: true
      state: stopped
  # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker2
spec:
  version: 3.6.0
  replicas: 3
  connectCluster: "my-cluster-target"
  clusters:
    # ...
  mirrors:
  - sourceCluster: "my-cluster-source"
    targetCluster: "my-cluster-target"
    sourceConnector:
      tasksMax: 10
      autoRestart:
        enabled: true
      state: stopped
  # ...

Stop and pause operations are also supported by the Kafka Connect REST API (when the KafkaConnector operator is not enabled) using the following REST endpoints:

A PUT /connectors/<connector_name>/resume request restarts stopped and paused connectors.

See KafkaConnectorSpec schema reference and KafkaMirrorMaker2ConnectorSpec schema reference.

You can now run parallel Kafka clusters that operate in KRaft mode (using Kafka Raft metadata) or use ZooKeeper for cluster management.

As KRaft mode is a developer preview, to be able to use a cluster in KRaft mode, you must do the following:

When the UseKRaft feature gate is enabled and the annotation is set, the Kafka cluster is deployed without ZooKeeper and operates in KRaft mode. If you don’t apply these settings, the Cluster Operator manages the Kafka resource as a ZooKeeper-based cluster.

To deploy a Kafka cluster in KRaft mode, you must now enable the UseKRaft and KafkaNodePools feature gates. As previews, both of these features are not intended for production. KRaft mode is supported only by using KafkaNodePool resources to manage the configuration of Kafka nodes. If you are using KRaft mode, you can specify roles for all nodes in a node pool to operate as brokers, controllers, or both. If you are using ZooKeeper, you can still use node pools, but nodes must be set as brokers only.

See Section 4.1, “UseKRaft feature gate”.

For an overview of the enhancements introduced with Kafka 3.6.0, refer to the Kafka 3.6.0 Release Notes.

It is now possible to pause reconciliation of KafkaTopic resources when the Topic Operator is operating in unidirectional mode (developer preview). To pause the reconciliation, you annotate the KafkaTopic resource with strimzi.io/pause-reconciliation="true".

See Pausing reconciliation of custom resources.

Metrics are now available for the Topic Operator running in unidirectional mode (technology preview). Strimzi operators automatically expose Prometheus metrics by default.

In addition to the standard JVM metrics, we provide the following custom metrics:

See Introducing metrics.

With the StableConnectIdentities feature gate enabled, you can now trigger manual rolling updates of Kafka Connect and Kafka MirrorMaker 2 pods.

You can annotate the StrimziPodSet resource that manages the pods you want to manually update with strimzi.io/manual-rolling-update=true to perform the rolling update:

oc annotate strimzipodset <cluster_name>-connect strimzi.io/manual-rolling-update=true

oc annotate strimzipodset <cluster_name>-mirrormaker2 strimzi.io/manual-rolling-update=true

oc annotate strimzipodset <cluster_name>-connect strimzi.io/manual-rolling-update=true

oc annotate strimzipodset <cluster_name>-mirrormaker2 strimzi.io/manual-rolling-update=true

Or you can update the actual Pod resource:

oc annotate pod <cluster_name>-connect-<index_number> strimzi.io/manual-rolling-update=true

oc annotate pod <cluster_name>-mirrormaker2-<index_number> strimzi.io/manual-rolling-update=true

oc annotate pod <cluster_name>-connect-<index_number> strimzi.io/manual-rolling-update=true

oc annotate pod <cluster_name>-mirrormaker2-<index_number> strimzi.io/manual-rolling-update=true

See Performing a rolling update using a pod management annotation and Performing a rolling update using a pod annotation.

When the auto-restart feature is enabled in KafkaConnector or KafkaMirrorMaker2 custom resources, it now continues to restart the connectors indefinitely rather than stopping after 7 restarts, as previously.

If you want to use the original behavior, use the .spec.autoRestart.maxRestarts option to configure the maximum number of restarts.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  autoRestart:
      enabled: true
      maxRestarts: 7
    # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector
  tasksMax: 2
  autoRestart:
      enabled: true
      maxRestarts: 7
    # ...

See Deploying KafkaConnector resources and Configuring Kafka MirrorMaker 2.

CPU capacity in Cruise Control is now determined using configuration values in the following order of precedence, with the highest priority first:

This order of precedence is the sequence in which different configuration values are considered when determining the actual capacity limit for a Kafka broker. For example, broker-specific overrides take precedence over capacity limits for all brokers. If none of the CPU capacity configurations are specified, the default CPU capacity for a Kafka broker is set to 1 CPU core.

See CruiseControlSpec schema reference.

The includeAcceptHeader configuration property has been added as an OAuth 2.0 server-side and client-side configuration option. Some authorization servers have issues with clients sending Accept: application/json headers. By setting includeAcceptHeader: false, the header is not sent. The default setting is true.

See Configuring OAuth 2.0 support for Kafka brokers, Configuring OAuth 2.0 for Kafka components, and Configuring OAuth 2.0 authorization support.

FIPS is now supported on the ppc64le (IBM Power) and s390x (IBM Z and IBM® LinuxONE) architectures.

You can now run Red Hat OpenShift Container Platform in disconnected environments on the ppc64le (IBM Power) and s390x (IBM Z and IBM® LinuxONE) architectures.

The KafkaNodePools feature gate and the new KafkaNodePool custom resource enables the configuration of different pools of Apache Kafka nodes. This feature gate is at an alpha level of maturity, which means that it is disabled by default, and should be treated as a technology preview.

A node pool refers to a distinct group of Kafka nodes within a Kafka cluster. The KafkaNodePool custom resource represents the configuration for nodes only in the node pool. Each pool has its own unique configuration, which includes mandatory settings such as the number of replicas, storage configuration, and a list of assigned roles. As you can assign roles to the nodes in a node pool, you can try the feature with a Kafka cluster that uses ZooKeeper for cluster management or KRaft mode.

To enable the KafkaNodePools feature gate, specify +KafkaNodePools in the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +KafkaNodePools

env:
  - name: STRIMZI_FEATURE_GATES
    value: +KafkaNodePools

See Configuring node pools.

The UnidirectionalTopicOperator feature gate introduces a unidirectional topic management mode. With unidirectional mode, you create Kafka topics using the KafkaTopic resource, which are then managed by the Topic Operator. This feature gate is at an alpha level of maturity, and should be treated as a technology preview.

To enable the UnidirectionalTopicOperator feature gate, specify +UnidirectionalTopicOperator in the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UnidirectionalTopicOperator

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UnidirectionalTopicOperator

Up to this release, the only way to use the Topic Operator to manage topics was in bidirectional mode, which is compatible with using ZooKeeper for cluster management. Unidirectional mode does not require ZooKeeper for cluster management, which is an important development as Kafka moves to using KRaft mode for managing clusters.

See Using the Topic Operator.

Use the technology preview of the Kafka Static Quota plugin to set throughput and storage limits on brokers in your Kafka cluster. You enable the plugin and set limits by configuring the Kafka resource. You can set a byte-rate threshold and storage quotas to put limits on the clients interacting with your brokers.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    config:
      client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback
      client.quota.callback.static.produce: 1000000
      client.quota.callback.static.fetch: 1000000
      client.quota.callback.static.storage.soft: 400000000000
      client.quota.callback.static.storage.hard: 500000000000
      client.quota.callback.static.storage.check-interval: 5

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    config:
      client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback
      client.quota.callback.static.produce: 1000000
      client.quota.callback.static.fetch: 1000000
      client.quota.callback.static.storage.soft: 400000000000
      client.quota.callback.static.storage.hard: 500000000000
      client.quota.callback.static.storage.check-interval: 5

See Setting limits on brokers using the Kafka Static Quota plugin.

Apache Kafka is in the process of phasing out the need for ZooKeeper. With the UseKRaft feature gate enabled, you can try deploying a Kafka cluster in KRaft (Kafka Raft metadata) mode without ZooKeeper.

To use KRaft mode, you must also use KafkaNodePool resources to manage the configuration of groups of nodes. To enable the UseKRaft feature gate, specify +UseKRaft,+KafkaNodePools as values for the STRIMZI_FEATURE_GATES environment variable in the Cluster Operator configuration.

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UseKRaft,+KafkaNodePools

env:
  - name: STRIMZI_FEATURE_GATES
    value: +UseKRaft,+KafkaNodePools

Currently, the KRaft mode in AMQ Streams has the following major limitations:

See UseKRaft feature gate and Feature gate releases.

Kafka no longer includes the example file connectors FileStreamSourceConnector and FileStreamSinkConnector in its CLASSPATH and plugin.path by default. AMQ Streams has been updated so that you can still use these example connectors. The examples now have to be added to the plugin path like any connector.

AMQ Streams provides an example connector configuration file with the configuration required to deploy the file connectors as KafkaConnector resources:

See Deploying example KafkaConnector resources and Extending Kafka Connect with connector plugins.

You can use configuration providers to load configuration data from external sources for all Kafka components, including producers and consumers.

Previously, you could enable the secrets.io.strimzi.kafka.EnvVarConfigProvider environment variable configuration provider using the config.providers properties in the spec configuration of a component. However, this provider is now deprecated and will be removed in the future. Therefore, it is recommended to update your implementation to use Kafka’s own environment variable configuration provider (org.apache.kafka.common.config.provider.EnvVarConfigProvider) to provide configuration properties as environment variables.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...
  config:
    # ...
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  # ...
  config:
    # ...
    config.providers: env
    config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
  # ...

The type property in the ListenerStatus schema has been deprecated and is no longer used. It has been replaced with the name property. The name property provides the name of a listener in the status of a Kafka resource.

status:
  clusterId: Y_RJQDGKRXmNF7fEcWldJQ
  conditions:
    - lastTransitionTime: '2023-11-29T14:59:37.113630Z'
      status: 'True'
      type: Ready
  kafkaVersion: 3.6.0
  listeners:
    # ...
    - addresses:
        - host: >-
            a8d4a6fb363bf44...
          port: 9094
      bootstrapServers: >-
        a8d4a6fb363bf447fb6e475...
      certificates:
        - |
          -----BEGIN CERTIFICATE-----

          -----END CERTIFICATE-----
      name: external3
  observedGeneration: 2
  operatorLastSuccessfulVersion: 2.6
 # ...

status:
  clusterId: Y_RJQDGKRXmNF7fEcWldJQ
  conditions:
    - lastTransitionTime: '2023-11-29T14:59:37.113630Z'
      status: 'True'
      type: Ready
  kafkaVersion: 3.6.0
  listeners:
    # ...
    - addresses:
        - host: >-
            a8d4a6fb363bf44...
          port: 9094
      bootstrapServers: >-
        a8d4a6fb363bf447fb6e475...
      certificates:
        - |
          -----BEGIN CERTIFICATE-----

          -----END CERTIFICATE-----
      name: external3
  observedGeneration: 2
  operatorLastSuccessfulVersion: 2.6
 # ...

The pause property of the KafkaConnectorSpec schema is now deprecated. Instead, a new state property allows you to choose a state value.

See Section 1.6, “Support for stopping connectors from running”.

The UseStrimziPodSets feature gate is now permanently enabled and cannot be disabled. For this reason, support for StatefulSet resources to manage pods is no longer available.

The StatefulSet template properties in the Kafka custom resource (.spec.zookeeper.template.statefulSet and .spec.kafka.template.statefulSet) are deprecated and ignored. You should remove them from your custom resources.

Support for Java 8 was deprecated in Kafka 3.0.0 and AMQ Streams 2.0. Support for Java 8 was removed in AMQ Streams 2.4.0. This applies to all AMQ Streams components, including clients.

AMQ Streams supports Java 11 and Java 17. Use Java 11 or 17 when developing new applications. Plan to migrate any applications that currently use Java 8 to Java 11 or 17.

If you want to continue using Java 8 for the time being, AMQ Streams 2.2 provides Long Term Support (LTS). For information on the LTS terms and dates, see the AMQ Streams LTS Support Policy.

Support for type: jaeger tracing is now removed.

The Jaeger clients are now retired and the OpenTracing project archived. As such, we cannot guarantee their support for future Kafka versions.

OpenTelemetry has replaced OpenTracing for distributed tracing.

See Introducing distributed tracing.

The operation property for configuring operations for ACL rules is deprecated. A new, more-streamlined configuration format using the operations property is now available.

authorization:
  type: simple
  acls:
    - resource:
        type: topic
        name: my-topic
      operations:
        - Read
        - Describe
        - Create
        - Write

authorization:
  type: simple
  acls:
    - resource:
        type: topic
        name: my-topic
      operations:
        - Read
        - Describe
        - Create
        - Write

The operation property for the old configuration format is deprecated, but still supported.

Identity replication policy is a feature used with MirrorMaker 2 to override the automatic renaming of remote topics. Instead of prepending the name with the source cluster’s name, the topic retains its original name. This setting is particularly useful for active/passive backups and data migration scenarios.

To implement an identity replication policy, you must specify a replication policy class (replication.policy.class) in the MirrorMaker 2 configuration. Previously, you could specify the io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy class included with the AMQ Streams mirror-maker-2-extensions component. However, this component is now deprecated and will be removed in the future. Therefore, it is recommended to update your implementation to use Kafka’s own replication policy class (org.apache.kafka.connect.mirror.IdentityReplicationPolicy).

See Configuring Kafka MirrorMaker 2.

Kafka MirrorMaker replicates data between two or more active Kafka clusters, within or across data centers. Kafka MirrorMaker 1 was deprecated in Kafka 3.0.0 and will be removed in Kafka 4.0.0. MirrorMaker 2 will be the only version available. MirrorMaker 2 is based on the Kafka Connect framework, connectors managing the transfer of data between clusters.

As a consequence, the AMQ Streams KafkaMirrorMaker custom resource which is used to deploy Kafka MirrorMaker 1 has been deprecated. The KafkaMirrorMaker resource will be removed from AMQ Streams when Kafka 4.0.0 is adopted.

If you are using MirrorMaker 1 (referred to as just MirrorMaker in the AMQ Streams documentation), use the KafkaMirrorMaker2 custom resource with the IdentityReplicationPolicy class. MirrorMaker 2 renames topics replicated to a target cluster. IdentityReplicationPolicy configuration overrides the automatic renaming. Use it to produce the same active/passive unidirectional replication as MirrorMaker 1.

See Configuring Kafka MirrorMaker 2.

The type property of ListenerStatus has been deprecated and will be removed in the future. ListenerStatus is used to specify the addresses of internal and external listeners. Instead of using the type, the addresses are now specified by name.

See ListenerStatus schema reference.

The Cruise Control TLS sidecar has been removed. As a result, the .spec.cruiseControl.tlsSidecar and .spec.cruiseControl.template.tlsSidecar properties are now deprecated. The properties are ignored and will be removed in the future.

The Cruise Control disk and cpuUtilization capacity configuration properties have been deprecated, are ignored, and will be removed in the future. The properties were used in setting capacity limits in optimization proposals to determine if resource-based optimization goals are being broken. Disk and CPU capacity limits are now automatically generated by AMQ Streams.

See Configuring and deploying Cruise Control with Kafka.

The new support for stopping connectors does not work in MirrorMaker 2 because the state is not passed to the connector operator. This will be fixed in the next release of AMQ Streams.

If Cross-Origin Resource Sharing (CORS) is enabled for the Kafka Bridge, a 400 bad request error is returned when sending a HTTP request to produce messages.

To use CORS, you can deploy Red Hat 3scale for the Kafka Bridge.

The AMQ Streams Cluster Operator does not start on Internet Protocol version 6 (IPv6) clusters.

Cruise Control for AMQ Streams has a known issue that relates to the calculation of CPU utilization estimation. CPU utilization is calculated as a percentage of the defined capacity of a broker pod. The issue occurs when running Kafka brokers across nodes with varying CPU cores. For example, node1 might have 2 CPU cores and node2 might have 4 CPU cores. In this situation, Cruise Control can underestimate and overestimate CPU load of brokers The issue can prevent cluster rebalances when the pod is under heavy load.

There are two workarounds for this issue.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
  zookeeper:
    # ...
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 10000KB/s
      outboundNetwork: 10000KB/s
    config:
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      default.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
  zookeeper:
    # ...
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 10000KB/s
      outboundNetwork: 10000KB/s
    config:
      hard.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
      default.goals: >
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,
        com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

For more information, see Insufficient CPU capacity.

When running AMQ Streams in FIPS mode with JMX authentication enabled, clients may fail authentication. To work around this issue, do not enable JMX authentication while running in FIPS mode. We are investigating the issue and working to resolve it in a future release.

The following platforms are tested for AMQ Streams 2.6 running with Kafka on the version of OpenShift stated.

OpenShift Local is a limited version of Red Hat OpenShift Container Platform (OCP). Use only for development and evaluation on the understanding that some features may be unavailable.

Only client libraries built by Red Hat are supported for AMQ Streams. Currently, AMQ Streams only provides a Java client library. Clients are supported for use with AMQ Streams 2.6 on the following operating systems and architectures:

Clients are tested with Open JDK 11 and 17. The IBM JDK is supported but not regularly tested against during each release. Open JDK 8, Oracle JDK 8 & 11, and IBM JDK 8 are not supported.

In AMQ Streams, only the following components released directly from the Apache Software Foundation are supported:

See also, Chapter 11, Supported integration with Red Hat products.

Kafka requires block storage; file storage options like NFS are not compatible.

AMQ Streams supports the use of OAuth 2.0 token-based authorization through Red Hat Single Sign-On Authorization Services, which allows you to manage security policies and permissions centrally.

If you deployed the Kafka Bridge on OpenShift Container Platform, you can use it with 3scale. 3scale API Management can secure the Kafka Bridge with TLS, and provide authentication and authorization. Integration with 3scale also means that additional features like metrics, rate limiting and billing are available.

For information on deploying 3scale, see Using 3scale API Management with the AMQ Streams Kafka Bridge.

The Red Hat build of Debezium is a distributed change data capture platform. It captures row-level changes in databases, creates change event records, and streams the records to Kafka topics. Debezium is built on Apache Kafka. You can deploy and integrate the Red Hat build of Debezium with AMQ Streams. Following a deployment of AMQ Streams, you deploy Debezium as a connector configuration through Kafka Connect. Debezium passes change event records to AMQ Streams on OpenShift. Applications can read these change event streams and access the change events in the order in which they occurred.

Debezium has multiple uses, including:

Debezium provides connectors (based on Kafka Connect) for the following common databases:

For more information on deploying Debezium with AMQ Streams, refer to the product documentation for the Red Hat build of Debezium.

You can use the Red Hat build of Apicurio Registry as a centralized store of service schemas for data streaming. For Kafka, you can use the Red Hat build of Apicurio Registry to store Apache Avro or JSON schema.

Apicurio Registry provides a REST API and a Java REST client to register and query the schemas from client applications through server-side endpoints.

Using Apicurio Registry decouples the process of managing schemas from the configuration of client applications. You enable an application to use a schema from the registry by specifying its URL in the client code.

For example, the schemas to serialize and deserialize messages can be stored in the registry, which are then referenced from the applications that use them to ensure that the messages that they send and receive are compatible with those schemas.

Kafka client applications can push or pull their schemas from Apicurio Registry at runtime.

For more information on using the Red Hat build of Apicurio Registry with AMQ Streams, refer to the product documentation for the Red Hat build of Apicurio Registry.

The Red Hat build of Apache Camel K is a lightweight integration framework built from Apache Camel K that runs natively in the cloud on OpenShift. Camel K supports serverless integration, which allows for development and deployment of integration tasks without the need to manage the underlying infrastructure. You can use Camel K to build and integrate event-driven applications with your AMQ Streams environment. For scenarios requiring real-time data synchronization between different systems or databases, Camel K can be used to capture and transform change in events and send them to AMQ Streams for distribution to other systems.

For more information on using the Camel K with AMQ Streams, refer to the product documentation for the Red Hat build of Apache Camel K.

Revised on 2023-12-19 17:46:51 UTC