Chapter 2. Enhancements

For an overview of the enhancements introduced with Kafka 3.0.0, refer to the Kafka 3.0.0 Release Notes.

You can now disable the automatic creation of NetworkPolicy resources for listeners and use your own custom network policies instead.

By default, AMQ Streams automatically creates a NetworkPolicy resource for every listener that is enabled on a Kafka broker. For a particular listener, the network policy allow applications across all namespaces to connect. This behavior can be restricted using networkPolicyPeers.

To disable the auto-created NetworkPolicies, set the STRIMZI_NETWORK_POLICY_GENERATION environment variable to false in the Cluster Operator configuration.

See Cluster Operator configuration and Network policies.

The User Operator now uses the Kafka Admin API instead of ZooKeeper to manage the credentials of SCRAM-SHA-512 users. The Operator connects directly to Kafka for SCRAM-SHA-512 credentials management.

This change is part of ongoing work in the Apache Kafka project to remove Kafka’s dependency on ZooKeeper.

Some ZooKeeper related configuration has been deprecated and removed.

See Deprecated features and Using the User Operator.

You can deploy Kafka Connect with build configuration that automatically builds a container image with the connector plugins you require for your data connections. You can now specify the connector plugin artifacts as Maven coordinates.

AMQ Streams supports the following types of artifacts:

The Maven coordinates identify plugin artifacts and dependencies so that they can be located and fetched from a Maven repository.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  #...
  build:
    output:
      #...
    plugins:
      - name: my-plugin
        artifacts:
          - type: maven 
          - repository: https://mvnrepository.com 
          - group: org.apache.camel.kafkaconnector 
          - artifact: camel-kafka-connector 
          - version: 0.11.0 
  #...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  #...
  build:
    output:
      #...
    plugins:
      - name: my-plugin
        artifacts:
          - type: maven 

          - repository: https://mvnrepository.com 

          - group: org.apache.camel.kafkaconnector 

          - artifact: camel-kafka-connector 

          - version: 0.11.0 

  #...

See Build schema reference

Use the Environment Variables Configuration Provider plugin to load configuration data from environment variables.

You can use the provider to load configuration data for all Kafka components, including producers and consumers. Use the provider, for example, to provide the credentials for Kafka Connect connector configuration.

The values for the environment variables can be mapped from secrets or config maps. You can use the Environment Variables Configuration Provider, for example, to load certificates or JAAS configuration from environment variables mapped from OpenShift secrets.

See Loading configuration values from external sources

It is now possible to download connector plugin artifacts from insecure servers. This applies to JAR, TGZ, and other files that are downloaded and added to a container image. You can set this using the insecure property in the connector build configuration. When you specify the plugin artifact for download, you set the property to true. This disables all TLS verification. If the server is insecure, the artifact still downloads.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  #...
  build:
    output:
      #...
    plugins:
      - name: my-plugin
        artifacts:
          - type: tgz
            url: https://my-domain.tld/my-connector-archive.tgz
            sha512sum: 158...jg10
            insecure: true
  #...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  #...
  build:
    output:
      #...
    plugins:
      - name: my-plugin
        artifacts:
          - type: tgz
            url: https://my-domain.tld/my-connector-archive.tgz
            sha512sum: 158...jg10
            insecure: true
  #...

See Build schema reference

You can now specify the container registry secret used to pull base images for Kafka Connect builds on OpenShift. The secret is configured as a template customization. You can use the buildConfig template in the Kafka Connect spec to specify the secret using the pullSecret property. The secret contains the credentials for pulling the base image.

# ...
template:
  buildConfig:
    metadata:
      labels:
        label1: value1
        label2: value2
      annotations:
        annotation1: value1
        annotation2: value2
    pullSecret: "<secret_credentials>"
# ...

# ...
template:
  buildConfig:
    metadata:
      labels:
        label1: value1
        label2: value2
      annotations:
        annotation1: value1
        annotation2: value2
    pullSecret: "<secret_credentials>"
# ...

You can now configure the total local storage size for /tmp, the temporary emptyDir volume. The specification currently works for single containers running in a pod. The default storage size is 1Mi. The size is configured as a template customization. You can use the pod template in the spec of the resource to specify the volume size using the tmpDirSizeLimit property.

# ...
template:
  pod:
    tmpDirSizeLimit: "2Mi"
# ...

# ...
template:
  pod:
    tmpDirSizeLimit: "2Mi"
# ...

As a cluster administrator, you can use the Metering tool that is available on OpenShift to analyze what is happening in your cluster. You can now use the tool to generate reports and analyze your installed AMQ Streams components. Using Prometheus as a default data source, you can generate reports on pods, namespaces, and most other Kubernetes resources. You can also use the OpenShift Metering operator to analyze your installed AMQ Streams components to determine whether you are in compliance with your Red Hat subscription.

To use metering with AMQ Streams, you must first install and configure the Metering operator on OpenShift Container Platform.

See Using Metering on AMQ Streams