Deploying and Upgrading AMQ Streams on OpenShift

Procedure

1. Deploy Kafka Connect to your OpenShift cluster. For a Kafka cluster with 3 or more brokers, use the examples/connect/kafka-connect.yaml file. For a Kafka cluster with less than 3 brokers, use the examples/connect/kafka-connect-single-node-kafka.yaml file.

   oc apply -f examples/connect/kafka-connect.yaml

2. Verify that Kafka Connect was successfully deployed:

   oc get deployments

Procedure

1. Edit the examples/connect/source-connector.yaml file:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-source-connector 1
     labels:
       strimzi.io/cluster: my-connect-cluster 2
   spec:
     class: org.apache.kafka.connect.file.FileStreamSourceConnector 3
     tasksMax: 2 4
     config: 5
       file: "/opt/kafka/LICENSE" 6
       topic: my-topic 7
       # ...

   1

   Name of the KafkaConnector resource, which is used as the name of the connector. Use any name that is valid for an OpenShift resource.

   2

   Name of the Kafka Connect cluster to create the connector instance in. Connectors must be deployed to the same namespace as the Kafka Connect cluster they link to.

   3

   Full name or alias of the connector class. This should be present in the image being used by the Kafka Connect cluster.

   4

   Maximum number of Kafka Connect Tasks that the connector can create.

   5

   Connector configuration as key-value pairs.

   6

   This example source connector configuration reads data from the /opt/kafka/LICENSE file.

   7

   Kafka topic to publish the source data to.

2. Create the source KafkaConnector in your OpenShift cluster:

   oc apply -f examples/connect/source-connector.yaml

3. Create an examples/connect/sink-connector.yaml file:

   touch examples/connect/sink-connector.yaml

4. Paste the following YAML into the sink-connector.yaml file:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: KafkaConnector
   metadata:
     name: my-sink-connector
     labels:
       strimzi.io/cluster: my-connect
   spec:
     class: org.apache.kafka.connect.file.FileStreamSinkConnector 1
     tasksMax: 2
     config: 2
       file: "/tmp/my-file" 3
       topics: my-topic 4

   1

   Full name or alias of the connector class. This should be present in the image being used by the Kafka Connect cluster.

   2

   Connector configuration as key-value pairs.

   3

   Temporary file to publish the source data to.

   4

   Kafka topic to read the source data from.

5. Create the sink KafkaConnector in your OpenShift cluster:

   oc apply -f examples/connect/sink-connector.yaml

6. Check that the connector resources were created:

   oc get kctr --selector strimzi.io/cluster=MY-CONNECT-CLUSTER -o name
   
   my-source-connector
   my-sink-connector

   Replace MY-CONNECT-CLUSTER with your Kafka Connect cluster.

7. In the container, execute kafka-console-consumer.sh to read the messages that were written to the topic by the source connector:

   oc exec MY-CLUSTER-kafka-0 -i -t -- bin/kafka-console-consumer.sh --bootstrap-server MY-CLUSTER-kafka-bootstrap.NAMESPACE.svc:9092 --topic my-topic --from-beginning

Procedure

1. Find the name of the KafkaConnector custom resource that controls the Kafka connector you want to restart:

   oc get KafkaConnector

2. To restart the connector, annotate the KafkaConnector resource in OpenShift. For example, using oc annotate:

   oc annotate KafkaConnector KAFKACONNECTOR-NAME strimzi.io/restart=true

3. Wait for the next reconciliation to occur (every two minutes by default).

   The Kafka connector is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the KafkaConnector custom resource.

Procedure

1. Find the name of the KafkaConnector custom resource that controls the Kafka connector task you want to restart:

   oc get KafkaConnector

2. Find the ID of the task to be restarted from the KafkaConnector custom resource. Task IDs are non-negative integers, starting from 0.

   oc describe KafkaConnector KAFKACONNECTOR-NAME

3. To restart the connector task, annotate the KafkaConnector resource in OpenShift. For example, using oc annotate to restart task 0:

   oc annotate KafkaConnector KAFKACONNECTOR-NAME strimzi.io/restart-task=0

4. Wait for the next reconciliation to occur (every two minutes by default).

   The Kafka connector task is restarted, as long as the annotation was detected by the reconciliation process. When Kafka Connect accepts the restart request, the annotation is removed from the KafkaConnector custom resource.

Procedure

1. Deploy Kafka MirrorMaker to your OpenShift cluster:

   For MirrorMaker:

   oc apply -f examples/mirror-maker/kafka-mirror-maker.yaml

   For MirrorMaker 2.0:

   oc apply -f examples/mirror-maker/kafka-mirror-maker-2.yaml

2. Verify that MirrorMaker was successfully deployed:

   oc get deployments

Procedure

1. Deploy Kafka Bridge to your OpenShift cluster:

   oc apply -f examples/bridge/kafka-bridge.yaml

2. Verify that Kafka Bridge was successfully deployed:

   oc get deployments

1. Update the Kafka resource in an editor.

   oc edit kafka KAFKA-CONFIG-FILE

2. Copy the example configuration in kafka-metrics.yaml to your own Kafka resource definition.
3. Save the file, and wait for the updated resource to be reconciled.

Procedure

1. Check that monitoring for user-defined projects is enabled:

   oc get pods -n openshift-user-workload-monitoring

   If enabled, pods for the monitoring components are returned. For example:

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-5cc59f9bc6-kgcq8   1/1     Running   0          25s
   prometheus-user-workload-0             5/5     Running   1          14s
   prometheus-user-workload-1             5/5     Running   1          14s
   thanos-ruler-user-workload-0           3/3     Running   0          14s
   thanos-ruler-user-workload-1           3/3     Running   0          14s

   If no pods are returned, monitoring for user-defined projects is disabled. See the Prerequisites in Section 7.3, “Viewing Kafka metrics and dashboards in OpenShift 4”.

2. Multiple PodMonitor resources are defined in examples/metrics/prometheus-install/strimzi-pod-monitor.yaml.

   For each PodMonitor resource, edit the spec.namespaceSelector.matchNames property:

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: cluster-operator-metrics
     labels:
       app: strimzi
   spec:
     selector:
       matchLabels:
         strimzi.io/kind: cluster-operator
     namespaceSelector:
       matchNames:
         - PROJECT-NAME 1
     podMetricsEndpoints:
     - path: /metrics
       port: http
   # ...

   1

   The project where the pods to scrape the metrics from are running, for example, Kafka.

3. Deploy the strimzi-pod-monitor.yaml file to the project where your Kafka cluster is running:

   oc apply -f strimzi-pod-monitor.yaml -n MY-PROJECT

4. Deploy the example Prometheus rules to the same project:

   oc apply -f prometheus-rules.yaml -n MY-PROJECT

Procedure

1. Create a ServiceAccount for Grafana. Here the resource is named grafana-serviceaccount.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: grafana-serviceaccount
     labels:
       app: strimzi

2. Deploy the ServiceAccount to the project containing your Kafka cluster:

   oc apply -f GRAFANA-SERVICEACCOUNT -n MY-PROJECT

3. Create a ClusterRoleBinding resource that assigns the cluster-monitoring-view role to the Grafana ServiceAccount. Here the resource is named grafana-cluster-monitoring-binding.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: grafana-cluster-monitoring-binding
     labels:
       app: strimzi
   subjects:
     - kind: ServiceAccount
       name: grafana-serviceaccount
       namespace: MY-PROJECT 1
   roleRef:
     kind: ClusterRole
     name: cluster-monitoring-view
     apiGroup: rbac.authorization.k8s.io

   1

   Name of your project.

4. Deploy the ClusterRoleBinding to the project containing your Kafka cluster:

   oc apply -f GRAFANA-CLUSTER-MONITORING-BINDING -n MY-PROJECT

Procedure

1. Get the access token of the Grafana ServiceAccount:

   oc serviceaccounts get-token grafana-serviceaccount -n MY-PROJECT

   Copy the access token to use in the next step.

2. Create a datasource.yaml file containing the Thanos Querier configuration for Grafana.

   Paste the access token into the httpHeaderValue1 property as indicated.

   apiVersion: 1
   
   datasources:
   - name: Prometheus
     type: prometheus
     url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
     access: proxy
     basicAuth: false
     withCredentials: false
     isDefault: true
     jsonData:
       timeInterval: 5s
       tlsSkipVerify: true
       httpHeaderName1: "Authorization"
     secureJsonData:
       httpHeaderValue1: "Bearer ${GRAFANA-ACCESS-TOKEN}" 1
     editable: true

   1

   GRAFANA-ACCESS-TOKEN: The value of the access token for the Grafana ServiceAccount.

3. Create a Config Map named grafana-config from the datasource.yaml file:

   oc create configmap grafana-config --from-file=datasource.yaml -n MY-PROJECT

4. Create a Grafana application consisting of a Deployment and a Service.

   The grafana-config Config Map is mounted as a volume for the datasource configuration.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     replicas: 1
     selector:
       matchLabels:
         name: grafana
     template:
       metadata:
         labels:
           name: grafana
       spec:
         serviceAccountName: grafana-serviceaccount
         containers:
         - name: grafana
           image: grafana/grafana:6.3.0
           ports:
           - name: grafana
             containerPort: 3000
             protocol: TCP
           volumeMounts:
           - name: grafana-data
             mountPath: /var/lib/grafana
           - name: grafana-logs
             mountPath: /var/log/grafana
           - name: grafana-config
             mountPath: /etc/grafana/provisioning/datasources/datasource.yaml
             readOnly: true
             subPath: datasource.yaml
           readinessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 5
             periodSeconds: 10
           livenessProbe:
             httpGet:
               path: /api/health
               port: 3000
             initialDelaySeconds: 15
             periodSeconds: 20
         volumes:
         - name: grafana-data
           emptyDir: {}
         - name: grafana-logs
           emptyDir: {}
         - name: grafana-config
           configMap:
             name: grafana-config
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: grafana
     labels:
       app: strimzi
   spec:
     ports:
     - name: grafana
       port: 3000
       targetPort: 3000
       protocol: TCP
     selector:
       name: grafana
     type: ClusterIP

5. Deploy the Grafana application to the project containing your Kafka cluster:

   oc apply -f GRAFANA-APPLICATION -n MY-PROJECT

Procedure

1. Get the details of the Route to the Grafana Service. For example:

   oc get routes
   
   NAME               HOST/PORT                         PATH  SERVICES
   MY-GRAFANA-ROUTE   MY-GRAFANA-ROUTE-amq-streams.net        grafana

2. In a web browser, access the Grafana login screen using the URL for the Route host and port.

3. Enter your user name and password, and then click Log In.

   The default Grafana user name and password are both admin. After logging in for the first time, you can change the password.

4. In Configuration > Data Sources, check that the Prometheus data source was created. The data source was created in Section 7.3.3, “Deploying Grafana with a Prometheus datasource”.
5. Click Dashboards > Manage, and then click Import.
6. In examples/metrics/grafana-dashboards, copy the JSON of the dashboard to import.
7. Paste the JSON into the text box, and then click Load.
8. Repeat steps 1 -7 for the other example Grafana dashboards.

Procedure

1. Edit the KafkaExporter properties for the Kafka resource.

   The properties you can configure are shown in this example configuration:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     # ...
     kafkaExporter:
       image: my-registry.io/my-org/my-exporter-cluster:latest 1
       groupRegex: ".*" 2
       topicRegex: ".*" 3
       resources: 4
         requests:
           cpu: 200m
           memory: 64Mi
         limits:
           cpu: 500m
           memory: 128Mi
       logging: debug 5
       enableSaramaLogging: true 6
       template: 7
         pod:
           metadata:
             labels:
               label1: value1
           imagePullSecrets:
             - name: my-docker-credentials
           securityContext:
             runAsUser: 1000001
             fsGroup: 0
           terminationGracePeriodSeconds: 120
       readinessProbe: 8
         initialDelaySeconds: 15
         timeoutSeconds: 5
       livenessProbe: 9
         initialDelaySeconds: 15
         timeoutSeconds: 5
   # ...

   1

   ADVANCED OPTION: Container image configuration, which is recommended only in special situations.

   2

   A regular expression to specify the consumer groups to include in the metrics.

   3

   A regular expression to specify the topics to include in the metrics.

   4

   CPU and memory resources to reserve.

   5

   Logging configuration, to log messages with a given severity (debug, info, warn, error, fatal) or above.

   6

   Boolean to enable Sarama logging, a Go client library used by Kafka Exporter.

   7

   Customization of deployment templates and pods.

   8

   Healthcheck readiness probes.

   9

   Healthcheck liveness probes.

2. Create or update the resource:

   oc apply -f kafka.yaml

Procedure

1. Access the Grafana user interface.

2. Select the Strimzi Kafka Exporter dashboard.

   When metrics data has been collected for some time, the Kafka Exporter charts are populated.

   AMQ Streams Kafka Exporter

   Shows metrics for:

   • Topic count
   • Partition count
   • Replicas count
   • In-sync replicas count
   • Under-replicated partitions count
   • Partitions which are at their minimum in sync replica count
   • Partitions which are under their minimum in sync replica count
   • Partitions not on a preferred node
   • Messages in per second from topics
   • Messages consumed per second from topics
   • Messages consumed per minute by consumer groups
   • Lag by consumer group
   • Number of partitions
   • Latest offsets
   • Oldest offsets

Procedure

1. Run the API conversion tool with the convert-file command and appropriate flags.

   Example 1, converts a YAML file and displays the output, though the file does not change:

   bin/api-conversion.sh convert-file --file input.yaml

   Example 2, converts a YAML file, and writes the changes into the original source file:

   bin/api-conversion.sh convert-file --file input.yaml --in-place

   Example 3, converts a YAML file, and writes the changes into a new output file:

   bin/api-conversion.sh convert-file --file input.yaml --output output.yaml

2. Update the custom resources using the converted configuration file.

   oc apply -f CONVERTED-CONFIG-FILE

3. Verify that the custom resources have been converted.

   oc get KIND CUSTOM-RESOURCE-NAME -o yaml

Procedure

1. Run the API conversion tool with the convert-resource command and appropriate flags.

   Example 1, converts all AMQ Streams resources in current namespace:

   bin/api-conversion.sh convert-resource

   Example 2, converts all AMQ Streams resources in all namespaces:

   bin/api-conversion.sh convert-resource --all-namespaces

   Example 3, converts all AMQ Streams resources in the my-kafka namespace:

   bin/api-conversion.sh convert-resource --namespace my-kafka

   Example 4, converts only Kafka resources in all namespaces:

   bin/api-conversion.sh convert-resource --all-namespaces --kind Kafka

   Example 5, converts Kafka and Kafka Connect resources in all namespaces:

   bin/api-conversion.sh convert-resource --all-namespaces --kind Kafka --kind KafkaConnect

   Example 6, converts a Kafka custom resource named my-cluster in the my-kafka namespace:

   bin/api-conversion.sh convert-resource --kind Kafka --namespace my-kafka --name my-cluster

2. Verify that the custom resources have been converted.

   oc get KIND CUSTOM-RESOURCE-NAME -o yaml

Procedure

1. If you have not done so, convert your custom resources to use v1beta2.

   You can use the API conversion tool to do this in one of two ways:

   • Section 8.2.2, “Converting custom resources configuration files using the API conversion tool”

   • Section 8.2.3, “Converting custom resources directly using the API conversion tool”

     Or you can make the changes manually.

2. Run the API conversion tool with the crd-upgrade command.

   bin/api-conversion.sh crd-upgrade

3. Verify that the CRDs have been upgraded so that v1beta2 is the storage version.

   For example, for the Kafka topic CRD:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: CustomResourceDefinition
   metadata:
     name: kafkatopics.kafka.strimzi.io
     #...
   spec:
     group: kafka.strimzi.io
     #...
     versions:
     - name: v1beta2
       served: true
       storage: true
       #...
   status:
     #...
     storedVersions:
     - v1beta2

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. If you have not already done so, update .spec.kafka.listener to the new generic listener format, as described in Section 8.1.4, “Updating listeners to the generic listener configuration”.

   Warning

   The old listener format is not supported in API version v1beta2.

3. If present, move affinity from .spec.kafka.affinity to .spec.kafka.template.pod.affinity.
4. If present, move tolerations from .spec.kafka.tolerations to .spec.kafka.template.pod.tolerations.
5. If present, remove .spec.kafka.template.tlsSidecarContainer.
6. If present, remove .spec.kafka.tlsSidecarContainer.

7. If either of the following policy configurations exist:

   • .spec.kafka.template.externalBootstrapService.externalTrafficPolicy

   • .spec.kafka.template.perPodService.externalTrafficPolicy

     1. Move the configuration to .spec.kafka.listeners[].configuration.externalTrafficPolicy, for both type: loadbalancer and type: nodeport listeners.
     2. Remove .spec.kafka.template.externalBootstrapService.externalTrafficPolicy or .spec.kafka.template.perPodService.externalTrafficPolicy.

8. If either of the following loadbalancer listener configurations exist:

   • .spec.kafka.template.externalBootstrapService.loadBalancerSourceRanges

   • .spec.kafka.template.perPodService.loadBalancerSourceRanges

     1. Move the configuration to .spec.kafka.listeners[].configuration.loadBalancerSourceRanges, for type: loadbalancer listeners.
     2. Remove .spec.kafka.template.externalBootstrapService.loadBalancerSourceRanges or .spec.kafka.template.perPodService.loadBalancerSourceRanges.

9. If type: external logging is configured in .spec.kafka.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

10. If the .spec.kafka.metrics field is used to enable metrics:

    1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.kafka.metrics field.

       kind: ConfigMap
       apiVersion: v1
       metadata:
         name: kafka-metrics
         labels:
           app: strimzi
       data:
         kafka-metrics-config.yaml: |
             <YAML>

    2. Add a .spec.kafka.metricsConfig property that points to the ConfigMap and key:

       metricsConfig:
         type: jmxPrometheusExporter
         valueFrom:
           configMapKeyRef:
             name: kafka-metrics
             key: kafka-metrics-config.yaml

    3. Delete the old .spec.kafka.metrics field.
11. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. If present, move affinity from .spec.zookeeper.affinity to .spec.zookeeper.template.pod.affinity.
3. If present, move tolerations from .spec.zookeeper.tolerations to .spec.zookeeper.template.pod.tolerations.
4. If present, remove .spec.zookeeper.template.tlsSidecarContainer.
5. If present, remove .spec.zookeeper.tlsSidecarContainer.

6. If type: external logging is configured in .spec.kafka.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

7. If the .spec.zookeeper.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.zookeeper.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-metrics
        labels:
          app: strimzi
      data:
        zookeeper-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.zookeeper.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-metrics
            key: zookeeper-metrics-config.yaml

   3. Delete the old .spec.zookeeper.metrics field.
8. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. If Kafka.spec.topicOperator is used:

   1. Move affinity from .spec.topicOperator.affinity to .spec.entityOperator.template.pod.affinity.
   2. Move tolerations from .spec.topicOperator.tolerations to .spec.entityOperator.template.pod.tolerations.
   3. Move .spec.topicOperator.tlsSidecar to .spec.entityOperator.tlsSidecar.
   4. After moving affinity, tolerations, and tlsSidecar, move the remaining configuration in .spec.topicOperator to .spec.entityOperator.topicOperator.

3. If type: external logging is configured in .spec.topicOperator.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j2.properties

   Note

   You can also complete this step as part of the Entity Operator upgrade.

4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. Move affinity from .spec.entityOperator.affinity to .spec.entityOperator.template.pod.affinity.
3. Move tolerations from .spec.entityOperator.tolerations to .spec.entityOperator.template.pod.tolerations.

4. If type: external logging is configured in .spec.entityOperator.userOperator.logging or .spec.entityOperator.topicOperator.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j2.properties

5. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. If type: external logging is configured in .spec.cruiseControl.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j2.properties

3. If the .spec.cruiseControl.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.cruiseControl.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-metrics
        labels:
          app: strimzi
      data:
        cruise-control-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.cruiseControl.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-metrics
            key: cruise-control-metrics-config.yaml

   3. Delete the old .spec.cruiseControl.metrics field.
4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the Kafka custom resource in an editor.

   oc edit kafka KAFKA-CLUSTER

2. Update the apiVersion of the Kafka custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaConnect custom resource in an editor.

   oc edit kafkaconnect KAFKA-CONNECT-CLUSTER

2. If present, move:

   KafkaConnect.spec.affinity

   KafkaConnect.spec.tolerations

   to:

   KafkaConnect.spec.template.pod.affinity

   KafkaConnect.spec.template.pod.tolerations

   For example, move:

   spec:
     # ...
     affinity:
       # ...
     tolerations:
       # ...

   to:

   spec:
     # ...
     template:
       pod:
         affinity:
           # ...
         tolerations:
           # ...

3. If type: external logging is configured in .spec.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

4. If the .spec.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-connect-metrics
        labels:
          app: strimzi
      data:
        connect-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-connect-metrics
            key: connect-metrics-config.yaml

   3. Delete the old .spec.metrics field.

5. Update the apiVersion of the KafkaConnect custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaConnectS2I custom resource in an editor.

   oc edit kafkaconnects2i S2I-CLUSTER

2. If present, move:

   KafkaConnectS2I.spec.affinity

   KafkaConnectS2I.spec.tolerations

   to:

   KafkaConnectS2I.spec.template.pod.affinity

   KafkaConnectS2I.spec.template.pod.tolerations

   For example, move:

   spec:
     # ...
     affinity:
       # ...
     tolerations:
       # ...

   to:

   spec:
     # ...
     template:
       pod:
         affinity:
           # ...
         tolerations:
           # ...

3. If type: external logging is configured in .spec.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

4. If the .spec.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-connect-s2i-metrics
        labels:
          app: strimzi
      data:
        connect-s2i-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-connect-s2i-metrics
            key: connect-s2i-metrics-config.yaml

   3. Delete the old .spec.metrics field

5. Update the apiVersion of the KafkaConnectS2I custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaMirrorMaker custom resource in an editor.

   oc edit kafkamirrormaker MIRROR-MAKER

2. If present, move:

   KafkaMirrorMaker.spec.affinity

   KafkaMirrorMaker.spec.tolerations

   to:

   KafkaMirrorMaker.spec.template.pod.affinity

   KafkaMirrorMaker.spec.template.pod.tolerations

   For example, move:

   spec:
     # ...
     affinity:
       # ...
     tolerations:
       # ...

   to:

   spec:
     # ...
     template:
       pod:
         affinity:
           # ...
         tolerations:
           # ...

3. If type: external logging is configured in .spec.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

4. If the .spec.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-mm-metrics
        labels:
          app: strimzi
      data:
        mm-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-mm-metrics
            key: mm-metrics-config.yaml

   3. Delete the old .spec.metrics field.

5. Update the apiVersion of the KafkaMirrorMaker custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

6. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaMirrorMaker2 custom resource in an editor.

   oc edit kafkamirrormaker2 MIRROR-MAKER-2

2. If present, move affinity from .spec.affinity to .spec.template.pod.affinity.
3. If present, move tolerations from .spec.tolerations to .spec.template.pod.tolerations.

4. If type: external logging is configured in .spec.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j.properties

5. If the .spec.metrics field is used to enable metrics:

   1. Create a new ConfigMap that stores the YAML configuration for the JMX Prometheus exporter under a key. The YAML must match what is currently in the .spec.metrics field.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kafka-mm2-metrics
        labels:
          app: strimzi
      data:
        mm2-metrics-config.yaml: |
            <YAML>

   2. Add a .spec.metricsConfig property that points to the ConfigMap and key:

      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            name: kafka-mm2-metrics
            key: mm2-metrics-config.yaml

   3. Delete the old .spec.metrics field.

6. Update the apiVersion of the KafkaMirrorMaker2 custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1alpha1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

7. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaBridge custom resource in an editor.

   oc edit kafkabridge KAFKA-BRIDGE

2. If type: external logging is configured in KafkaBridge.spec.logging:

   Replace the name of the ConfigMap containing the logging configuration:

   logging:
     type: external
     name: my-config-map

   With the valueFrom.configMapKeyRef field, and specify both the ConfigMap name and the key under which the logging is stored:

   logging:
     type: external
     valueFrom:
       configMapKeyRef:
         name: my-config-map
         key: log4j2.properties

3. Update the apiVersion of the KafkaBridge custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1alpha1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

4. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaUser custom resource in an editor.

   oc edit kafkauser KAFKA-USER

2. Update the apiVersion of the KafkaUser custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaTopic custom resource in an editor.

   oc edit kafkatopic KAFKA-TOPIC

2. Update the apiVersion of the KafkaTopic custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1beta1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaConnector custom resource in an editor.

   oc edit kafkaconnector KAFKA-CONNECTOR

2. Update the apiVersion of the KafkaConnector custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1alpha1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

1. Update the KafkaRebalance custom resource in an editor.

   oc edit kafkarebalance KAFKA-REBALANCE

2. Update the apiVersion of the KafkaRebalance custom resource to v1beta2:

   Replace:

   apiVersion: kafka.strimzi.io/v1alpha1

   with:

   apiVersion: kafka.strimzi.io/v1beta2

3. Save the file, exit the editor and wait for the updated custom resource to be reconciled.

Procedure

1. Update the Kafka cluster configuration.

   oc edit kafka KAFKA-CONFIGURATION-FILE

2. Change the Kafka.spec.kafka.version to specify the previous version.

   For example, if downgrading from Kafka 2.7.0 to 2.6.0:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.6.0 1
       config:
         log.message.format.version: "2.6" 2
         inter.broker.protocol.version: "2.6" 3
         # ...

   1

   Kafka version is changed to the previous version.

   2

   Message format version is unchanged.

   3

   Inter-broker protocol version is unchanged.

   Note

   You must format the value of log.message.format.version and inter.broker.protocol.version as a string to prevent it from being interpreted as a floating point number.

3. If the image for the Kafka version is different from the image defined in STRIMZI_KAFKA_IMAGES for the Cluster Operator, update Kafka.spec.kafka.image.

   See Section 8.1.3.1, “Kafka version and image mappings”

4. Save and exit the editor, then wait for rolling updates to complete.

   Check the update in the logs or by watching the pod state transitions:

   oc logs -f CLUSTER-OPERATOR-POD-NAME | grep -E "Kafka version downgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

   oc get pod -w

   Check the Cluster Operator logs for an INFO level message:

   Reconciliation #NUM(watch) Kafka(NAMESPACE/NAME): Kafka version downgrade from FROM-VERSION to TO-VERSION, phase 1 of 1 completed

5. Downgrade all client applications (consumers) to use the previous version of the client binaries.

   The Kafka cluster and clients are now using the previous Kafka version.

6. If you are reverting back to a version of AMQ Streams earlier than 0.22, which uses ZooKeeper for the storage of topic metadata, delete the internal topic store topics from the Kafka cluster.

   oc run kafka-admin -ti --image=registry.redhat.io/amq7/amq-streams-kafka-27-rhel7:1.7.0 --rm=true --restart=Never -- ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi-topic-operator-kstreams-topic-store-changelog --delete && ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __strimzi_store_topic --delete