19.5. Configuring and deploying Cruise Control with Kafka

Procedure

1. Edit the cruiseControl property for the Kafka resource.

   The properties you can configure are shown in this example configuration:

   apiVersion: kafka.strimzi.io/v1beta2
   kind: Kafka
   metadata:
     name: my-cluster
   spec:
     # ...
     cruiseControl:
       brokerCapacity: 

   1

   
         inboundNetwork: 10000KB/s
         outboundNetwork: 10000KB/s
         overrides: 

   2

   
         - brokers: [0]
           inboundNetwork: 20000KiB/s
           outboundNetwork: 20000KiB/s
         - brokers: [1, 2]
           inboundNetwork: 30000KiB/s
           outboundNetwork: 30000KiB/s
         # ...
       config: 

   3

   
         # Note that `default.goals` (superset) must also include all `hard.goals` (subset)
         default.goals: > 

   4

   
           com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
           com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
           com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal
           # ...
         hard.goals: >
           com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal
           # ...
         cpu.balance.threshold: 1.1
         metadata.max.age.ms: 300000
         send.buffer.bytes: 131072
         webserver.http.cors.enabled: true 

   5

   
         webserver.http.cors.origin: "*"
         webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type"
         # ...
       resources: 

   6

   
         requests:
           cpu: 1
           memory: 512Mi
         limits:
           cpu: 2
           memory: 2Gi
       logging: 

   7

   
           type: inline
           loggers:
             rootLogger.level: INFO
       template: 

   8

   
         pod:
           metadata:
             labels:
               label1: value1
           securityContext:
             runAsUser: 1000001
             fsGroup: 0
           terminationGracePeriodSeconds: 120
       readinessProbe: 

   9

   
         initialDelaySeconds: 15
         timeoutSeconds: 5
       livenessProbe:
         initialDelaySeconds: 15
         timeoutSeconds: 5
       metricsConfig: 

   10

   
         type: jmxPrometheusExporter
         valueFrom:
           configMapKeyRef:
             name: cruise-control-metrics
             key: metrics-config.yml
   # ...

   1

   Capacity limits for broker resources.

   2

   Overrides set network capacity limits for specific brokers when running on nodes with heterogeneous network resources.

   3

   Cruise Control configuration. Standard Cruise Control configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka.

   4

   Optimization goals configuration, which can include configuration for default optimization goals (default.goals), main optimization goals (goals), and hard goals (hard.goals).

   5

   CORS enabled and configured for read-only access to the Cruise Control API.

   6

   Requests for reservation of supported resources, currently cpu and memory, and limits to specify the maximum resources that can be consumed.

   7

   Cruise Control loggers and log levels added directly (inline) or indirectly (external) through a ConfigMap. A custom Log4j configuration must be placed under the log4j.properties key in the ConfigMap. Cruise Control has a single logger named rootLogger.level. You can set the log level to INFO, ERROR, WARN, TRACE, DEBUG, FATAL or OFF.

   8

   Template customization. Here a pod is scheduled with additional security attributes.

   9

   Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness).

   10

   Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter (the default metrics exporter).

2. Create or update the resource:

   oc apply -f <kafka_configuration_file>

3. Check the status of the deployment:

   oc get deployments -n <my_cluster_operator_namespace>

   Output shows the deployment name and readiness

   NAME                      READY  UP-TO-DATE  AVAILABLE
   my-cluster-cruise-control 1/1    1           1

   my-cluster is the name of the Kafka cluster.

   READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1.