홈
제품
Streams for Apache Kafka
2.9
Deploying and Managing Streams for Apache Kafka on OpenShift
Chapter 22. Using Cruise Control to modify topic replication factor

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 22. Using Cruise Control to modify topic replication factor

Change the replication factor of topics by updating the KafkaTopic resource managed by the Topic Operator. You can adjust the replication factor for specific purposes, such as:

Setting a lower replication factor for non-critical topics or because of resource shortages
Setting a higher replication factor to improve data durability and fault tolerance

The Topic Operator uses Cruise Control to make the necessary changes, so Cruise Control must be deployed with Streams for Apache Kafka.

The Topic Operator watches and periodically reconciles all managed and unpaused KafkaTopic resources to detect changes to .spec.replicas configuration by comparing the replication factor of the topic in Kafka. One or more replication factor updates are then sent to Cruise Control for processing in a single request.

Progress is reflected in the status of the KafkaTopic resource.

Prerequisites

The Cluster Operator must be deployed.
The Topic Operator must be deployed to manage topics through the KafkaTopic custom resource.
Cruise Control is deployed with Kafka.

Procedure

Edit the KafkaTopic resource to change the replicas value.

In this procedure, we change the replicas value for my-topic from 1 to 3.

Kafka topic replication factor configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 1
  replicas: 3
  # ...

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 1
  replicas: 3
  # ...

Copy to Clipboard

Toggle word wrap

Apply the change to the KafkaTopic configuration and wait for the Topic Operator to update the topic.

Check the status of the KafkaTopic resource to make sure the request was successful:

oc get kafkatopics my-topic -o yaml

oc get kafkatopics my-topic -o yaml

Copy to Clipboard

Toggle word wrap

Status for the replication factor change

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 1
  replicas: 3
  # ...

# ...
status:
  conditions:
  - lastTransitionTime: "2024-01-18T16:13:50.490918232Z"
    status: "True"
    type: Ready
  observedGeneration: 2
  replicasChange:
    sessionId: 1aa418ca-53ed-4b93-b0a4-58413c4fc0cb 
    state: ongoing 
    targetReplicas: 3 
  topicName: my-topic

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 1
  replicas: 3
  # ...

# ...
status:
  conditions:
  - lastTransitionTime: "2024-01-18T16:13:50.490918232Z"
    status: "True"
    type: Ready
  observedGeneration: 2
  replicasChange:
    sessionId: 1aa418ca-53ed-4b93-b0a4-58413c4fc0cb


    state: ongoing


    targetReplicas: 3


  topicName: my-topic

Copy to Clipboard

Toggle word wrap

1: The session ID for the Cruise Control operation, which is shown when process moves out of a pending state.
2: The state of the update. Moves from pending to ongoing, and then the entire replicasChange status is removed when the change is complete.
3: The requested change to the number of replicas.

An error message is shown in the status if the request fails before completion. The request is periodically retried if it enters a failed state.

Changing topic replication factor using the standalone Topic Operator

If you are using the standalone Topic Operator and aim to change the topic replication factor through configuration, you still need to use the Topic Operator in unidirectional mode alongside a Cruise Control deployment. You also need to include the following environment variables in the standalone Topic Operator deployment so that it can integrate with Cruise Control.

Example standalone Topic Operator deployment configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: strimzi-topic-operator
  labels:
    app: strimzi
spec:
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
        - name: strimzi-topic-operator
          # ...
          env:
            # ...
            - name: STRIMZI_CRUISE_CONTROL_ENABLED 
              value: true
            - name: STRIMZI_CRUISE_CONTROL_RACK_ENABLED 
              value: false
            - name: STRIMZI_CRUISE_CONTROL_HOSTNAME 
              value: cruise-control-api.namespace.svc
            - name: STRIMZI_CRUISE_CONTROL_PORT 
              value: 9090
            - name: STRIMZI_CRUISE_CONTROL_SSL_ENABLED 
              value: true
            - name: STRIMZI_CRUISE_CONTROL_AUTH_ENABLED 
              value: true

apiVersion: apps/v1
kind: Deployment
metadata:
  name: strimzi-topic-operator
  labels:
    app: strimzi
spec:
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
        - name: strimzi-topic-operator
          # ...
          env:
            # ...
            - name: STRIMZI_CRUISE_CONTROL_ENABLED


              value: true
            - name: STRIMZI_CRUISE_CONTROL_RACK_ENABLED


              value: false
            - name: STRIMZI_CRUISE_CONTROL_HOSTNAME


              value: cruise-control-api.namespace.svc
            - name: STRIMZI_CRUISE_CONTROL_PORT


              value: 9090
            - name: STRIMZI_CRUISE_CONTROL_SSL_ENABLED


              value: true
            - name: STRIMZI_CRUISE_CONTROL_AUTH_ENABLED


              value: true

Copy to Clipboard

Toggle word wrap

1: Integrates Cruise Control with the Topic Operator.
2: Flag to indicate whether rack awareness is enabled on the Kafka cluster. If so, replicas can be spread across different racks, data centers, or availability zones.
3: Cruise Control hostname.
4: Cruise control port.
5: Enables TLS authentication and encryption for accessing the Kafka cluster.
6: Enables basic authorization for accessing the Cruise Control API.

If you enable TLS authentication and authorization, mount the required certificates as follows:

Public certificates of the Cluster CA (certificate authority) in /etc/tls-sidecar/cluster-ca-certs/ca.crt
Basic authorization credentials (user name and password) in /etc/eto-cc-api/topic-operator.apiAdminName and /etc/eto-cc-api/topic-operator.apiAdminPassword

맨 위로 이동

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 22. Using Cruise Control to modify topic replication factor

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat 소개

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links