Ce contenu n'est pas disponible dans la langue sélectionnée.
Chapter 6. Topics page
The Topics page shows all the topics created for a Kafka cluster. Use this page to check information on topics.
The Topics page shows the overall replication status for partitions in the topic, as well as counts for the partitions in the topic and the number of associated consumer groups. The overall storage used by the topic is also shown.
Internal topics must not be modified. You can choose to hide internal topics from the list of topics returned on the Topics page.
By clicking on a topic name, additional topic information is presented on a series of tabs:
- Messages
- Messages shows the message log for a topic.
- Partitions
- Partitions shows the replication status of each partition in a topic.
- Consumer groups
- Consumer groups lists the names and status of the consumer groups and group members connected to a topic.
- Configuration
- Configuration shows the configuration of a topic.
If a topic is shown as Managed, it means that is managed using the Streams for Apache Kafka Topic Operator and was not created directly in the Kafka cluster.
Use the information provided on the tabs to check and modify the configuration of your topics.
6.1. Checking topic messages
Track the flow of messages for a specific topic from the Messages tab. The Messages tab presents a chronological list of messages for a topic.
Procedure
- From the Streams for Apache Kafka Console, click the name of the Kafka cluster, then click Topics.
- Click the name of the topic you want to check.
Check the information on the Messages tab.
For each message, you can see its timestamp (in UTC), offset, key, and value.
By clicking on a message, you can see the full message details.
Click the Manage columns icon (represented as two columns) to choose the information to display.
Click the search dropdown and select the advanced search options to refine your search.
Choose to display the latest messages or messages from a specified time or offset. You can display messages for all partitions or a specified partition.
When you are done, you can click the CSV icon (represented as a CSV file) to download the information on the returned messages.
Refining your search
In this example, search terms, and message, retrieval, and partition options are combined:
-
messages=timestamp:2024-03-01T00:00:00Z retrieve=50 partition=1 Error on page load where=value
The filter searches for the text "Error on page load" in partition 1 as a message value, starting from March 1, 2024, and retrieves up to 50 messages.
- Search terms
Enter search terms as text (has the words) to find specific matches and define where in a message to look for the term. You can search anywhere in the message or narrow the search to the key, header, or value.
For example:
-
messages=latest retrieve=100 642-26-1594 where=key
This example searches the latest 100 messages on message key
642-26-1594.-
- Message options
Set the starting point for returning messages.
Latest to start from the latest message.
-
messages=latest
-
Timestamp to start from an exact time and date in ISO 8601 format.
-
messages=timestamp:2024-03-14T00:00:00Z
-
Offset to start from an offset in a partition. In some cases, you may want to specify an offset without a partition. However, the most common scenario is to search by offset within a specific partition.
-
messages=offset:5600253 partition=0
-
Unix Timestamp to start from a time and date in Unix format.
-
messages=epoch:1
-
- Retrieval options
Set a retrieval option.
Number of messages to return a specified number of messages.
-
messages=latest retrieve=50
-
Continuously to return the latest messages in real-time. Click the pause button (represented by two vertical lines) to pause the refresh. Unpause to continue the refresh.
-
retrieve=continuously
-
- Partition options
- Choose to run a search against all partitions or a specific partition.
6.2. Checking topic partitions
Check the partitions for a specific topic from the Partitions tab. The Partitions tab presents a list of partitions belonging to a topic.
Procedure
- From the Streams for Apache Kafka Console, click the name of the Kafka cluster, then click Topics.
- Click the name of the topic you want to check from the Topics page.
- Check the information on the Partitions tab.
For each partition, you can see its replication status, as well as information on designated partition leaders, replica brokers, and the amount of data stored by the partition.
You can view partitions by replication status:
- In-sync
-
All partitions in the topic are fully replicated. A partition is fully-replicated when its replicas (followers) are 'in-sync' with the designated partition leader. Replicas are 'in-sync' if they have fetched records up to the log end offset of the leader partition within an allowable lag time, as determined by
replica.lag.time.max.ms. - Under-replicated
- A partition is under-replicated if some of its replicas (followers) are not in-sync. An under-replicated status signals potential issues in data replication.
- Offline
- Some or all partitions in the topic are currently unavailable. This may be due to issues such as broker failures or network problems, which need investigating and addressing.
You can also check information on the broker designated as partition leader and the brokers that contain the replicas:
- Leader
- The leader handles all produce requests. Followers on other brokers replicate the leader’s data. A follower is considered in-sync if it catches up with the leader’s latest committed message.
- Preferred leader
- When creating a new topic, Kafka’s leader election algorithm assigns a leader from the list of replicas for each partition. The algorithm aims for a balanced spread of leadership assignments. A "Yes" value indicates the current leader is the preferred leader, suggesting a balanced leadership distribution. A "No" value may suggest imbalances in the leadership assignments, requiring further investigation. If the leadership assignments of partitions are not well-balanced, it can contribute to size discrepancies. A well-balanced Kafka cluster should distribute leadership roles across brokers evenly.
- Replicas
- Followers that replicate the leader’s data. Replicas provide fault tolerance and data availability.
Discrepancies in the distribution of data across brokers may indicate balancing issues in the Kafka cluster. If certain brokers are consistently handling larger amounts of data, it may indicate that partitions are not evenly distributed across the brokers. This could lead to uneven resource utilization and potentially impact the performance of those brokers.
6.3. Checking topic consumer groups
Check the consumer groups for a specific topic from the Consumer groups tab. The Consumer groups tab presents a list of consumer groups associated with a topic.
Procedure
- From the Streams for Apache Kafka Console, click the name of the Kafka cluster, then click Topics.
- Click the name of the topic you want to check from the Topics page.
- Check the information on the Consumer groups tab.
- To check consumer group members, click the consumer group name.
For each consumer group, you can see its status, the overall consumer lag across all partitions, and the number of members. For more information on checking consumer groups, see Chapter 8, Consumer groups page.
For each group member, you see the unique (consumer) client ID assigned to the consumer within the consumer group, overall consumer lag, and the number of assigned partitions. For more information on checking consumer group members, see Section 8.1, “Checking consumer group members”.
Monitoring consumer group behavior is essential for ensuring optimal distribution of messages between consumers.
6.4. Checking topic configuration
Check the configuration of a specific topic from the Configuration tab. The Configuration tab presents a list of configuration values for the topic.
Procedure
- From the Streams for Apache Kafka Console, click the name of the Kafka cluster, then click Topics.
- Click the name of the topic you want to check from the Topics page.
- Check the information on the Configuration tab.
You can filter for the properties you wish to check, including selecting by data source:
- DEFAULT_CONFIG properties have a predefined default value. This value is used when there are no user-defined values for those properties.
- STATIC_BROKER_CONFIG properties have predefined values that apply to the entire broker and, by extension, to all topics managed by that broker. This value is used when there are no user-defined values for those properties.
- DYNAMIC_TOPIC_CONFIG property values have been configured for a specific topic and override the default configuration values.
The Streams for Apache Kafka Topic Operator simplifies the process of creating managing Kafka topics using KafkaTopic resources.
