Chapter 3. MirrorMaker components

The MirrorSourceConnector mirrors the topics and their records. It periodically checks topics on the source cluster and applies any partition and configuration changes it detects to the target cluster. However topic deletions are not mirrored, so if a topic is deleted on the source cluster, its copy on the target cluster is left untouched. It provides a safety net in case the deletion was a mistake. Otherwise, an operator should go and delete that topic in the target cluster.

An important concept to understand is that records on each cluster are highly likely to have different offsets. This happens for several reasons. First, partitions always start at offset 0. So if a partition on the primary cluster has already had offsets deleted due to retention policies, there is always a difference in offsets. Another reason is that MirrorMaker does not mirror transaction markers. Transaction markers are emitted when a transaction is committed or aborted and they occupy offsets. In order to enable offset translation between both clusters, the connector periodically writes a record into the offset-syncs topic that contains both the source and target offsets of a record it just mirrored.

This connector also mirrors topic ACLs, but note that it does not mirror the ALLOW WRITE topic ACLs. This avoids clients (other than MirrorMaker) writing to the target cluster, which might cause inconsistency. It also does not mirror ACLs set on other resources such as groups, transactional IDs, users, or cluster. In case ACLs are used, you need another process to maintain them. For these reasons, if you require authorizations in your clusters, you can’t only rely on MirrorMaker to mirror them. Operational practices like GitOps are a great way to set and maintain ACLs.

The MirrorCheckpointConnector mirrors committed consumer group offsets. It uses the mappings from the offset-syncs topic to translate the committed offsets from the source cluster, and writes the results to the checkpoints topic. By enabling the sync offsets feature, the connector is also able to directly commit the translated offset to the target cluster. Committing the offsets makes it easier to move consuming applications across the clusters, otherwise applications must use the RemoteClusterUtils tool to find their offsets from the checkpoints topic when connecting to the new primary cluster.

The MirrorHeartbeatConnector is not required, and it is recommended that you do not run it for disaster recovery scenarios. This is because the MirrorHeartbeatConnector sends records to the source cluster whereas the other two connectors send records to the target cluster. Since we are deploying MirrorMaker near the target cluster for better performance with the data replication, the MirrorHeartbeatConnector would need to be deployed separately near the source cluster. This added complexity and cost do not justify the benefit, as sufficient metrics to monitor MirrorMaker can be obtained from the MirrorSourceConnector.