Chapter 4. Translating offsets between clusters

As mentioned, records on the source and target clusters have different offsets. To enable applications to move from one cluster to another, MirrorMaker provides offset translation. The offset translation process is lossy. When translating an offset from the source cluster to the target cluster, it’s not guaranteed to get the exact same record. It’s just not possible for MirrorMaker to keep track of the source and target offsets for all records it mirrors. If an exact translation is not possible, MirrorMaker guarantees that the record at the translated offset is always earlier than the actual record. This ensures consuming applications never skip data at the cost of potential reprocessing.

The translation mechanism builds on two processes that run in parallel. First MirrorSourceConnector tasks emit periodic offset-syncs records. Then MirrorCheckpointConnector tasks read the offset-syncs topic and build an in-memory cache of the mappings for each partition.

One key impact is that committed offsets more recent than the latest mirrored offsets can’t be translated. For example with the following state, the last offset-sync emitted by the MirrorSourceConnector is offset 7. If a consumer commits offset 9, the MirrorCheckpointConnector is not able to translate that offset, and returns -1.

Figure 4.1. The difference in offsets between the two clusters

When starting, each MirrorCheckpointConnector task consumes the whole offset-syncs topic. For this reason it’s important to keep this topic relatively small by setting retention policies. For each partition each MirrorCheckpointConnector task keeps up to 64 offset-syncs. It always keeps the latest offset-sync, and then keeps mappings separated by approximately exponential space. For this reason, translating offsets of records that have recently been mirrored is accurate but the accuracy of the translation degrades exponentially for older offsets.

At a high level, each MirrorCheckpointConnector task builds a cache of offset mappings for a partition. For example after mirroring 1000000 records for a partition, the cache may look something like this:

When we try to translate an offset for which we have the mapping the exact matching offset is returned. For example, offsets 1000000, 999900 translate exactly. For older offsets, the target offset + 1 for the previous matching offset is returned. For example 999250 is translated to E + 1. This means the number of records consumers may reprocess is in the worst case 400 (999600 - 999200) records. As the gap between offset-syncs is doubling, the quality of the translation degrades exponentially. So offset 420000 is translated to N + 1 which can be up to 204800 (614400 - 409600) records off. Finally, offsets more recent than 1000000 can’t be translated.