Chapter 8. Failing over to a disaster recovery cluster

When doing a planned failover or migration you can expect little to no data loss and a minimal impact on the applications. However, it requires certain conditions to be met before it can be done.

The first condition is to validate the gap (lag) between both clusters. MirrorMaker mirrors data asynchronously, so there is always a small data gap between the clusters. As described in the Metrics section, you can estimate the time delay between the clusters using the record-age-ms metric. For a planned failover, you ideally want this duration to be as short as possible. It should also match the RTO specified in the disaster recovery plan.

The second condition is the ability to shut down and restart applications in a controlled fashion, and with their configuration updated with the bootstrap servers for the other cluster. Before restarting, wait for the following to pass:

The default value for these intervals is the same (600 seconds). If you want to adjust the values, it’s best to keep them aligned to not complicate this process.

Once these conditions are met, you can start the process. In this example, failover is from cluster A to cluster B. In this initial state, cluster B is very close to cluster A, it is just missing a small amount of records as the mirroring process is asynchronous.

Figure 8.1. Mirroring is asynchronous, so there is always some data on the source cluster that has not been mirrored yet

Unfortunately, a failover is not always planned and if a hard failure impacts the primary cluster you may have to perform an unplanned failover. There is a large spectrum of possible failures, so the impact on your data and workload can vary significantly.

In the worst case, you can lose all data that has not been mirrored. The amount of data can be estimated using metrics. The record-age-ms metric indicates the gap in seconds between the two clusters. You can also use the last incoming byte rate from the source cluster to estimate the amount of data potentially lost.

The accuracy of the translated consumer group offsets is lower than in a planned failover because translation happens at a regular interval and the time of the last translation may be unknown.

Depending on the state of cluster A, attempting to recover data might be possible. However, plan for a full loss of the cluster. Do not rely on extracting data after a disaster.

Figure 8.5. Cluster A is impacted by a failure

In this scenario, the failover is straightforward, as you simply need to stop the two MirrorMaker connectors and then restart all your applications, now configured to connect to cluster B, which has become the new primary cluster.

Before restarting consuming applications, you can use the RemoteClusterUtils utility to translate consumer group offsets using the latest available checkpoints as this can produce slightly better offsets than what MirrorCheckpointConnector last committed. For example, you can run the following logic that translates consumer group offsets and commits them to the target cluster:

Map<String, Object> props = Map.of(
    CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "<TARGET_BOOTSTRAP_SERVERS>",
    MirrorClientConfig.REPLICATION_POLICY_CLASS, IdentityReplicationPolicy.class.getName()
);

// Get the translated offsets
Map<TopicPartition, OffsetAndMetadata> translatedOffsets =
    RemoteClusterUtils.translateOffsets(
        props, 

        "A", 

        "my-consumer-group", 

        Duration.ofSeconds(600)); 

Map<String, Object> adminProps = Map.of(
    CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "<TARGET_BOOTSTRAP_SERVERS>"
);
try (Admin admin = Admin.create(adminProps)) {
    // Use Admin API to override the offsets for the consumer group
    admin.alterConsumerGroupOffsets("my-consumer-group", translatedOffsets)
        .all()
        .get(10, TimeUnit.SECONDS);
}

The arguments are as follows:

Figure 8.6. The state of the disaster recovery environment after losing cluster A and performing an unplanned failover

Following a planned or unplanned failover, you don’t have a disaster recovery environment anymore, so you need to set up a new disaster recovery cluster.