Chapter 2. Debezium 2.5.4 release notes

If you prefer to retain the original behavior, customize the connector configuration by adding the following option to the connection string:

&readPreference=secondaryPreferred

If the connector reads from a sharded cluster, and the mongodb.connection.mode is set to replica_set, use the new mongodb.connection.string.shard.params property to specify a connection string that sets the read preference to secondary. For more information, see the documentation for the connector in the Debezium User Guide.

After you upgrade the connector, the new default setting invalidates existing offsets, and triggers a silent re-execution of the initial snapshot (DBZ-7272). To prevent this behavior, a check was added that fails the connector if the restart would trigger a snapshot.

If your connector reads from a sharded cluster, after you upgrade, review your connector configuration and adjust the connection mode as needed. You can either set the replica_set connection mode explicitly, or remove the existing offsets.

If you use a schema registry with a connector that is configured with the preceding settings, this change can lead to incompatibilities if the registry enforces strict compatibility rules. After you upgrade the connector, take action as needed to ensure that output values are accurately represented, with no loss of precision, for example, adjusting compatibility settings.

Unlike implementations from other vendors,the Debezium JDBC sink connector can ingest change events emitted by Debezium source connectors without the need to first process through an event flattening transoformation. The Debezium JDBC connector can reduce the processing footprint of your pipeline and simplify its configuration. The connector can also take advantage of Debezium source connector features, such as column type propagation. For more information, see the Debezium User Guide

Use of the MongoDB connector to perform incremental snapshots multi-replica a sharded clusters is now fully supported. For more information, see the Debezium User Guide.

In previous versions, when you used the Debezium connector for MongoDB in a sharded cluster deployment, the connector opened connections with each of the shard’s replica sets directly. This is not a recommended approach and instead MongoDB suggests that the connector open a connection with the mongos instance (the router) instead.

This release aligns with the recommended strategy. It might be necessary to adjust your configurations to point the connector to the mongos instance.

To retain backwards compatibility, a new configuration property mongodb.connection.mode is available. The default value for this property, replica_set preserves the current behavior so that the connector processes each shard as an individual replica set. Set the value to sharded to instruct the connector to internally create a single instance of ReplicaSet with an artificially named cluster. This configuration results in the connector using the initially supplied connection string to process connections to all members of ths shard.

For more information, see MongoDB sharded cluster support.

Incremental snapshots are a popular method for handling a variety of re-snapshot use cases. However, in some cases where snapshot read events are intertwined with streaming events (create, update, and delete), the use of incremental snapshots might not be indicated or even supported by a consuming application. In these cases, you can perform an ad hoc blocking snapshot.

Although you trigger both ad hoc blocking snapshots and ad hoc incremental snapshots by sending a signal to Debezium, when the connector processes the blocking snapshot signal, it places a hold on streaming for the duration of the snapshot process. As a result, the snapshot maintains separation between the snapshot read events and streaming create, update, and delete events. In other words, the blocking snapshot process is similar to the process followed by a traditional snapshot. As a result, the throughput is generally higher than with incremental snapshots.

The signal to initiate an ad hoc blocking snapshot is similar to the signal that you use to initiate an ad hoc incremental snapshot. The following example shows the payload of a signal to initiate a blocking snapshot a specific table with a condition. The only difference between this signal and the signal for an incremental snapshots is that the type of the signal is set to BLOCKING rather than INCREMENTAL.

{
  "type": "execute-snapshot",
  "data": {
    "data-collections": ["public.my_table"],
    "type": "BLOCKING",
    "additional-condition": "last_update_date >= '2023-01-01'"
  }
}

For more information, see Blocking snapshots in the Debezium User Guide chapter for your connector type.

Customers often request the ability to emit temporal columns in which the time zone is set to a value other than UTC. In past releases you could create a CustomConverter, or write your own single message transformation to specify how a connector emitted temporal columns.

In this release, Debezium provides a new time zone transformation that enables you to modify the timezone of temporal columns in events that the connector emits. You can convert columns from their default UTC timezone to any other time zone, based on the needs of your pipeline. To get started with this new transformation, add the following basic configuration to your connector:

{
  "transforms": "tz",
  "transforms.tz.type": "io.debezium.transforms.TimezoneConverter",
  "transforms.tz.converted.timezone": "America/New_York"
}

After you add the preceding configuration, when the connector emits an event, all temporal columns that were in UTC are converted to the` America/New_York` time zone.

Not only can you change the timezone for temporal fields, you can also modify timezone settings for other fields, by using the include.fields property as shown in the following example:

{
  "transforms": "tz",
  "transforms.tz.type": "io.debezium.transforms.TimezoneConverter",
  "transforms.tz.converted.timezone": "America/New_York",
  "transforms.tz.include.fields": "source:customers:created_at,customers:updated_at"
}

In the preceding example, the first value in include.fields converts the timezone of the created_at field for the source table with the name customers. The next value converts the updated_at field for the topic with the name customers.

Or, if you want to apply the conversion to all but subset of fields, you can set the exclude.fields property, as in the following example:

{
  "transforms": "tz",
  "transforms.tz.type": "io.debezium.transforms.TimezoneConverter",
  "transforms.tz.converted.timezone": "America/New_York",
  "transforms.tz.exclude.fields": "source:customers:updated_at"
}

In the preceding example, the transformation converts all temporal fields to the America/New_York timezone, except where the source table name is customers, and the field is updated_at. For more information, see Converting timezone values in Debezium event records in the Debezium User Guide.

You can now specify custom tags to append to the name of the connector MBean object name. Custom tags enable you to configure more reliable connector monitoring.

Debezium connectors expose metrics that provide data about the behavior of snapshots, streaming, and schema history processes. These metrics are available for each connector, and are exposed via the MBean name for the connector.

By default, when you deploy a correctly configured connector, Debezium generates a unique MBean for the connector. You can configure your observability stack to monitor the metrics associated with that MBean. If you later change the connector configuration, these changes can modify the original MBean name. If the MBean name changes, the linkage between the connector instance and the MBean breaks, and monitoring tools can no longer retrieve metrics data for the connector unless you reconfigure the observability stack to use the new MBean name.

To avoid the potential churn that results from MBean name changes, you can configure connectors to use custom metrics tags by adding the the custom.metric.tags property to the connector configuration. The custom.metric.tags property accepts key-value pairs. Each key represents a tag for the MBean object name, and the corresponding value represents the value of that tag. For example: k1=v1,k2=v2.

After you configure the custom.metric.tags property for a connector, configure the observability stack to retrieve metrics associated with the specified tags. The observability stack then uses the specified tags, rather than the mutable MBean names to uniquely identify connectors. Later, if Debezium redefines how it constructs MBean names, or if the topic.prefix in the connector configuration changes, metrics collection is uninterrupted, because the metrics scrape task uses the specified tag patterns to identify the connector.

A further benefit of using custom tags, is that you can use tags that reflect the architecture of your data pipeline, so that metrics are organized in a way that suits you operational needs. For example, you might specify tags with values that declare the type of connector activity, the application context, or the data source, for example, db1-streaming-for-application-abc. If you specify multiple key-value pairs, all of the specified pairs are appended to the connector’s MBean name.

The following example illustrates how tags modify the default MBean name.

debezium.oracle:type=connector-metrics,context=streaming,server=<topic.prefix>

debezium.oracle:type=connector-metrics,context=streaming,server=<topic.prefix>,database=salesdb-streaming,table=inventory

The Debezium MongoDB connector now supports large document processing for MongoDB 6. This change prevents the connector from returning a BSONObjectTooLarge exception when it attempts to capture a document that exceeds 16MB. To configure the connector to take advantage of this capability, set the oversize.handling.mode option.

The Debezium notification subsystem provides an easy way to integrate third-party tools and applications with Debezium to gain insight into the ongoing change data capture process, beyond the traditional JMX approach. In earlier release, the Debezium notification subsystem provided only basic information about initial snapshots, such as when the snapshot started, when each table started and concluded, and when the snapshot ended. The notification subsystem now includes the ability to notify you about the ongoing status of an initial snapshot.

Initial snapshot notifications are emitted with an aggregatetType of Initial Snapshot and contain a type field that exposes the current status of the snapshot. The possible values include: STARTED, ABORTED, PAUSED, RESUMED, IN_PROGRESS, TABLE_SCAN_COMPLETED, and COMPLETED.

DBZ-6878 extends the basic functionality to provide enhanced details about the snapshot. For example, the IN_PROGRESS notification now provides additional details about the tables being captured and which table is currently in-progress, as in the following example:

{
   "id":"6d82a3ec-ba86-4b36-9168-7423b0dd5c1d",
   "aggregate_type":"Initial Snapshot",
   "type":"IN_PROGRESS",
   "additional_data":{
      "connector_name":"my-connector",
      "data_collections":"table1, table2",
      "current_collection_in_progress":"table1"
   },
   "timestamp": "1695817046353"
}

This release introduces the property incremental.snapshot.watermarking.strategy, which permits you to specify the watermarking strategy to use during an incremental snapshot. Past releases, use the insert_insert approach, in which Debezium created two entries in the signaling data collection during the snapshot for each chunk. The first entry signaled the opening of the snapshot window, while the next entry marked its closure.

The new insert_delete option writes a single entry to the signaling data collection for each chunk at the beginning of the snapshot window. After the snapshot completes, the entry is removed, and no corresponding entry is added to signify the closure of the snapshot window. This method provides for more efficient management of the signaling data collection.

Because of the way that certain source databases function, when a Debezium connector emits a change event, the event might exclude values for column types that store large volumes of data. For example, values for TOAST columns in PostgreSQL, LOB columns in Oracle, or Extended String columns in Oracle Exadata, might all be excluded from a change event.

Debezium 2.5.4 introduces a Reselect columns post processor, which provides a mechanism for reselecting one or more columns that the connector captures from a database table, and fetching the current state of those columns. You can configure the post processor to re-select the following column types:

Configuring a PostProcessor is similar to configuring a custom converter or single message transformation, except that it works on the mutable payload’s Struct rather than on the SourceRecord. For more information, see using-the-reselect-columns-post-processor-to-add-source-fields-to-change-event-records in the Debezium User Guide.

When the Debezium connector for MongoDB emits a change event, you can configure the connector so that the event payload includes the full source document that was changed in an update. In past releases, the connector did not consistently stream full documents in the event payload.

To explicitly control how the connector looks up the full document in a change stream, set the configuration option, capture.mode.full.update.type. The default value for this option looks up the full document, which requires the database to perform a separate look-up to fetch the full document. If you use the connector with MongoDB 6 or greater, you can configure the connector to use post_image to rely on MongoDB change stream’s post-image support.

For more information, see capture.mode.full.update.type in the MongoDB connector chapter of the Debezium User Guide.

Beginning with Debezium 2.5.4, you can run the MySQL connector against a single MariaDB database deployment. Both MySQL and MariaDB can use Global Transaction Identifiers (GTIDs) for replication. GTIDs help to uniquely identify transactions within a cluster.

For the MySQL connector to work with MariaDB GTIDs, you must apply a supplemental configuration to the connector. For more information, see the Debezium User Guide

With PostgreSQL 16, you can now define replication slots on a stand-by instance. Based on this change to the database, in Debezium you can now connect to a standby PostgreSQL 16 server, and stream changes from it. By performing change data capture from a replica, rather than from the production system, you can better distribute server load, particularly in a very active database.

The CloudEvwents converter emits change event records that conform to the CloudEvents specification. The CloudEvents change event envelope can be JSON or Avro and each envelope type supports JSON or Avro as the data format. For more information, see CloudEvents converter.

In cases where the default data type conversions do not meet your needs, you can create custom converters to use with a connector. For more information, see Custom-developed converters.

For more information, see Oracle binary character LOB types in the Debezium User Guide.

The ability to use multiple parallel threads to perform an initial snapshot is available as a Technology Preview feature for all of the Debezium SQL connectors, except MySQL. Parallel initial snapshots for the Debezium MySQL connector are available as a Developer Preview feature.

To configure connectors to use parallel initial snapshots, set the snapshot.max.threads property in the connector configuration to a value greater than 1.

For more information, see snapshot.max.threads in the Debezium User Guide documentation for your connector.

To improve snapshot performance the MySQL connector, implements parallelization to concurrently snapshot change events and geneate schema events for tables

This reduces snapshot time when capturing the schema for many tables in your database.

After you configure this property, the connector uses the name pattern that is based on these key-value pairs when it gathers metrics. This establishes a stable name so that the connector always collects the same set of metrics, regardless of whether you modify the topic.prefix, or if the MBean name changes. customize the MBean object name for each connector instance. By adding these custom metric tags, you can ensure that each connector has a unique MBean name, preventing conflicts between multiple instances.

The Debezium initial snapshot for MySQL has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

In this release, you can configure a MySQL connector to use multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, add the snapshot.max.threads property to the connector configuration, and set the property to a value greater than 1.

snapshot.max.threads=4

In the preceding example, if the connector needs to snapshot more than 4 tables, a maximum of 4 tables can be snapshot in parallel. When one thread finishes processing a table, it will find the next table in the queue and begin to perform a snapshot. The process continues until the connector finishes performing snapshots on all of the designated tables.

For more information, see snapshot.max.threads in the Debezium User Guide.

When the Debezium connector for Oracle connects to a production or primary database, it uses an internal flush table to manage the flush cycles of the Oracle Log Writer Buffer (LGWR) process. The flush process requires that the user account through which the connector accesses the database has permission to create and write to this flush table. However, logical stand-by databases often have restrictive data manipulation rules, and may even be read-only. As a result, writing to the database might not be feasible.

To support an Oracle read-only logical stand-by database, Debezium introduces a property to disable the creation and management of the flush table. You can use this feature with both Oracle Standalone and Oracle RAC installations.

To enable the Oracle connector to use a read-only logical stand-by, add the following connector option:

internal.log.mining.read.only=true

For more information, see the Oracle connector documentation in the Debezium User Guide.

In this release, Debezium provides support for exactly-once semantics for the PostgreSQL connector. Exactly-once delivery for PostgreSQL applies only to the streaming phase; exactly-once delivery does not apply to snapshots.

Debezium was designed to provide at-least-once delivery with a goal of ensuring that connectors capture all change events that occur in the monitored sources. In KIP-618 the Apache Kafka community proposes a solution to address problems that occur when a producer retries a message. Source connectors sometimes resend batches of events to the Kafka broker, even after the broker previously committed the batch. This situation can result in duplicate events being sent to consumers (sink connectors), which can cause problems for consumers that do not handle duplicates easily.

No connector configuration changes are required to enable exactly-once delivery. However, exactly-once delivery must be configured in your Kafka Connect worker configuration. For information about setting the required Kafka Connect configuration properties, see KIP-618.