Chapter 1. Debezium 2.7.3 release notes

Under certain circumstances, such as database communication errors, JDBC queries can become stuck in a persistent hung state. To mitigate this problem, Debezium now provides a query.timeout.ms property that enables you configure how long the connector waits for query responses. To disable timeout handling, you can set the value to 0. The default value is 600000 ms (600 seconds).

The Debezium TimezoneConverter SMT enables you to convert outgoing payload time values to a specified time zone. The original implementation was specifically restricted to allow conversion of values within the before or after parts of the payload. You can now use the converter to convert other time-based fields in the metadata, such as ts_ms in the source information block.
This change helps to improve the calculation of lag metrics when the timezone setting of the JVM that runs the connector does not match the setting on the source database server, resulting in errors in the difference between the envelope ts_ms and source ts_ms values. By configuring the connector to use the TimezoneConverter to convert metadata fields to use a consistent timezone, you can more easily calculate the accurate lag time between those two fields.

Except when you run the MySQL connector in read-only mode, running an incremental snapshot requires a signal table to write open and close markers to coordinate the change boundaries with the data recorded in the transaction logs. In some cases, having a record of when the snapshot window opened and when it closed can be helpful.
Debezium now provides these details about these signal markers in the data column of the signal table, as shown in the following examples:

{"openWindowTimestamp": "<window-open-time>"}

{"openWindowTimestamp": "<window-open-time>"}

{"openWindowTimestamp": "<window-open-time>", "closeWindowTimestamp": "<window-close-time>"}

{"openWindowTimestamp": "<window-open-time>", "closeWindowTimestamp": "<window-close-time>"}

By default, the CloudEvents converter automatically generate the schema for a CloudEvent message. However, if the automatically generated schema names are not sufficient, you can adjust the configuration by specifying a dataSchemaName in the metadata.source property. You can set the dataSchemaName to one of the following values:

In some cases, when the JDBC sink connector consumed an event that included fields with null values, when writing the event record to the target database, it incorrectly wrote the default field value rather, than the the value NULL.

The deprecated replica_set mode for the MongoDB connector is now removed. The replica_set connection mode for the Debezium enabled the MongoDB connector to use separate Kafka Connect tasks to process each database shard individually, within the confines of the tasks.max configuration. This configuration circumvented the MongoDB router (mongos). However, the MongoDB database is designed with the expectation that clients always connect through the router. As a result, the replica_set connection mode could causes problems with signaling, incremental snapshot support, and overall complexity of our codebase.

Removing the replica_set connection mode restricts the connector to single task operation. If you previously configured the MongoDB connector to use replica_set mode, to continue using the connector after the upgrade to Debezium 2.7.3, you must adjust the configuration. For more information, see the MongoDB connector documentation.

Support for Oracle 12 was deprecated in a previous release. Beginning with Debezium 2.7.3, you can no longer use Debezium with an Oracle 12 database.

Oracle Database 12c Release 1 (12.1) reached its end of life on July 31, 2022. After this date, Oracle no longer provides bug fixes, security updates, or patches for this version. Debezium 2.1.4 was the last release that advertised support for Oracle 12c.

This has been fixed and the connector now correctly emits these columns according to the configured decimal handling mode. After you upgrade the connector, this change in behavior might cause problems in deployments that use strict schema registry compatibility rules.

The SQL Server connector was not capturing all schemas when the connector was first deployed, and instead, was only capturing the schemas based on the tables defined in the configuration’s include list. This was a bug that could prevent users from easily adding new tables to the connector when expecting that the new table’s schema would already exist in the schema history topic. The connector now correctly honors the store.only.captured.tables.ddl configuration option (DBZ-7593).

For existing connector deployments, if you do not specifically set the store.only.captured.tables.ddl property for the schema history topic, the connector will begin capturing schema changes for all relevant tables in your database. If you want to prevent this and retain the prior behavior, you will need to adjust your connector configuration by adding schema.history.internal.store.only.captured.tables.ddl with a value of true.

In older versions of Debezium, the SQL Server connector would process all transactions available during a poll iteration. When there was a high volume of traffic, memory overloads could result, leading to poor performance. Although the max.iteration.transactions configuration property already exists to address this problem, it was previously configured with a default value of 0. Based on the former default value, the connector would still attempt to process all transactions. To limit the number of transactions that the connector processes during a poll iteration, in this release, the default value of the property is now set to 500.

Normal JMX metrics are registered with a taskId attribute, because SQL Server supports spawning a unique task per database mapping. In past releases, the JMX signal channel did not honor this, which could prevent the connector from starting a channel for each task. This has been fixed, and the MBean name of the JMX signal channel now includes a taskId in its name. By including the taskId in the MBean name, it’s now possible to uniquely identify the signal channel for each database task when you deploy a single connector to stream changes from multiple SQL Server databases.

In earlier releases, a Technology Preview feature provided the option to test the use of the Debezium connector for Oracle Database in an Oracle RAC environment. Use of the connector in an Oracle RAC environment is now fully supported.

When you use the Debezium connector with a PostgreSQL 16 database, you can now define replication slots on stand-by instances of the database. Based on this change to the database, in Debezium you can now connect to a standby PostgreSQL 16 server, and stream changes from it. By performing change data capture from a replica, rather than from the production system, you can better distribute server load, particularly in a very active database.

To improve the precision of timestamps in change events, change events now include four additional fields, two in the envelope, and two in the source information block, as shown in the following example:

{
  "source": {
    ...,
    "ts_us": "1559033904863123",
    "ts_ns": "1559033904863123000"
  },
  "ts_us": "1580390884335451",
  "ts_ns": "1580390884335451325",
}

{
  "source": {
    ...,
    "ts_us": "1559033904863123",
    "ts_ns": "1559033904863123000"
  },
  "ts_us": "1580390884335451",
  "ts_ns": "1580390884335451325",
}

The envelope values provide microsecond (ts_us) and nanosecond (ts_ns) values. The source information block can include both microsecond and nanosecond precision values. If the source database does not provide a sufficient level of precision, the timestamp values are truncated to a lower precision.

In earlier versions of Debezium, the column.truncate.* functionality returned a sliced ByteBuffer based on the truncation configuration. Although this worked when using Avro, the truncation was not honored if your connector configuration used the JsonConverter as it operated on the entire underlying array rather than the given slice.

The column truncation logic now explicitly creates a ByteBuffer based on a new array. This change allows the JsonConverter to respect the truncated column value during the serialization to Kafka.

The column.truncate.to.length.chars configuration property was improved to support array field types along with the string types that it previously supported. For more information, see the documentation for the SQL-based source connectors in the Debezium User Guide.

The snapshot process is an integral part of each connector’s lifecycle, and enables you to collect and send all of the historical data that is present in a data store to a specified target system. In earlier releases, Debezium connectors did not support a consistent set of snapshot modes. This lack of consistency was particularly confusing for users who work with multiple connector types.

In this release, Debezium implements a consistent set of snapshot modes for all connectors. The same snapshot modes are available to all connectors, with the exception of the never mode, which remains specific to the MySQL connector. As a result, certain snapshot modes, such as when_needed, that were previously not available for some connectors, can now be used with any connector.

By default, after a snapshot completes, Debezium immediately transitions to the streaming phase. You now add the streaming.delay.ms property to the connector configuration to specify a delay between the end of the snapshot and the start of the streaming phase. This property can help you to prevent unnecessary re-snapshotting, and improve the overall stability of the connector during the transition from snapshot to streaming.

Use this property, in combination with the offset.flush.interval.ms property to ensure that offsets are properly flushed before streaming begins. To permit at least one offset flush interval to occur before streaming begins, set the value of streaming.delay.ms to a value that is slightly more than the offset.flush.interval.ms.

In some pipelines, event ordering is critical for consuming applications. Certain operations, such as re-partitioninf a Kafka topic, can disrupt the normal ordering of events in a data pipeline. Errors might occur when an application attempts to reconstruct the order of events.

In this release, when you enable transaction metadata, metadata events encode their transaction order. The encoded ordering field provides consumers with the information that they require to correctly reconstruct the transaction timeline.

The JDBC sink connector supports the use of mapping source columns to Kafka ARRAY-based payload field types. With Debezium 2.7.3, you can now serialize ARRAY-based fields to a target PostgreSQL database, with no change in configuration.

The incremental snapshot process is an instrumental part in various recovery situations to collect whole or part of the data set from a source table or collection. Relational connectors have long supported the idea of supplying an additional-conditions value on the incremental snapshot signal to restrict the data set, providing for targeted resynchronization of specific rows of data. We’re happy to announce that this is now possible with MongoDB (DBZ-7138). Unlike relational databases, the additional-conditions should be supplied in JSON format. It will be applied to the specified collection using the find operation to obtain the subset list of documents that are to be incrementally snapshotted.

Beginning with this release, the Debezium MongoDB ExtractNewDocumentState single message transformation now adds an _id attribute to the event payload of delete events. The addition of this attribute enhances the ability consuming applications to make greater use of the events that the SMT generates.

See Section 1.4.2.1.1, “Support for Oracle Real Application Clusters (RAC)”.

Debezium treats a RAW column type as a series of bytes, and therefore, change events that contain RAW columns use a schema type of BYTES. In the absence of information about how a consuming application uses the data in a RAW columns, this default makes sense. However, to support applications that are designed to consume data emitted as STRING types, rather than as BYTES, Debezium now provides a RawToStringConverter, which automatically emits RAW columns as STRING based.

For information about how to configure the converter, see the Debezium User Guide.

While ROW_ID is not unique across all rows of a table for the table’s lifespan, it can be used in certain situations when the lifecycle of the table and rows are managed in a very strict way. At the community’s request, we’ve added a new row_id field to the Oracle connector’s change event source information block. This field is be populated with the ROW_ID value under the following conditions:

Any event that does not match the criteria will not include a row_id field as its marked as optional.

In earlier versions of Debezium, the Oracle connector was designed to create the LogMiner flush table in the default tablespace of the connector user account. This could cause problems if the DBA wanted to move the flush table to a different tablespace. Previously, to add the flush table to the preferred tablespace required you to complete several steps. First, you would modify the user account to specify a new default table space, or create a different account that specified that table space. Then, you would have to re-create the table so that it was placed in the correct location.

Beginning with this release, you can use the log.mining.flush.table.name property to specify where to create the flush table, as shown in the following example:

log.mining.flush.table.name=THE_OTHER_SCHEMA.LOG_MINING_FLUSH_TABLE

log.mining.flush.table.name=THE_OTHER_SCHEMA.LOG_MINING_FLUSH_TABLE

The schema name is optional. If you do not specify a schema name, the connector follows the legacy behavior and uses the default tablespace of the connector user account to create the flush table and check for its existence.

The Debezium Oracle connector can support thousands of tables in a single connector deployment. However, you might want to customize the query filter using the IN mode. When a high volume of changes occur in the tables of a configured database, you might want to filter that dataset at the database level before changes are passed to Debezium for processing.

In earlier versions, a SQL error would result if the connector’s log.mining.query.filter.mode was set to in, and the table include list contained more than 1000 elements. Oracle does not permit more than 1000 elements within an IN clause. To address this limitation, Debezium 2.7.3 uses a disjunction between multiple buckets of 1000 item IN clause lists.

The Debezium Oracle connector can now report the original SQL for an operation in the source information block of INSERT, UPDATE, and DELETE event records. To configure the connector to include the REDO SQL as part of the change event, set the log.mining.include.redo.sql property as shown in the following example:

"log.mining.include.redo.sql": "true"

"log.mining.include.redo.sql": "true"

This feature is an opt-in only feature; to use it, you must explicitly enable it.

When this option is enabled, the source information block contains a new field redo_sql, as shown in the following example:

"source": {
  ...
  "redo_sql": "INSERT INTO \"DEBEZIUM\".\"TEST\" (\"ID\",\"DATA\") values ('1', 'Test');"
}

"source": {
  ...
  "redo_sql": "INSERT INTO \"DEBEZIUM\".\"TEST\" (\"ID\",\"DATA\") values ('1', 'Test');"
}

A new delay strategy for transaction registration has been added when using LogMiner. This strategy effectively delays the creation of the transaction record in the buffer until after the connector captures the first change for that transaction.

Delaying transaction registration has a number of benefits, which include:

The Debezium SQL Server connector uses a common SQL Server stored procedure called fn_cdc_get_all_changes…​ to fetch all of the relevant captured changes for a given table. This query performs several unions and then returns data from only one of the union sub-queries, which can be inefficient.

This release of the connector introduces the configuration property data.query.mode, which specifies the method that the connector uses to gather details about table changes. The default behavior remains unchanged from earlier releases, and is set by using the value function to delegate to the previously mentioned stored procedure.

To enhance the efficiency with which the connector gathers changes, configure the connector to build the query internally by setting the value of data.query.mode to direct.

Debezium 2.7.3 adds support for the heartbeat.action.query connector configuration property to the SQL Server connector. The heartbeat.action.query property enables the connector to perform a write operation to the source database on an interval defined by heartbeat.interval.ms. The write operation is meant to produce a change event that is captured by the connector, and is then sent to Kafka or the target system.

In an active database that is capturing changes regularly, the constant stream of changes is sufficient to keep offsets synchronized with the read position in the transaction logs. However, if the connector is capturing changes from a source more changes occur in non-captured tables than in captured tables, setting heartbeat.action.query can help to keep the read position in the offsets synchronized with the lower capture activity.

In previous iterations of the Debezium MongoDB connector, change streams could be opened against the deployment and database scopes, which was not always ideal for restrictive permission environments. Debezium 2.7.3 introduces a new change stream mode in which the connector operates within the scope of a single collection only, allowing for such granular permissive configurations.

To limit the capture scope of a MongoDB connector to a specific collection, set the value of the capture.scope property in the connector configuration to collection. Use this setting when you intend for a connector to capture changes only from a single MongoDB collection.

Certain limitations exist when using this feature. If you set value of capture.scope to collection, the connector cannot use the default source signaling channel. Enabling the source channel for a connector is required to permit processing of incremental snapshot signals, including signals sent via the Kafka, JMX, or File channels. Thus, if the connector configuration sets the collection option for the capture-scope property, the connector cannot perform incremental snapshots.

To improve snapshot performance the MySQL connector, implements parallelization to concurrently snapshot change events and generate schema events for tables. By running snapshots and generating schema events in parallel, the connector reduces the time required to capture the schema for many tables in your database.

The Debezium initial snapshot for MySQL has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

In this release, you can configure a MySQL connector to use multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, add the snapshot.max.threads property to the connector configuration, and set the property to a value greater than 1.

snapshot.max.threads=4

snapshot.max.threads=4

Based on the configuration in the preceding example, the connector can snapshot a maximum of four tables at once. If the number of tables to snapshot is greater than four, after a thread finishes processing one of the first four tables, it then finds the next table in the queue and begins to perform a snapshot. The process continues until the connector finishes performing snapshots on all of the designated tables.

For more information, see snapshot.max.threads in the Debezium User Guide.

When the Debezium connector for Oracle connects to a production or primary database, it uses an internal flush table to manage the flush cycles of the Oracle Log Writer Buffer (LGWR) process. The flush process requires that the user account through which the connector accesses the database has permission to create and write to this flush table. However, logical stand-by databases often have restrictive data manipulation rules, and may even be read-only. As a result, writing to the database might not be feasible.

To support an Oracle read-only logical stand-by database, Debezium introduces a property to disable the creation and management of the flush table. You can use this feature with both Oracle Standalone and Oracle RAC installations.

To enable the Oracle connector to use a read-only logical stand-by, add the following connector option:

internal.log.mining.read.only=true

internal.log.mining.read.only=true

For more information, see the Oracle connector documentation in the Debezium User Guide.

You can now configure the Debezium connector for Oracle to use the LogMiner hybrid mining strategy. This strategy is designed to support all of the schema evolution features of the default mining strategy while also taking advantage of the performance optimizations offered by the online catalog strategy. To configure the connector to use this strategy, set the log.mining.strategy configuration property to the value hybrid.

With the online_catalog strategy, a schema change and a data change can occur within the same mining step. When this happens, LogMiner cannot reconstruct the SQL correctly, and the resulting table is assigned a name such as OBJ# xxxxxx; similarly, columns are represented as COL1, COL2, and so on. To avoid this situation when the online catalog strategy is used, requires you to carefully choreograph schema changes so that a mining step never observes both a schema change and a data change together; however, this is not always feasible.

By contrast, the hybrid strategy tracks a table’s object ID at the database level, and then uses this ID to look up the schema associated with the table from Debezium relational table model. Debezium is thus able to do what Oracle LogMiner is unable to do in these specific corner cases. The table name is taken from the relational model’s table name, and columns are mapped by column position.

Unfortunately, Oracle does not provide a way to reconstruct failed SQL operations for CLOB, BLOB, and XML data types. As a result, you cannot configure the connector to use the hybrid strategy if the value of lob.enabled is set to true. If you attempt to start a connector that is configured to use the hybrid strategy and has lob.enabled set to true, the connector fails to start and report a configuration failure.

In this release, Debezium provides support for exactly-once semantics for the PostgreSQL connector. Exactly-once delivery for PostgreSQL applies only to the streaming phase; exactly-once delivery does not apply to snapshots.

Debezium was designed to provide at-least-once delivery with a goal of ensuring that connectors capture all change events that occur in the monitored sources. In KIP-618 the Apache Kafka community proposes a solution to address problems that occur when a producer retries a message. Source connectors sometimes resend batches of events to the Kafka broker, even after the broker previously committed the batch. This situation can result in duplicate events being sent to consumers (sink connectors), which can cause problems for consumers that do not handle duplicates easily.

No connector configuration changes are required to enable exactly-once delivery. However, exactly-once delivery must be configured in your Kafka Connect worker configuration. For information about setting the required Kafka Connect configuration properties, see KIP-618.