Chapter 1. Debezium 3.0.8 release notes

The Java requirements for running Debezium have changed. Beginning with this release, all Debezium connectors require a runtime baseline of Java 17.

If you use Java 11 with new connectors, Kafka Connect silently fails to find the connector. The connector does not report any bytecode errors

To run Debezium Server, a runtime baseline of Java 21 is required.

In earlier releases, after a connector restart, Debezium automatically reprocessed Kafka-based signals, which could lead to unintended side effects. Beginning with this release, if a connector stops, and you want to process a Kafka-based signal, you must resubmit the signal.

The deprecated additional-condition field for incremental snapshot signals has been removed and is no longer supported. This field is replaced with the new additional-conditions property, which enables you to specify conditions by table. If have scripts or workflows that refer to the old field, update them to use the new property.

Earlier versions of the MariaDB and MysSQL documentation provided an incorrect default value for the schema.history.internal.store.only.captured.databases.ddl property. The correct default value is false.

Although this change does not affect the code, if you previously set this property, review your current configuration to verify that it contans the value that is best suited to your environment. For more information, see the Debezium User Guide.

The default mining strategy for the Oracle connector is now online_catalog. If you previously relied on the default redo_log_catalog strategy, and you want to continue using that strategy, you must explicitly specify it in the connector configuration.

For more information, see: Section 1.4.2.3.1, “New default mining strategy (DBZ-3656)”.

The following Oracle connector configuration properties that were deprecated in the previous release are now removed in Debezium 3.0.8:

The ReselectColumnsPostProcessor behavior has changed. In this release, the post processor now reselects Oracle LOB columns regardless of the value of the lob.enabled configuration property. With this change, you can now use the column reselection post processer to populate LOB columns without mining those columns during streaming.

The standalone connector implementation for MariaDB is now fully supported.

The Oracle connector now offers full support for processing BLOB, CLOB, and NCLOB data types. For more information, see Oracle binary and character LOB types in the Debezium User Guide.

The ability to use multiple parallel threads to perform an initial snapshot is now available for general availability for all of the Debezium SQL connectors, except for MySQL and MariaDB. Parallel initial snapshots for the Debezium MariaDB and MySQL connectors is available as a Developer Preview feature.

To configure connectors to use parallel initial snapshots, set the snapshot.max.threads property in the connector configuration to a value greater than 1.

For more information, see snapshot.max.threads in the Debezium User Guide documentation for your connector.

The JDBC sink connector uses a set of buffers to improve the throughput of writes to the target database. In some cases, the flush operation of these buffers faces specific exceptions when other applications lock a specific row or table. To improve the user experience, two configuration properties are now available for the connector:

The retry feature is enabled by default. The connector attempts makes a maximum of 5 attempts to retry flush operations, with a 1-second delay between retry attempts. If you do not want the connector to retry flush operations, set flush.max.retries to 0 to disable this feature.

In Debezium 3.0.8, the default value for the Oracle connector log.mining.strategy property has changed from redo_log_catalog to the generally more performant online_catalog option. If you previously used the default redo_log_catalog strategy, and you want to retain that setting, after you upgrade to the current version, you must explicitly set the value of the log.mining.strategy property to redo_log_catalog.

In an earlier release, the Oracle RAC node flush strategy was changed to force a 3-second delay whenever a database administrator takes an Oracle RAC node offline. During this pause, the connector attempts to flush the log writer (LGWR) process. However, the pause would be implemented even in situations in which the LGWR buffers are not flushed. Because RAC nodes cannot write to the redo logs when they are offline, this delay introduced unnecessary latency.

In Debezium 3.0.8, to improve stability, the 3-second delay is now imposed only for connections that are active to an Oracle RAC node. As a result, after an administrator takes a RAC node offline, the connector no longer imposes any latency.

In Oracle, to ensure that LOB columns are non-NULL, you can use the EMPTY_CLOB() or EMPTY_BLOB() functions to specify default values for CLOB or BLOB fields in a table. Earlier Debezium releases did not evaluate these special functions, which resulted in the connector marking these columns as optional, because they could contain null values.

Beginning in Debezium 3.0.8, when you use the EMPTY_BLOB() or EMPTY_CLOB() functions to define the default value for an Oracle BLOB column, the connector now emits the field as not optional. The field in the resulting change event contains the appropriate default value, either an empty byte array, or an empty string.

Through the addition of a new connector configuration property, snapshot.isolation.mode, this feature addresses a longstanding request to provide snapshot isolation support for PostgreSQL. You can now specify how to balance the need for consistency and performance when executing initial and ad-hoc blocking snapshots, by specifying one of four snapshot isolation levels:

These isolation levels specify how consistent the snapshot is and what type of locking to apply to the source data, if any, when the connector reads it during a snapshot.

For example, you can set snapshot.isolation.mode to serializable to enforce strict isolation and locking, preventing administrators and users from performing such operations as creating a table index, while a snapshot is running. Or you can apply the least restrictive read_uncommitted setting, which permits snapshots to read data from user transactions that are not yet committed. Permitting such "dirty reads" as they are known, can lead to inconsistent data, but at the same time, because data is not locked, the snapshot does not interfere with database performance.

For more information, see snapshot.isolation.mode in the PostgreSQL connector documentation.

PostgreSQL 15 introduced the pgvector extension, which provides the following data types:

Beginning in Debezium 3.0.8, the PostgreSQL connector supports streaming of events that use these pgvector data types. When the connector emits a change event record for an operation that involves one of these data types, it converts each vector type in the source to a semantic type, according to the mappings in following list:

After you enable the pgvector extension in the database, no further configuration is required for Debezium to convert vector values. For more information, see pgvector types in the PostgreSQL connector documentation.

Transparent Data Encryption (TDE) is available in SQL Server 2017 Enterprise Edition and in both the Standard Edition and Enterprise Edition of SQL Server 2019 and 2022. As its name suggests, TDE operates automatically and transparently to encrypt data at rest in the database and transaction log. If TDE is enabled for the database, data cannot be read by unauthorized users. At the same time, access by authorized users is not affected, including access by a properly configured Debezium SQL Server connector. No special configuration is required to enable Debezium to stream changes from a SQL Server database that has TDE enabled. That is, if a connector can connect to a database that is not enabled for TDE, and the database later has TDE enabled, the original connector configuration still works.

It’s now possible to uniquely identify the signal channel for each database task when you deploy a single connector to stream changes from multiple SQL Server databases.

The format of the MBean names that the Debezium SQL Server connector uses in signaling and notification operations was changed to ensure uniqueness per task. Normal JMX metrics are registered with a taskId attribute, because SQL Server supports spawning a unique task per database mapping. In past releases, the JMX signal channel did not honor this, which could prevent the connector from starting a channel for each task. Adding the taskId to the MBean name enables the connector to use multiple tasks when these operations span databases. For more information, see Debezium SQL Server connector schema history metrics in the Debezium User Guide.

Documentation for the Debezium SQL Server connector now correctly describes the format of schema history MBean names. The updated description specifies that names now include the same task attribute that is present in the snapshot and streaming metrics MBean names.

PostgreSQL is unique in that you can implement the Outbox pattern without creating an outbox table. Instead, the database can use the pg_logical_emit_message function to write logical messages directly to the write-ahead log (WAL). However, when Debezium captures these events from the WAL, the data that it sends to Kafka is formatted as a series of bytes. Consuming applications that require structured messages might not be able to consume data in this format.

The new Logical message decoder SMT (DecodeLogicalDecodingMessageContent) is a PostgreSQL-specific single message transformation. The transformation is designed to convert pg_logical_emit_message event bytes into a structured event format so that consuming applications can process them. In this way you can implement the Outbox pattern without creating a physical outbox table. For more information, see the Decoding binary content in PostgreSQL logical decoding messages in the Debezium User Guide.

To store data types whose sizes exceed the standard capacity of a column, some databases use specialized mechanisms to store the oversized data in an external location, outside of the main table. For example PostgreSQL uses The Oversized-Attribute Storage Technique (TOAST), and Oracle uses Large Object (LOB) storage, or Oracle Exadata Extended Strings.

By default, when Debezium emits an event record in response to a change in a table row that includes oversized data fields, the resulting event record only includes the values of oversized data fields if they are changed by the SQL operation. The connector populates these values only if the table operation modifies the value of the oversized data column. If the value of the oversized data column remains unchanged, the connector inserts a placeholder for the field. This placeholder incorrectly implies that the field does not contain a value.

The ReselectColumnsPostProcessor was introduced to populate the values of oversized data fields in change event records. This post processor now supports reselection of int and bigint array data types, so that these fields are always populated, even when they’re values are unchanged following a SQL operation.

This release introduces a MongoDB sink connector implementation. The Debezium MongoDB sink connector differs from other vendor implementations in that it can ingest raw change events emitted by Debezium connectors without first applying an event flattening transformation. The MongoDB sink connector can take advantage of native Debezium source connector features, such as column type propagation, enabling you to potentially reduce the processing footprint of your data pipeline, and simplify its configuration.

Unlike the JDBC sink relational connector that requires an additional plug-in to be installed to use it, the MongoDB sink connector is bundled alongside the MongoDB source connector in the same artifact. So if you install the Debezium 3.0.8 MongoDB source connector, you also have the MongoDB sink connector.

Minimal configuration is required to get started with the MongoDB sink connector, for example:

{
  "connector.class": "io.debezium.connector.mongodb.MongoDbSinkConnector",
  "connection.string": "...",
  "topics": "topic1,topic2",
  "sink.database": "targetdb"
}

The following configuration properties are mandatory:

The ability to use parallel schema snapshots with the MySQL connector continues to be provided as a Developer Preview feature.

To improve snapshot performance, the MySQL connector implements parallelization to concurrently snapshot change events and generate schema events for tables. By running snapshots and generating schema events in parallel, the connector reduces the time required to capture the schema for many tables in your database.

The Debezium initial snapshot for MySQL has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

In this release, you can configure a MySQL connector to use multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, add the snapshot.max.threads property to the connector configuration, and set the property to a value greater than 1.

snapshot.max.threads=4

Based on the configuration in the preceding example, the connector can snapshot a maximum of four tables at once. If the number of tables to snapshot is greater than four, after a thread finishes processing one of the first four tables, it then finds the next table in the queue and begins to perform a snapshot. The process continues until the connector finishes performing snapshots on all of the designated tables.

For more information, see snapshot.max.threads in the Debezium User Guide.

The ability for the Oracle connector to capture changes from a logical standby continues to be available as a Developer Preview feature. When the Debezium connector for Oracle connects to a production or primary database, it uses an internal flush table to manage the flush cycles of the Oracle Log Writer Buffer (LGWR) process. The flush process requires that the user account through which the connector accesses the database has permission to create and write to this flush table. However, logical stand-by databases often have restrictive data manipulation rules, and may even be read-only. As a result, writing to the database might not be feasible.

To support an Oracle read-only logical stand-by database, Debezium introduces a property to disable the creation and management of the flush table. You can use this feature with both Oracle Standalone and Oracle RAC installations.

To enable the Oracle connector to use a read-only logical stand-by, add the following connector option:

internal.log.mining.read.only=true

For more information, see the Oracle connector documentation in the Debezium User Guide.

In this release, Debezium continues to provide exactly-once semantics for the PostgreSQL connector as a Developer Preview feature. Exactly-once delivery for PostgreSQL applies only to the streaming phase; exactly-once delivery does not apply to snapshots.

Debezium was designed to provide at-least-once delivery with a goal of ensuring that connectors capture all change events that occur in the monitored sources. In KIP-618 the Apache Kafka community proposes a solution to address problems that occur when a producer retries a message. Source connectors sometimes resend batches of events to the Kafka broker, even after the broker previously committed the batch. This situation can result in duplicate events being sent to consumers (sink connectors), which can cause problems for consumers that do not handle duplicates easily.

No connector configuration changes are required to enable exactly-once delivery. However, exactly-once delivery must be configured in your Kafka Connect worker configuration. For information about setting the required Kafka Connect configuration properties, see KIP-618.