Release Notes for Red Hat build of Debezium 3.2.4

Debezium is now built and tested using Apache Kafka 4.0

For details about compatibility between Debezium, Streams for Apache Kafka, and Kafka, check the Debezium supported configurations document.

DBZ-8875

Debezium change events contain a source information block that includes attributes that describe the origin of a change event. The source information block is a Kafka Struct data type, and can be versioned. However, in earlier Debezium versions, no version information was associated with the block.

Beginning in this release, the source information block is now versioned, and its initial version is set to 1 Future changes will increment the version value.

DBZ-8499

In earlier releases, the connector stored certain DDL events, such as TRUNCATE and REPLACE, in the internal Debezium schema history topic. Debezium does not require these types of statements to represent schema evolution. Beginning with this release, the connector no longer captures these DDL events in the internal schema history topic.

DBZ-9085

In past release, when you set the snapshot.mode of the Debezium MySQL connector to a value other than when_needed, if the connector could not find the binary log position, it logged a warning and reported that it would resume from the last available position in the logs. However, after the snapshot completed and the connector transitioned to the streaming phase, it immediately failed, reporting that the binary log position could not be located.

Beginning with this release, for consistency with the behavior of other connectors, the connector reports any error that it detects during the validation phase.

DBZ-9118

In this release, a number of Oracle LogMiner JMX metrics that were deprecated in an earlier release are no longer available. Most of the removed metrics are replaced with new metrics, but in once case a metric is removed with no replacement. Refer to the following table to learn more about the status of the removed metrics.

Review your monitoring and observability settings and adjust any that rely on metrics that are no longer available

DBZ-8647

Beginning in this release, the ReselectColumnsPostProcessor reselects Oracle LOB columns even if you configure the connector so that it does not not emit values for these columns (that is, lob.enabled is set to false). Because LOB columns store large amounts of data, mining these columns during the streaming phase increases the load on the database, the connector, and on the network, contributing to a decrease in performance. Rather than mining LOB content directly from the redo logs during the streaming phase, by using the column reselection process, you can retrieve LOB data separately, improving control and efficiency by only populating the data when you actually need it.

DBZ-8653

When the Oracle connector executes its initial query to fetch data from LogMiner, database.query.timeout.ms connector configuration property will control the duration of the query before the query is canceled. When upgrading, check the connector metric MaxDurationOfFetchQueryInMilliseconds to determine whether this new property may need adjustments. By default, the timeout is 10 minutes, but can be disabled when set to 0.

DBZ-8830

The Debezium for Oracle connector supports using JKS to configure TLS connections. To use JKS, special connector configurations are required to provide the right information to the Oracle JDBC driver. This release adds documentation that describes the process for using JKS to configure TLS connectons.

DBZ-8788

The PostgreSQL extension vector (also known as pgvector) provides an implementation of a variety of vector data types, including one called sparsevec. A sparse vector is one that stores only the populated key and value entries within the vector. Unpopulated fields are ignored to reduce the amount of space that is required to represent the data set.

The Debezium 3.0.8 release introduced the SparseVector logical type, io.Debezium.data.SparseVector. With this release, the name changes to io.Debezium.data.SparseDoubleVector.

DBZ-8585

In this release, the Debezium JDBC sink connector provides support for processing some vector data types. For details about how the connector maps vector data type, see the JDBC connector documentation.

DBZ-8571

PostgreSQL 15 introduced the pgvector extension, which provides the following data types:

In this release, the the earlier MongoDB sink connector Developer Preview feature is promoted to Technology Preview. The Debezium MongoDB sink connector differs from other vendor implementations in that it can ingest raw change events emitted by Debezium connectors without first applying an event flattening transformation. The MongoDB sink connector can take advantage of native Debezium source connector features, such as column type propagation, enabling you to potentially reduce the processing footprint of your data pipeline, and simplify its configuration.

Unlike the JDBC sink relational connector that requires an additional plug-in to be installed to use it, the MongoDB sink connector is bundled alongside the MongoDB source connector in the same artifact. So if you install the Debezium 3.2.4 MongoDB source connector, you also have the MongoDB sink connector.

Minimal configuration is required to get started with the MongoDB sink connector, for example:

{
  "connector.class": "io.debezium.connector.mongodb.MongoDbSinkConnector",
  "connection.string": "...",
  "topics": "topic1,topic2",
  "sink.database": "targetdb"
}

{
  "connector.class": "io.debezium.connector.mongodb.MongoDbSinkConnector",
  "connection.string": "...",
  "topics": "topic1,topic2",
  "sink.database": "targetdb"
}

The following configuration properties are mandatory:

For more information, see the Debezium MongoDB sink connector documentation.

DBZ-8339

In this release, the Developer Preview of the exactly-once semantics for the PostgreSQL connector is promoted a Technology Preview. Exactly-once delivery for PostgreSQL applies only to the streaming phase; exactly-once delivery does not apply to snapshots.

Debezium was designed to provide at-least-once delivery with a goal of ensuring that connectors capture all change events that occur in the monitored sources. In KIP-618 the Apache Kafka community proposes a solution to address problems that occur when a producer retries a message. Source connectors sometimes resend batches of events to the Kafka broker, even after the broker previously committed the batch. This situation can result in duplicate events being sent to consumers (sink connectors), which can cause problems for consumers that do not handle duplicates easily.

No connector configuration changes are required to enable exactly-once delivery. However, exactly-once delivery must be configured in your Kafka Connect worker configuration. For information about setting the required Kafka Connect configuration properties, see KIP-618.

This feature is promoted from Developer Preview to Technology Preview.

In previous versions of the Debezium MongoDB source connector, change streams could be opened against the deployment and database scopes, which was not always ideal for restrictive permission environments. Debezium 3.2.4 introduces a new change stream mode in which the connector operates within the scope of a single collection only, allowing for such granular permissive configurations.

To limit the capture scope of a MongoDB connector to a specific collection, set the value of the capture.scope property in the connector configuration to collection. Use this setting when you intend for a connector to capture changes only from a single MongoDB collection.

The following limitation applies to the use of this feature:

In this release, when you configure the Oracle connector to use the native Oracle LogMiner API to read and stream changes from database transaction logs, you can specify that the connector streams committed changes only. In this committed changes mode, the connector forwards change event records immediately to destination topics without first buffering them. Because the connector does not maintain an internal transaction buffer, it requires less connector memory than the default buffered logminer setting. However the source database experiences a corresponding increase in its processing load and memory requirements. To set this option, assign the value logminer_unbuffered to the Oracle connector property, database.connection.adapter.

DBZ-9351

In this release, the Debezium source connector for Informix is available as a Developer Preview feature.

The Debezium Informix source connector captures row-level changes from tables in an IBM Informix database. The connector is based on the Debezium Db2 connector, and uses the Informix Change Data Capture API for Java to capture transactional data. This API captures data from the current logical log and processes transactions sequentially.

The connector is compatible with Informix Database 12 and 14, and with version 4.50.11 of the Informix JDBC driver.

The ability to use parallel schema snapshots with the MySQL connector continues to be provided as a Developer Preview feature.

To improve snapshot performance, the MySQL connector implements parallelization to concurrently snapshot change events and generate schema events for tables. By running snapshots and generating schema events in parallel, the connector reduces the time required to capture the schema for many tables in your database.

DBZ-6472

The Debezium initial snapshot for MySQL has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

In this release, you can configure a MySQL connector to use multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, add the snapshot.max.threads property to the connector configuration, and set the property to a value greater than 1.

snapshot.max.threads=4

snapshot.max.threads=4

Based on the configuration in the preceding example, the connector can snapshot a maximum of four tables at once. If the number of tables to snapshot is greater than four, after a thread finishes processing one of the first four tables, it then finds the next table in the queue and begins to perform a snapshot. The process continues until the connector finishes performing snapshots on all of the designated tables.

For more information, see snapshot.max.threads in the Debezium User Guide.

DBZ-823

The ability for the Oracle connector to capture changes from a logical standby continues to be available as a Developer Preview feature. When the Debezium connector for Oracle connects to a production or primary database, it uses an internal flush table to manage the flush cycles of the Oracle Log Writer Buffer (LGWR) process. The flush process requires that the user account through which the connector accesses the database has permission to create and write to this flush table. If stand-by database policies restrict data manipulation or prohibit write operations, the connector cannot write to the flush table.

To support an Oracle read-only logical stand-by database, Debezium introduces a property to disable the creation and management of the flush table. You can use this feature with both Oracle Standalone and Oracle RAC installations.

To enable the Oracle connector to use a read-only logical stand-by, add the following connector option:

internal.log.mining.read.only=true

internal.log.mining.read.only=true

For more information, see the Oracle connector documentation in the Debezium User Guide.

By default, the Debezium Oracle connector uses an adapter that connects to the native Oracle LogMiner utiltiy to retrieve data from the redo logs. In database environments that support XStream, you can instead configure the connector to ingest change events through the Oracle XStream API.

For information about configuring Debezium to use the Oracle XStream adapter, see the Debezium Oracle connector documentation.