Releases: apache/flink-cdc
Release 3.1.0
Release Announcement
https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/
Release Notes
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354387
Downloads
Apache Flink CDC 3.1.0 tar
Apache Flink CDC 3.1.0 (asc, sha512)
Apache Flink CDC 3.1.0 Source Release (asc, sha512)
Pipeline Connectors Jars
- MySQL Pipeline Connector 3.1.0 (jar, asc, sha1)
- Doris Pipeline Connector 3.1.0 (jar, asc, sha1)
- StarRocks Pipeline Connector 3.1.0 (jar, asc, sha1)
- Kafka Pipeline Connector 3.1.0 (jar, asc, sha1)
- Paimon Pipeline Connector 3.1.0 (jar, asc, sha1)
Source Connector Jars
- Db2 source connector 3.1.0 (jar, asc, sha1)
- MongoDB source connector 3.1.0 (jar, asc, sha1)
- MySQL source connector 3.1.0 (jar, asc, sha1)
- OceanBase source connector 3.1.0 (jar, asc, sha1)
- Oracle source connector 3.1.0 (jar, asc, sha1)
- Postgres source connector 3.1.0 (jar, asc, sha1)
- Microsoft SQL Server source connector 3.1.0 (jar, asc, sha1)
- TiDB source connector 3.1.0 (jar, asc, sha1)
- Vitess source connector 3.1.0 (jar, asc, sha1)
Release 3.0.1
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0
Downloads
Flink CDC
Flink CDC 3.0.1 (tar, asc, sha512)
The binary release can be verified by public key with fingerprint F8822354BF424A5CFF9AFD57BAF388953EF5327A
.
Flink CDC Pipeline Connectors
All connectors are release in JAR and available in Maven central repository.
- Apache Doris pipeline connector 3.0.1 (jar, asc, sha1)
- MySQL pipeline connector 3.0.1 (jar, asc, sha1)
- StarRocks pipeline connector 3.0.1 (jar, asc, sha1)
- Values pipeline connector 3.0.1 (for testing purpose, jar, asc, sha1)
Flink CDC Source Connectors
All connectors are release in JAR and available in Maven central repository.
- Db2 source connector 3.0.1 (jar, asc, sha1)
- MongoDB source connector 3.0.1 (jar, asc, sha1)
- MySQL source connector 3.0.1 (jar, asc, sha1)
- OceanBase source connector 3.0.1 (jar, asc, sha1)
- Oracle source connector 3.0.1 (jar, asc, sha1)
- Postgres source connector 3.0.1 (jar, asc, sha1)
- Microsoft SQL Server source connector 3.0.1 (jar, asc, sha1)
- TiDB source connector 3.0.1 (jar, asc, sha1)
- Vitess source connector 3.0.1 (jar, asc, sha1)
Improvements and Bug fixes
1.[cdc-base][hotfix] Fix SourceSplitSerializer to get totalFinishedSplitSize from recovery in 4th version
2.[hotfix][cdc-common] Fix NoSuchMethodError of CollectionUtil.newHashMapWithExpectedSize
3.[cdc][e2e] Unify logic to clean docker image when connector e2e test finished (#2978)
4.[pipeline-connector][mysql] fix timestamp with timezone format (#2952)
5.[doc] fix invalid page in website and update yaml in tutorial.
6.[cdc-connector][oracledb] Fix oracle read exceed high_watermark in backfill phase
7.[pipeline-connector][starrocks] Optimize char type mapping for starrocks pipeline connector (#2895)
8.[pipeline-connector][doris][starrocks] Remove SimpleDateFormat for thread safe (#2954)
9.[build] Add github labeler for pipeline connector. (#2851)
10.[docs][cdc-connector][oceanbase] update docker image and os compatibility in quickstart (#2897)
11.[docs] Add Supported Connectors and Supported Flink Versions to Overview. (#2920)
12.[cdc-common][pipeline-doris] Fix typo and wrong value for 'precision' of time type. (#2886)
13.[cdc-connector][mongodb] Avoid mongodb source to read data after high_watermark in backfill phase (#2893)
14.[cdc-connector][oceanbase] Fallback to current timestamp when query gts failure (#2868)
15.[cdc-connector][sqlserver][tests] Fix UT errors by correcting right output (#2864)
16.[cdc] Add Deprecated annotation to source function which connector implemented incremental
17.[hotfix] Remove mysqlToDoris.yml
18.[hotfix] Fix BinaryRecordData deserialize
19.[postgres] Add commits of PostgresConnection back to distinguish with debezium's origin class (#2839)
20.[hotfix][cdc-rumtime][minor] Clean up useless code in BinaryRecordDataSerializer (#2845)
21.[cdc-connector][sqlserver] Fix SqlServerTableSource params order error (#2847)
22.[docs] Fix overview docs to remove prefix of pipeline options (#2833)
23.[3.0][docs] fix mysql pipeline connector option in readme (#2834)
Contributor List (sorted alphabetically)
- GOODBOY008
- He Wang
- Hongshun Wang
- joyCurry30
- Kunni
- Shawn Huang
- TorinJie
- Xin Gong
- gongzhongqiang
- xuzifu666
- 张田
Release 3.0.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-3.0
Downloads
Flink CDC
Flink CDC 3.0.0 (tar, asc, sha512)
The binary release can be verified by public key with fingerprint C460A307EAF2701FC9D774EF96719B746D3D0C8A
.
Flink CDC Pipeline Connectors
All connectors are release in JAR and available in Maven central repository.
- Apache Doris pipeline connector 3.0.0 (jar, asc, sha1)
- MySQL pipeline connector 3.0.0 (jar, asc, sha1)
- Starrocks pipeline connector 3.0.0 (jar, asc, sha1)
- Values pipeline connector 3.0.0 (for testing purpose, jar, asc, sha1)
Flink CDC Source Connectors
All connectors are release in JAR and available in Maven central repository.
- Db2 source connector 3.0.0 (jar, asc, sha1)
- MongoDB source connector 3.0.0 (jar, asc, sha1)
- MySQL source connector 3.0.0 (jar, asc, sha1)
- OceanBase source connector 3.0.0 (jar, asc, sha1)
- Oracle source connector 3.0.0 (jar, asc, sha1)
- Postgres source connector 3.0.0 (jar, asc, sha1)
- Microsoft SQL Server source connector 3.0.0 (jar, asc, sha1)
- TiDB source connector 3.0.0 (jar, asc, sha1)
- Vitess source connector 3.0.0 (jar, asc, sha1)
Features and Improvements
- [pipeline-connector][starrocks] Fix char/varchar length inconsistency between cdc and starrocks (#2830)
- [cdc-cli][cdc-composer] Use "source-table" and "sink-table" as keywords of route (#2825)
- [build][hotfix] Remove duplicate dependency (#2826)
- [cdc-cli][cdc-dist] Support loading global config from FLINK_CDC_HOME (#2822)
- [cdc-common][hotfix] Remove redundant prefix of pipeline options (#2821)
- [cdc-source-connector][oceanbase][e2e] Add OceanBase e2e test case (#2521)
- [build] Use flink-shaded-force-shading to force all submodules to generate dependency-reduced-pom.xml
- [build] Use ${project.version} instead of ${revision} for dependency version
- [build] Skip maven-deploy-plugin for test modules and flink-cdc-dist
- [cdc-pipeline-connector][mysql] Fix precision problem of BIT type conversion (#2820)
- [cdc-source-connector][mysql] fix deserialization issue on table metadata binlog event (#2682)
- [pipeline-connector][mysql] Remove unnecessary serverTimeZone in DebeziumEventDeserializationSchema (#2816)
- [build][hotfix] Remove useless suppressions. (#2817)
- [cdc-source-connector][mysql] Fix attempted load of isSuspended in MySqlSplitSerializer::deserializeSplit when cdc version <= 2.2.0 (#2550)
- [cdc-pipeline-connector][mysql] Ensure the inference of MEDIUMINT_UNSIGNED type matches INT type in table schema (#2811)
- [cdc-connector][oceanbase] Add option for obcdc extra configs (#2543)
- [pipeline-connector][mysql] Enable send schema change by default (#2815)
- [3.0][cdc-runtime] Remove waitForFlushSuccess field in request handler (#2812)
- [pipeline-connector][starrocks] Add starrocks pipeline connector (#2765)
- [pipeline-connector][doris] add doris pipeline connector. (#2810)
- [cdc-pipeline-connector][mysql] Ensure the precision inference of DECIMAL type matches table schema
- [3.0][cdc-common & runtime] Remove useless flink imports
- [cdc-pipeline-connector][mysql] Add tests for mysql pipeline connector
- [cdc-pipeline-connector][mysql] Support more mysql types
- [cdc-pipeleine-connecotr][mysql] Send CreateTableEvent in the mysql pipeline connector
- [cdc-pipeline-connector][mysql] Parse the alter statement to generate SchemaChangeEvent
- [cdc-pipeline-connector][mysql] Introduce mysql cdc pipeline DataSource
- [cdc-cli][cdc-composer][cdc-dist] Support submitting job to general Flink environments (#2807)
- [cdc-runtime][hotfix] Setup waitFlushSuccess before responding operator to flush. (#2805)
- [cdc-common] Introduce "pipeline.local-time-zone" config option which help handle time zone well (#2797)
- [cdc-common][hotfix] Remove useless flink imports (#2803)
- [build] Let surefire plugin to keep the full stacktrace on test failure (#2804)
- [pipeline-common] Use column name to check whether the schema contains specific column or not. (#2801)
- [cdc-composer][route][sink] Add name for route and improve name for sink
- [cdc-composer][hotfix] Always chain operator by default
- [cdc-composer][sink] Use correct name for sink operator
- [cdc-runtime] SchemaRegistry should complete the future after making the checkpoint (#2800)
- [cdc-connector][sqlserver][test...
Release 2.4.2
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4
Improvements and Bug fixes
- [sqlserver] Fix get latest lsn (#2551)
- [sqlserver] Add support bigint type as table split column (#2416)
- [hotfix] Fix vitess docker image tag
- [e2e] Remove db docker image after test cases are executed. (#2508)
- [postgres] Not drop replication slot for stream split (#2436)
- [postgres] Close jdbc connection after creating replication slot for stream split (#2425)
- [cdc-base] Fix TM hangs caused by uncaught exception (#2511)
- [postgres] Close idle readers when snapshot finished (#2400)
- [hotfix][mysql] Keep assigned splits in order to fix wrong meta group calculation
- [mysql] Add finished unack splits to state for the MysqlSourceReader (#2399)
- [cdc-base] Fix parsing error when serializing and deserializing the table name with dot (#2443)
- [oracle] Correct the naming error (#2405)
- [sqlserver] Add table filter to speed up SqlServerSchema read (#2369)
- [hotfix][sqlserver] Fix backfill stream task hang (#2374)
- [hotfix][mysql] Trim default value for BIGINT and SMALLINT types when parsing MySQL DDL (#2418)
- [Doc] Use redo log instead of binlog in Oracle document (#2408)
- [mysql] Filter databases that do not need to be read when discovering tables (#2160)
- [test] Fix TiDBE2eITCase by updating the container version to avoid
tikv disk full
failure (#2402) - [test][postgres] Close postgres containers after tests and fix container reuse bug (#2394)
- [mysql] quota the field names when executing mysql query (#2381) (#2388)
- [test][mysql] Provide the server-time-zone setting in MySqlSourceExampleTest#testConsumingAllEvents (#2364)
- [hotfix][mysql] Catch underlying throwable in snapshot reading.
- [hotfix][mysql] Fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (#2384)
- [db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure
- [mongodb][hotfix] Fix SourceRecord range check (#2343)
- [polish] Add type parameter for MySqlRecords(#2300)
- [doc] fix errors in PostgresParallelSourceExample (#2324)
- [docs][oracle] Add an example of Incremental Snapshot based DataStream for Oracle CDC Connector
- [mongodb] [tests] Loosen E2E timeout limitations
- [build] Add vcs.xml for IDEA (#2064)
- [hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(xx = database, schema or table) (#2212)
Contributor List (sorted alphabetically)
- BIN
- FocusComputing
- GOODBOY008
- Hang Ruan
- Hongshun Wang
- Jiabao Sun
- Kunni
- Leonard Xu
- Malcolmjian
- North.Lin
- Paddy Gu
- Tyrantlucifer
- e-mhui
- gongzhongqiang
- wuzhenhua
- yuxiqian
- zhaomin
Release-2.4.1
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4
Features and Improvements
- [hotfix] Fix sqlserver monitor same table in other database (#2335)
- [debezium] Fix DebeizumSourceFunction can not do savepoint after close (#2259)
- [mysql] skip closing reader when the reader received the binlog split (#2261)
- [MySQL] Optimize how to construct config table.include.list (#2274)
- [oracle] Fix class conflict when using multi cdc sql connector
- [MySQL] Fix the startup mode exception message (#2264)
- [hotfix][mysql] remove unused code (#2231)
- [mysql] Fix assigning duplicate snapshot splits when enable scan newly added tables (#2326)
- [docs] Add connector features table to help users quickly understand the supported features of each connector
- [mysql] Remove the finished snapshot splits for the binlog split when restoring from the checkpoint (#2292) (#2318)
- [hotfix][sqlserver] Fix sqlserver close idle and chunk key column missing (#2310)
- [docs] Update Postgres CDC document for slot.name option (#2280)
- [postgres] Fix the slot name conflict bug (#2251)
- [docs] add docs for the mysql tables without primary keys (#2238)
- [oracle] Use oracle connection in context for each reader subtask (#2254)
- [hotfix] Update support version (#2256)
Contributor List (sorted alphabetically)
- Hang Ruan
- Leonard Xu
- SileiWu
- empcl
- gongzhongqiang
- skylines
- syyfffy
Release-2.4.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.4
Features and Improvements
- [hotfix] Add vitess connector to the release profile
- [docs][hotfix] Update debezium reference links to 1.9 version
- [build] Update the copyright year to 2023 (#2205)
- [postgres] Fix postgres e2e test
- [postgres] scan.incremental.snapshot.enabled is closed by default
- [postgres] Backfill task will be able to end when there is not new change data but read the ending lsn
- [postgres] Create slot for backfill task before snapshot reading
- [postgres] Prepare a slot for the unique global stream split
- [mysql] Fix GTID issues to recover from checkpoint normally in specifying startup mode (#2220)
- [doc][hotfix] Fix driver version and correct vitess doc path (#2224)
- [vitess] Add e2e case for Vitess connector
- [vitess] Add Vitess tests in Azure pipeline
- [vitess] Add Vitess CDC connector (#456)
- [debezium] Bump debezium version to 1.9.7.Final (#2156)
- [oracle] Fix the backfill task not running bug in oracle cdc connector (#2218)
- [hotfix][sqlserver] Merge high and low watermark data during snapshot reading process (#2179)
- [Postgres] Use Incremental Snapshot Framework for Postgres CDC Connector (#2216)
- [oracle] Fix the incremental phase not skip the data that has been read in the snapshot full phase (#2215)
- [mongodb] Support specific timestamp startup mode. (#2207)
- [tests][mysql] Shutdown mysql container timely for legacy mysql tests
- [mongodb][hotfix] Fix regex inference of database or collection only contains dash. (#2209)
- [hotfix][build] Use different name for each job
- [hotfix][Mysql] MySqlBinlogSplitAssigner should support method isStreamSplitAssigned
- [MySQL] Remove deleted tables when using a new table filter and restarting from checkpoint
- [build][mysql][e2e] Increase the azure test job timeout to 90 minutes
- [mysql] Fix MySql CDC Connector outputs duplicated binlog when enable newly-added-table
- [mysql][minor] Print clear log information for key path
- [mongodb][hotfix] Fix drop or rename record cause documentKey being empty. (#2210)
- [ci][oceanbase] Only compile necessary modules for self hosted pipeline
- [doc] Add docs for close idle readers feature. (#2208)
- [cdc-base] Close idle readers when snapshot finished (#2202)
- [mongodb] Add support for mongodb+srv connection protocol. (#2203)
- [oceanbase] Always cleanup resources after reading before exit (#2189)
- [mongodb][hotfix] Fix NullPointerException when executing ddl operation in MongoDB (#2107)
- [mysql-cdc] Supports MYSQL_TYPE_TYPED_ARRAY column type when parsing the table map event
- [mysql-cdc][hotfix] Optimize the error msg when binlog expire in source
- [mysql-cdc] Optimize the error msg when binlog expire or server id conflict This closes #2010.
- [oceanbase] add jdbc options and support oracle mode (#1854)
- [mysql-cdc] Add handler for catching async exceptions in snapshot reading executor This closes #2016.
- [hotfix][mysql-cdc] Use session timezone instead of local time zone as the default server time zone
- [mysql] Fix NullPointerException when database name or table name contains dot
- [mysql] Fix NullPointerException caused by mysql ignores the capitalization when splitting chunks
- [sqlserver] Fix old change data that will be captured when the latest mode starts (#2176)
- [core] support print configuration options for connectors (#2099)
- [oceanbase] support libobcdc 4.x and fix restore timestamp config (#2161)
- [sqlserver] Sqlserver incremental source. (#1732)
- [OceanBase][test] Update oceanbase and oblogproxy docker image (#2060)
- [oceanbase] use real charset to convert change record (#2159)
- [mysql] Support tables which do not contain a primary key (#2150)
- [common] Bump flink to 1.17.0 (#2057)
- [doc] add mongodb cdc chinese doc (#2042)
- [hotfix][test] ignore OracleConnectorITCase.testConsumingAllEvents util issue 1875 fix (#2111)
- [docs][db2] Correct CHAR(n) to CHARACTER(n) (#2052)
- [postgres] Change config option slot.name to be required (#1996)
- [docs] Update the outdated DingTalk QR code
- [mongodb][hotfix] Fix pollAwaitTimeMillis does not take effect (#1995)
- [tests][mysql] Skip unnecessary IT Cases which uses legacy debezium source implementation
- [mysql] Add note for MySQL TINYINT(1) type to Flink SQL Boolean type
- [github] Minor typo in issue bug-report (#2020)
- [mysql-cdc] Fix race condition between split and coordinator thread that async exception might not be caught
- [mysql/oracle] Fix the wrong comments in ChunkSplitter
- [hotfix][docs] Correct the flink file path in MySQL docs
- [hotfix][mongodb] Fix resume token not found #1879 (#1938)
- [mongodb][hotfix] Fix authentication failed when using a non admin auth source and duplicated usernames #1934 (#1935)
- [mysql] Add various test cases to cover newly added table process from read binlog blocking to read binlog un-blocking
- [hotfix][base] Fix typo in java docs
- [mysql] Optimize the newly added table process from read binlog blocking to read binlog un-blocking
- [minor][mysql] Rename SplitsSize to SplitsNumber
- [hotfix][mysql] Correct meta group id as it is 0 based
- [hotfix][base][tidb][mysql] Fix wrong condition for execution timeout
- [mongodb] Improve code structure (#1900)
- [build] Use issue form templates (#1830)
- [ci] Fix for dubious repo ownership issue (#1843)
- [docs] Update mysql-cdc.md to recommend latest version (#1842)
- [ci] Fix for dubious repo ownership issue (#1840)
- [docs][sqlserver] Update data type mapping (#1802)
- [ci][tidb] Fix dns cache works incorrect in latest jdk 8. (#1809)
- [postgresql] Upgrade postgresql driver from 42.2.26 to 42.2.27 to avoid CVE-2022-41946 (#1773)
- [minor][common] Improve the changelogMode expression (#1793)
- [hotfix][debezium] Forward root cause in DebeziumSourceFunction (#1791)
- [docs][oracle] Add new introduced config options from incremental snapshot framework (#1719)
- [mysql] Fix backward compatibility on deserializing binlog offset from old versions (#1758)
- [hotfix][docs] Correct the startup option's data type in Mysql docs and duplicate batch.size in MongoDB docs (#1751)
- [hotfix][docs] Correct the mysql driver version in MySQL docs
- [docs][site] update the supported cdc connectors picture
- [oracle] Add missing supported config options for oracle table source (#1727)
Contributor List (sorted alphabetically)
- Hang Ruan
- He Wang
- JasonLee
- Jiabao Sun
- Jingsong Lee
- Xiao Meng
- Leonard Xu
- Qingsheng Ren
- Robert Metzger
- Sergey Nuyanzin
- Simonas Gelazevicius
- Yaroslav Tkachenko
- Tigran Manasyan
- Tyrantlucifer
- ZhongLinLeo
- bb chen
- ehui
- emhui
- gintarasm
- gongzhongqiang
- liangyaohui97
- lin.zl
- molsionmo
- skylines
- tison
- wallkop
- wangxiaojing
- wuzhenhua
- xixingya
- zhaomin
- zhuyuan03
- 含风
Release-2.3.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.3
Features and Improvements
- [mysql] Improve error message under the case that start reading from earliest but schema change happened before (#1724)
- [mysql] Use event filter to drop events earlier than the specified timestamp in timestamp startup mode (#1724)
- [docs][oracle] Add new introduced config options from incremental snapshot framework (#1718)
- [docs][oceanbase][zh] Update outdated sections and translate them to Chinese
- [hotfix][docs] Add link for Oceanbase ZH docs
- [docs][oceanbase] Add Chinese documents for oceanbase-cdc connector(#1611)
- [db2] Fix stream api example in db2 doc. (#1725)
- [oracle] Add missing supported config options for oracle table source (#1720)
- [mysql] Fix MySqlBinlogSplitAssigner may reassign split after exception restart (#1713)
- [docs] Repair Connector Data Type links
- [docs] Update the connector download link in the document
- [docs] Update the flink cdc picture with supported database vendors.
- [tidb] Fix unstable TiDB region changed test. (#1702)
- [docs][mongodb] Add docs for MongoDB incremental source
- [oracle][mysql] Improve the Oracle all data types test and clean up debug logs
- [oracle] Properly support TIMESTAMP_LTZ type for oracle cdc connector
- [oracle]support use user given column as split column
- [oracle][minor] Clean up useless methods and classes
- [oracle] Parallel read supports startup of startupOptions
- [oracle] Add full type field test
- [oracle]OracleTableSource support splitSize, fetchSize ,connectMaxRetries, connectionPoolSize and add test
- [hotfix][oracle][db2] Update the license header
- [cdc-base][oralce] Remove useless method of JdbcSourceFetchTaskContext
- [cdc-base][oracle] Improve the base methods base on Oracle specific requirements
- [oracle] Support custom url for incremental snapshot source
- [oracle] Introduce SCAN_INCREMENTAL_SNAPSHOT_ENABLED option for OracleTableSourceFactory
- [minor][oracle] Improve tests and code format
- [oracle] Ignore testCheckpointAndRestore util DBZ-5245 and DBZ-4936 fix
- [oracle][test] Add tests for Oracle incremental snapshot source
- [oracle] Use ROWID as split column and replace limit syntax with ROWNUM
- [oracle] Let LogMinerStreamingChangeEventSource do not rely on oracle table LOG_MINING_FLUSH
- [oracle] Use Incremental Snapshot Framework for Oracle CDC Connector (#1079)
- [docs] Bump Flink version to 1.16.0
- [common] Bump Flink version to 1.16.0
- [docs][db2] Add db2 to README.md (#1699)
- [tidb] Checkpoint is not updated long after a task has been running (#1686)
- [hotfix] Add method getMaxResolvedTs back to class CDCClient. (#1695)
- [docs] Bump connector version to flink 1.15.2 in docs (#1684)
- [tidb] Fix data lost when region changed (#1632)
- [hotfix][docs] Correct reference link for DB2 docs (#1683)
- [mysql] Update docs of specifying starting offset feature of MySQL CDC source
- [hotfix][mysql] Remove unused constructor in MySqlTableSource
- [mysql] Support starting reading binlog from earliest offset / specific offset / timestamp (#1661)
- [mongodb] Support incremental snapshot for MongoDB CDC Connector
- [cdc-base] Extract a higher level IncrementalSource independent of JDBC dialects
- [hotfix][docs] Fix db2 doc download link (#1681)
- [docs][db2] Add Db2 CDC connector doc link
- [docs][db2] Add Db2 cdc connector docs
- [build][db2] Add Db2 pipeline script
- [e2e][db2] Add Db2 e2e test
- [db2] Add sql connector for Db2 CDC (#1626)
- [db2] Introduce Db2 cdc connector (#450)
- [tidb] Fix data loss in tidb-cdc when switch from snapshot read to change event read (#1207)
- [hotfix][cdc-base] Fix OffsetFactory is not initialized when restored from checkpoint (#1657)
- [common] Bump Flink version to 1.15.2 (#1504)
- [cdc-base] Abstract naming for variables of incremental framework (#1646)
- [mysql] ChunkSplitter supports to snapshot when splitting chunks for a table (#1463)
- [cdc-base] Fix StreamSplit's endingOffset of StreamSplitAssigner (#1659)
- [cdc-base] JdbcSourceConfig can be template type of DataSourceDialect (#1639)
- [mysql] Use the latest received meta of binlog split when request the next meta (#1479)
- [minor] Use Collections.singletonList instead of Arrays.asList if possible (#1569)
- [oceanbase] Add new deserialization schema with runtime converter (#1356)
- [mysql] Avoid duplicate split requests when add new table (#1156)
- [postgres] Upgrade org.postgresql:postgresql from 42.2.14 to 42.2.26 for CVE-2022-31197 (#1616)
- [test][e2e] Upgrade org.postgresql:postgresql from 42.3.1 to 42.2.26 for CVE-2022-31197(#1616)
- [hotfix][typo] Fix typo from schemaLess to schemaless
- [cdc-base] Reduce the memory usage of JM by sharing table schemas between splits
- [cdc-base] Release the scan fetcher thread resource after reading finished (#1619)
- [cdc-base] Optimize pure binlog phase check logic to improve performance (#1620)
- [cdc-base] Let all records of one snapshot split don't cross checkpoints (#1622)
- [postgres] Introduce changelog mode configuration with upsert/all modes
- [common] Introduce changelog mode configuration with upsert/all modes
- [hotfix][docs] Fix docs for scan.incremental.snapshot.chunk.size (#1617)
- [cdc-base] Adopt streaming merge during snapshotSplit scan to avoid OOM
- [docs][oceanbase][zh] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1604)
- [docs][oceanbase] Add video tutorials that synchronizes data from OceanBase to Elasticsearch (#1605)
- [mysql] Fix parse error when ALTER SQL statement contains default string value and collate. (#1544)
- [docs][mysql] Add regex examples for multiple tables (#1564)
- [oceanbase] Shade apache commons dependency to avoid class conflict (#1532)
- [postgres] add geometry type support. (#1554)
- [build] Polish CI scripts and remove useless code (#1540)
- [mysql] fix parsing ddl error when use string default values for numeric types (#1522)
- [mysql] Enumerate captured tables lazily (#1507)
- [mysql] Reduce the memory usage of JM by sharing table schemas between splits (#1523)
- [hotfix][docs] Fix comment typo in MysqlSnapshotSplit (#1513)
- [mysql] Let all records of one snapshot split don't cross checkpoints (#1505)
- [tests] Remove usage of deprecated EnvironmentSettings.Builder#useBlinkPlanner (#1493)
- [cdc-base][mysql] Fix split key comparison when may contain different types (#1499)
- [docs][mysql] Update the description of mysql server-time-zone option (#1497)
- [mysql] Use local timezone as the default value of 'server-time-zone' option (#1407)
- [docs][postgres] Add two frequently debezium options in Postgres connector document (#1142)
- [mongodb] Allow mongo ARRAY to be converted to string type in Flink (#1475)
- [hotfix][docs] Fix the page links in MySQL Chinese document (#1466)
- [oceanbase][test] Use memory mode of obcdc and double startup timeout duration (#1474)
- [mysql][polardbx] Stop container timely in tests (#1468)
- [mysql][ploardbx] Support all charsets in column level (#1166)
- [mongodb] Expose updateLookup option for stream source. (#1461)
- [oceanbase] Use utc offset as 'server-time-zone' option's default value (#1443)
- [mysql][minor] Skip capture newly added tables according snapshotassigner's status (#1449)
- [polardbx][test] Bump the polardb-x image version from 2.0.1 to 2.10 (#1423)
- [build][test] Specify surefire provider to surefire-junit47 (#1423)
- [hotfix][docs] Fix typo in bug report doc (#1460)
- [minor][oracle] Rename JdbcConfigurationUtil to OracleJdbcUrlUtils
- [oracle] Support custom JDBC URL for Oracle (#1258)
- [docs] Correct the data type of 'server-id' option to String (#1444)
- [hotfix][test] Fix MySqlSourceITCase failure due to StreamExecutionEnvironment#getConfiguration returned unmodifiable object (#1446)
- [test] Fix tests missed 'update_before' records if the primary keys of source and sink are different (#1451)
- [hotfix][oracle] Fix the shell script interpreter assignment should be at first line (#1452)
- [hotfix][tidb-cdc] TiDB CDC dealUnsignedColumnValue throw NullPointerException. (#1448)
- [polardbx][test] Rename PolardbxSourceTCase to PolardbxSourceITCase (#1447)
- [mysql] Enable pass custom jdbc properties for debezium mysql connection (#1217)
- [mysql] Release the debezium reader thread resources after reading finished (#1358)
- [oceanbase] Introduce 'table-list' option to support capture list of tables (#1369)
- [cdc-base] Flink CDC base registers the identical history engine on multiple tasks (#1340)
- [hotfix][mysql] Fix compile error due to merge conflict
- [mysql] Generates multiple chunks when approximate row count is bigger than chunk size (#1193)
- [cdc-base] Fix NPE during snpashot scan phase (#1339)
- [docs] Improve MySQL ZH document and update the outdated links (#1435)
- [docs] Translate mysql cdc doc to Chinese (#1391)
- [mysql] Introduce 'scan.incremental.snapshot.chunk.key-column' option to support splitting table by custom column (#1415)
- [mysql] Optimize pure binlog phase check logic to improve performance (#1392)
- [oracle] Fix mining session stopped due to 'No more data to read from socket'
- [common] Ignore retriableException to make the debezium engine retry works
- [test][oracle] Fix unstable tests OracleE2eITCase due to reuse container
- [tidb] Fix NPE when the specified table does not exist (#1411)
- [docs] Add step up SQLServer Database cdc document in sqlserver-cdc.md.
- [mongodb] Expose batch.size config option for table source. (#1387)
- [mongodb] Example tests for MongoDBSource (#138...
Release-2.2.1
Release Notes
Improvements and Bug fixes
- [docs] Update the dependencies version used in docs to 2.2.0
- [hotfix][docs] Fix jar link for PolarDB-X docs (#1013)
- [docs] Add example for Scan Newly Added tables feature (#1016)
- [hotfix][docs] Fix misspell of docs title
- [docs][mysql] Add note to make the tableList usage and dependencies clearer (#1037)
- [docs] Update the supported sink vendors
- [sqlserver] Add more supported(tested) SqlServer versions (#1049)
- [oceanbase] Remove useless dependency and code of test (#1046)
- [docs][mysql] Fix connector document link error (#1043)
- [mysql] Serialize isSuspended flag for MySqlBinlogSplit (#1055)
- [common] Apply filter on ResolvedSchema to keep physical columns only (#1097)
- [docs][mongodb] Update document about MongoDB database and collection regex filter (#1090)
- [mysql] Throw RuntimeException timely in snapshot scan phase (#1099)
- [hotfix][tidb] Fix RowKind.INSERT judgement condition (#1104)
- [mysql] Catch and hold exception in chunk splitter thread and rethrow in getNext() (#1110)
- [mysql] Use the gtid set firstly when compare the binlog offset (#1111)
- [mysql] Add heartbeatInterval field to equals() and hashCode() (#1118)
Download
- flink-sql-connector-mongodb-cdc-2.2.1.jar
- flink-sql-connector-mysql-cdc-2.2.1.jar
- flink-sql-connector-oceanbase-cdc-2.2.1.jar
- flink-sql-connector-oracle-cdc-2.2.1.jar
- flink-sql-connector-postgres-cdc-2.2.1.jar
- flink-sql-connector-sqlserver-cdc-2.2.1.jar
- flink-sql-connector-tidb-cdc-2.2.1.jar
Documentation
Release-2.2.0
Release Notes
Documentation: https://ververica.github.io/flink-cdc-connectors/release-2.2/
Features and Improvements
- [docs] Update the supported database vendors and improve the notes.
- [docs][mysql][polardbx] Add polardbx tutorials (#1003)
- [docs][mysql] Remove redundant docker environment variable in tutorials (#1005)
- [cdc-base] Add MySQL Incremental Source to validate the Incremental Snapshot Framework(#776)
- [cdc-base] Improve the Incremental Snapshot Interfaces
- [cdc-base] The first version of Incremental Snapshot Framework abstraction
- [hotfix][docs] Fix markdown title due to typo in readme
- [oceanbase] Bump the oblogclient version to 1.0.3 to fix the NPE in oblogclient connection (#998)
- [docs] Clarify ownership to comply with ASF trademark policies (#1000)
- [build] Decrease the azure job timeout to reasonable 40 minutes (#991)
- [hotfix][site] Use Flink with its trademark
- [docs] Adopt the full name 'Apache Flink' with its trademark notice (#988)
- [hotfix][docs] Update the copyright
- [docs][mysql] Correct the jdbc driver version in documentation
- [docs] Organize the connector docs in alphabetical order
- [docs][tidb] Add tidb cdc connector document
- [hotfix][docs] Fix typo in tidb-tutorial.md and tidb-tutorial-zh.md
- [mysql] Suspend SourceReader eagerly to reduce restoration time
- [docs][oceanbase] Add quickstart tutorials for OceanBase CDC connector
- [docs][oceanbase] Add docs for OceanBase CDC connector
- [build][oceanbase] Use self-hosted agent for oceanbase cdc tests
- [oceanbase] Add OceanBase CDC connector
- [tidb] remove useless code.
- [test][e2e] Use single fork in tests
- [test][e2e] Clean up database resources for every e2e tests.
- [e2e] bump flink e2e test version.
- [tidb] Add tidb cdc connector e2e test.
- [tidb] Add flink1.14 support.
- [hotfix][mysql] Avoid schema isn't known error when scanNewlyAddedTable enabled (#965)
- [mysql][polardbx] Remove debug print code in PolardbxSourceTCase (#968)
- [tidb] Introduce runtime converter to improve the deserialization
- [tidb][test] Add timezone tests for tidb
- [tidb] Remove useless config options
- [docs][site] Add favicon fot website index html
- [docs][site] Introduce static index html for the project website (#961)
- [mysql] Split snapshot chunks asynchronously (#931)
- [docs][tidb] Add index for tidb tutorial docs
- [tidb][docs] Add tidb connector tutorial (#938)
- [tidb] Add metadata column ITCase and improve the code style
- [tidb][test] Fix tikv port conflicts
- [tidb] Support metadata column
- [tidb][test] Add tidb pipeline in azure CI
- [tidb][test] Add TiDBTableSourceFactoryTest and TiDBConnectorITCase
- [tidb] Fix tikv.pd.addresses not work and use ResolvedSchema instead of TableSchema.
- [tidb] Bump tikv client version to v3.2.0
- [tidb] The first draft of tidb cdc connector
- [docs] List supported databases in docs (#954)
- [mysql] Fallback to build schema by 'desc table' case when parsing DDL failure (#949)
- [docs] Fix download url in mongodb tutorial (#951)
- [docs] Update DingTalk QR code to English version (#950)
- [mongodb] Match multiple database and collection names using a regular expression in MongoDB. (#940)
- [mysql] Allow custom JDBC URL parameters in MySQL CDC source (#921)
- [mongodb][test] Fix unstable tests of MongoDBConnectorITCase (#944)
- [mysql] Improve mysql heartbeat feature and tests
- [mysql] Original implementation of mysql heartbeat feature
- [common] Update issue templates
- [hotfix][mysql] Fix MySqlSourceReader#snapshotState should return unfinishedSplits (#926)
- [docs] Fix the outdated link of flink download link (#922)
- [hotfix][mysql] Fix the binlog position didn't update when emit DDL (#925)
- [mongodb] Correct the TypeInformation argument of DebeziumDeserializationSchema in SourceFuntion (#780)
- [common] Allow user to pass custom JsonConverter configs in JsonDebeziumDeserializationSchema (#615)
- [mysql] Improve the test to cover table with varbinary primary key (#879)
- [mysql] Support primary key with varbinary type (#879)
- [format][changelog-json] Remove the support of changelog-json format (#912)
- [hotfix][minor] Fix the test code typo
- [mysql] Fix ArrayIndexOutOfBoundsException in mysql binlog read phase (#911)
- [mongodb][tests] Track MongoDBITCase hang in azure (#881)
- [docs] Update the supported flink version mapping (#885)
- [common] Fix emitDelay and fetchDelay metrics to be 0L during snapshot phase (#878)
- [hotfix][docs][mongodb] Update supported mongodb versions in README.md
- [docs][sql-server] Add sqlserver-cdc connector tutorial (#687) This closes #687.
- [common] Use ResolvedSchema instead of deprecated TableSchema API (#782)
- [docs][sqlserver] Add SqlServer CDC connector document (#705)
- [doc][sqlserver] Add SqlServer English tutorial document. (#693)
- [common] Make Flink CDC Compatible with Flink 1.14
- [build] Bump flink version to 1.13.5
- [docs][oracle] Add guidance about oracle CDB database (#792)
- [docs] Add connection.pool.size and connect.max-retries option in mysql document (#857)
- [sqlserver] Add e2e tests for sqlserver cdc connector (#837)
- [hotfix][mysql] Rename method from dispatchHighWatermark to dispatchBinlogEndEvent (#810) This closes #810.
- [mysql] Use READ type for all records in snapshot reading (#842)
- [mysql] Rewrite the newly added table tests to cover more cases (#777) This closes #777.
- [mysql] Improve newly added tables process
- [mysql] Support capture newly added tables for existed pipeline
- [docs] Fix typo in mongodb-cdc.md (#836)
- [mysql] add test to cover the datatype GeomCollection that's synonymous with GeometryCollection for mysql8 (#822)
- [sqlserver] Add SqlServer cdc connector (#662)
- [build] Remove redundant maven enforce plugin (#794)
- [hotfix] Fix MySqlSource comment typo (#793)
- [mysql] Support MySQL 5.6 (#396)
- [build] Switch to Log4j 2 by default (#791)
- [oracle] Fix Oracle numeric types conversion (#560)
- [hotfix][mysql] Fix message timestamp of snapshot record to 0L (#752)
- [docs] Translate the Chinese FAQ document to English (#737)
- [docs] Remove the fixed version of website (#730)
- [hotfix][mysql] Set minimum connection pool size to 1 (#692)
- [build] Bump log4j2 version to 2.16.0 (#714)
- [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
- [docs] Add FAQ for tableList option usage in DataStream API (#634)
- [mongodb] Support to connect MongoDB without authentication (#642)
- [hotfix] Fix the parameter typo in java doc (#651)
- [mysql] Add a method to get MySqlSourceConfigFactory from MySqlSource (#648)
- [mysql] Set default driver class name for PooledDataSource (#659)
- [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
- [docs] Improve column name note in oracle cdc document. (#625)
- [docs] Add Flink cdc eco-system picture (#623)
- [hotfix][docs] Fix typo in oracle-cdc.md (#613)
- [docs] Add supported Flink versions for Flink CDC 2.1(#611)
- [build] Bump version to 2.2-SNAPSHOT
Download
- flink-sql-connector-mongodb-cdc-2.2.0.jar
- flink-sql-connector-mysql-cdc-2.2.0.jar
- flink-sql-connector-oceanbase-cdc-2.2.0.jar
- flink-sql-connector-oracle-cdc-2.2.0.jar
- flink-sql-connector-postgres-cdc-2.2.0.jar
- flink-sql-connector-sqlserver-cdc-2.2.0.jar
- flink-sql-connector-tidb-cdc-2.2.0.jar
Contributor List (sorted alphabetically)
- Dian Qi
- He Wang
- Jark Wu
- Jiabao Sun
- Jingsong Lee
- Junwang Zhao
- Konstantin Knauf
- Leonard Xu
- Mustang Kong
- Paul Zhang
- Qingsheng Ren
- Sergey Nuyanzin
- Shengkai
- Tuple
- Xianxun Ye
- Xuebing Li
- Zongwen Li
- camelusluo
- eastfisher
- ehui
- fangpc
- gongzhongqiang
- hele.kc
- hiscat
- luoyuxia
- paul8263
- skylines
- szgeqin
- taox
- tsreaper
- wangxiaojing
- wysstartgo
- yexianxun
- 元组
Release-2.1.1
Release Notes
Improvements and Bug fixes
- [docs] Remove the fixed version of website (#730)
- [hotfix][mysql] Set minimum connection pool size to 1 (#692)
- [build] Bump log4j2 version to 2.16.0 (#714) Note: This project only uses log4j2 in test code and won't be influenced by log4shell vulnerability
- [build] Remove override definition of maven-surefire-plugin in connectors pom (#673)
- [docs] Add FAQ for tableList option usage in DataStream API (#634)
- [mongodb] Support to connect MongoDB without authentication (#642)
- [hotfix] Fix the parameter typo in java doc (#651)
- [mysql] Set default driver class name for PooledDataSource (#659) (#661)
- [mysql] Fix there is duplicate records outputted when keep updating table during snapshot read (#592)
- [docs] Improve column name note in oracle cdc document (#633)
- [docs] Add Flink cdc eco-system picture (#624)
- [hotfix][docs] Fix typo in oracle-cdc.md(#616)
- [docs] Add supported Flink versions for Flink CDC 2.1(#608)
Download
- flink-sql-connector-mysql-cdc-2.1.1.jar
- flink-sql-connector-postgres-cdc-2.1.1.jar
- flink-sql-connector-oracle-cdc-2.1.1.jar
- flink-sql-connector-mongodb-cdc-2.1.1.jar
- flink-format-changelog-json-2.1.1.jar