Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-35243][cdc] Support table schema change events & PreSchema backfill #3296

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented May 6, 2024

This PR closes FLINK-35243:

  • Supports AlterColumnCommentEvent, AlterTableCommentEvent, DropTableEvent, RenameTableEvent, and TruncateTableEvent in pipeline framework
  • Adds SchemaChangeEventWithPreSchema interface, allowing SchemaRegistry backfilling schema before change payload
  • Fixed MySQL pipeline source doesn't capture ALTER TABLE ... MODIFY COLUMN DDL
  • Adds corresponding test cases

@yuxiqian
Copy link
Contributor Author

yuxiqian commented May 8, 2024

@PatrickRen @ruanhang1993 PTAL

@melin
Copy link

melin commented May 10, 2024

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

@yuxiqian
Copy link
Contributor Author

yuxiqian commented May 10, 2024

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

Hi @melin, thanks for your comments. It's a common use case to apply some "safe" schema change events (like AddColumnEvent) and ignore "dangerous" events (like Truncate / Drop tables). So we're designing a new feature allowing users to configure schema change behaviour for each type of schema change events in FLINK-35242. Also, we're trying to expose interfaces allowing Sink connectors to report their supported kind of schema change events.

This PR is still in early draft stage, so feel free to leave your comments if you have any other concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants