[FLINK-36813] MySQLCDC supports synchronization of specified fields

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: cdc-3.2.1
• Fix Version/s: cdc-3.3.0
• Component/s: Flink CDC
• Labels: CDC pull-request-available
• Assignee: Unassigned
• Reporter: Di Wu
• Created: Yesterday 03:08
• Updated: Yesterday 03:23

Description

Background
In some scenarios, MySQL synchronization only expects to synchronize specified fields instead of all fields in the table.
1. The user only has the permission for some fields in MySQL
2. The user has too many fields in a single table and only wants to synchronize some fields, for example, here https://github.com/apache/flink-cdc/discussions/3058

Current situation
For the incremental stage, you only need to configure the column.include.list property of debezium to support the synchronization of some fields in the incremental stage, refer to: https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-property-column-include-list

For the full snapshot stage, * is currently used in MySqlSnapshotSplitReadTask, refer to

if (isScanningData) {
return buildSelectWithRowLimits(
tableId, limitSize, "*", Optional.ofNullable(condition), Optional.empty()); 

 

Solution
We can refer to debezium [RelationalSnapshotChangeEventSource](https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/relational/RelationalSnapshotChangeEventSource.java#L752-L776), The user configures column.include.list, and then captures the specific columns in MySqlSnapshotSplitReadTask, and splices them when constructing Scan SQL.

Issue Links

• links to: GitHub Pull Request #3767