[FLINK-36081] Flink CDC MySQL source connector missing some columns data of newly added tables

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: cdc-3.1.1
• Fix Version/s: cdc-3.2.0, cdc-3.3.0
• Component/s: Flink CDC
• Labels: mysql-cdc-connector pull-request-available

Environment

jdk 11

flink 1.17

flinkcdc 3.0.0

Language

Java

• Assignee: Mingya Wang
• Reporter: Mingya Wang
• Created: 18/Aug/24 02:00
• Updated: 27/Aug/24 17:00
• Resolved: 21/Aug/24 02:41

Description

Problem Description:

When adding a new table, the Flink CDC MySQL source connector experiences missing data for some columns of the newly added table.

Reproduction Scenario:

1. Remove a table from a cdc job that is running normally, then start the job with resume functionality.
2. Perform a column addition operation on the removed table.
3. Add the table back to the job. The job continues to run without interruption upon table addition, but data for the newly added columns is missing in the synchronized data.

Cause Analysis:

The issue arises because the MySQL CDC Source maintains the table schema in state. When adding a new table, it recovers the schema from the previous state. Since the prior schema exists and represents the structure before the column addition, the MySQL CDC Source provides the downstream with data based on the schema cached in the state. Consequently, records outputted to downstream systems are missing the fields corresponding to the newly added columns.

Proposed Solution:

Upon removing a table from the cdc job, it is necessary to also correspondingly remove the table from the MySQLBinlogSplit.

Issue Links

• links to: GitHub Pull Request #3548