[FLINK-36541] Occasional met commit conflict problem in PaimonSink.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: cdc-3.2.0
• Fix Version/s: cdc-3.3.0, cdc-3.2.1
• Component/s: Flink CDC
• Labels: pull-request-available
• Assignee: Yanquan Lv
• Reporter: Yanquan Lv
• Created: 15/Oct/24 12:37
• Updated: 19/Nov/24 03:46
• Resolved: 19/Nov/24 03:46

Description

When writing records to Paimon, we occasional met commit conflict problem even if Parallelism is 1.

It turns out that PaimonWriter preparecommit Committable with a checkpointId which is always 1, as the following code:
https://github.com/apache/flink-cdc/blob/a1781f432d906fa2a864642a5f74ac5bdc963d9c/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/PaimonWriter.java#L164-L178

This is actually an incorrect usage, we must pass in an increasing ID to ensure the correctness of the snapshot information.

Issue Links

• links to: GitHub Pull Request #3652
• links to: GitHub Pull Request #3727