[FLINK-19795] Fix Flink SQL throws exception when changelog source contains duplicate change events

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.2
• Fix Version/s: 1.12.0
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Jark Wu
• Reporter: jinxin
• Created: 25/Oct/20 02:38
• Updated: 27/Nov/20 14:22
• Resolved: 25/Nov/20 13:50

Description

We are using Canal to synchornize MySQL data into Kafka, the synchornization delivery is not exactly-once, so there might be dupcliate INSERT/UPDATE/DELETE messages for the same primary key. We are using 'connecotr' = 'kafka', 'format' = 'canal-json' to consume such topic. However, when appling TopN query on this created source table, the TopN operator will thrown exception: Caused by: java.lang.RuntimeException: Can not retract a non-existent record. This should never happen.

Issue Links

• fixes: FLINK-20205 CDC source shouldn't send UPDATE_BEFORE messages if the downstream doesn't need it
• links to: GitHub Pull Request #14160