[HUDI-3614] Fix Flink/JavaDeleteHelper deduplication logic

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Component/s: code-quality, writer-core
• Labels: pull-request-available
• Epic Link: Restructuring hudi client module for multi engine support
• Assignee: liujinhui
• Reporter: Shiyan Xu

Reviewers

Ethan Guo

• Created: 14/Mar/22 01:25
• Updated: 11/Nov/23 07:15

Description

The logic in org.apache.hudi.table.action.commit.FlinkDeleteHelper#deduplicateKeys does not dedup correctly. 

 

Flink/JavaDeleteHelper should merge with HoodieDeleteHelper and change API to use HoodieData. The dedup logic will be fixed along the way. 

Have dedicated test class for HoodieDeleteHelper will be great.

Issue Links

• Discovered while testing: HUDI-2439 Replace RDD with HoodieData in HoodieSparkTable and commit executors
• links to: GitHub Pull Request #5087