[HUDI-2496] Inserts are precombined even with dedup disabled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: writer-core
• Labels: pull-request-available sev:critical
• Assignee: Helias Antoniou
• Reporter: Sagar Sumit
• Created: 28/Sep/21 03:24
• Updated: 03/Nov/21 11:21
• Resolved: 03/Nov/21 11:20

Description

Original GH issue https://github.com/apache/hudi/issues/3709

Test case by xushiyan : https://github.com/apache/hudi/pull/3723/files

RCA by shivnarayan :

Within HoodieMergeHandle, we use a hashmap to store incoming records, where keys are record keys.
and so, if you see 1st batch, duplicates would remain intact. but wrt 2nd batch, only unique records are considered and later concatenated w/ 1st batch.
https://github.com/apache/hudi/blob/36be28712196ff4427c41b0aa885c7fcd7356d7f/hudi-[]…-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

Issue Links

• links to: GitHub Pull Request #3740