[HUDI-163] Updates to HoodieRecord are missing during MergeHandle with ExternalSpillableMap

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Unresolved
• Component/s: Common Core
• Assignee: omkar vinit joshi
• Reporter: Balaji Varadarajan
• Created: 01/Jul/19 22:19
• Updated: 30/Aug/19 16:25

Description

Copy of https://github.com/apache/incubator-hudi/issues/773

MergeHandle uses ExternalSpillableMap for storing all new inserts to a file. Today during merge handle if number of records which we are adding to a file are a lot then ExternalSpillableMap may end up writing it to disk and future updates done to in memory copy of HoodieRecord will not be reflected into on disk copy. This creates duplicate key issue if we are using HBaseIndex. Creating this issue to address this.

https://github.com/apache/incubator-hudi/blob/master/hoodie-client/src/main/java/com/uber/hoodie/io/HoodieMergeHandle.java#L211 Here we are updating HoodieRecord after adding to the map.

 

Issue Links

• Blocked: HUDI-170 Updating hoodie record before inserting it into ExternalSpillableMap