[HUDI-4283] The memory cost in HoodieMergeHandle is increasing as the size of base file

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Abandoned
• Component/s: archiving, compaction, performance
• Assignee: sherhomhuang
• Reporter: sherhomhuang
• Created: 19/Jun/22 14:50
• Updated: 20/Aug/22 01:57
• Resolved: 20/Aug/22 01:57

Description

Suppose a situation, there is a large history data in one dimension table. All files are in base path. There are few changes in the table, but for the large num of history data. It costs a large memory in HoodieMergeHandle, because it will cache writtenRecordKeys in memory.  

Issue Links

• links to: GitHub Pull Request #6030