[HUDI-5679] Deduplication in row-writing bulk-insert ends up w/ OOMs

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: spark, spark-sql
• Epic Link: Hudi Spark Datasource
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 01/Feb/23 17:23
• Updated: 07/Feb/23 23:43
• Resolved: 07/Feb/23 23:43

Description

I'm running a small (1Gb) scale benchmark on a fat cluster (50Gb memory) and am still somehow getting OOMs when deduplication is enabled in row-writing bulk-insert path in CTAS

Issue Links

• is fixed by: HUDI-5684 Fix CTAS to make combine-on-insert configurable