[HUDI-5338] Adjust coalesce behavior within "NONE" sort mode for bulk insert

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Labels: pull-request-available
• Assignee: Ethan Guo
• Reporter: Ethan Guo
• Created: 07/Dec/22 00:47
• Updated: 06/Feb/23 23:46
• Resolved: 13/Dec/22 21:52

Description

The NONE sort mode for bulk insert does coalesce for the input records or rows based on the shuffle parallelism of bulk insert (hoodie.bulkinsert.shuffle.parallelism) to reduce the parallelism. This could affect write latency if the cluster workers are not fully utilized due to reduced parallelism.

Issue Links

• contains: HUDI-3883 Bulk-insert w/ sort-mode "NONE" leads to file-sizing issues
• contains: HUDI-4261 OOM in bulk-insert when using "NONE" sort-mode for table w/ large # of partitions
• links to: GitHub Pull Request #7396