[HUDI-1628] [Umbrella] Improve data locality during ingestion

• Type: Epic
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: writer-core
• Labels: hudi-umbrellas
• Story Points: 0
• Epic Name: Improve data locality during ingestion
• Assignee: Ethan Guo
• Reporter: satish
• Created: 18/Feb/21 20:48
• Updated: 21/Dec/23 15:33

Description

Today the upsert partitioner does the file sizing/bin-packing etc for
inserts and then sends some inserts over to existing file groups to
maintain file size.
We can abstract all of this into strategies and some kind of pipeline
abstractions and have it also consider "affinity" to an existing file group
based
on say information stored in the metadata table?

See http://mail-archives.apache.org/mod_mbox/hudi-dev/202102.mbox/browser
for more details