[HUDI-1492] Enhance DeltaWriteStat with block level metadata correctly for storage schemes that support appends

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Won't Do
• Fix Version/s: 0.11.0
• Component/s: metadata
• Story Points: 2
• Epic Link: RFC-27 Multi Modal Indexing
• Assignee: Manoj Govindassamy
• Reporter: Vinoth Chandar
• Created: 25/Dec/20 22:14
• Updated: 28/Feb/22 08:24
• Resolved: 28/Feb/22 08:24

Description

Current implementation simply uses the

String pathWithPartition = hoodieWriteStat.getPath(); 

to write the metadata table. this is problematic, if the delta write was merely an append. and can technically add duplicate files into the metadata table 

(not sure if this is a problem per se. but filing a Jira to track and either close/fix ) 

 

Issue Links

• Is contained by: HUDI-3258 Support multiple metadata index partitions - bloom and column stats