[HUDI-2535] Late arriving records and global index with partition path update set to true

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: writer-core
• Labels: sev:normal user-support-issues
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 08/Oct/21 14:31
• Updated: 31/Mar/23 20:06

Description

incase of a global index, we have a config to update partition path. if this is set to true, if there is an incoming record to a newer partition compared to whats in storage, older record will be deleted and new incoming record will be routed to new partition. 

But it could run into issues if new incoming is a late arriving record. Expected behavior is, old record is retained and new one is discarded it it has lower preCombine value. But in this case, we may not honor that. 

 

Issue Links

• is related to: HUDI-1508 Partition update with global index in MOR tables resulting in duplicate values during read optimized queries