[HUDI-167] Rethink Metadata archiving in Hoodie

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Common Core
• Assignee: Vinoth Chandar
• Reporter: Balaji Varadarajan
• Created: 13/Jul/19 00:38
• Updated: 21/Oct/19 00:47
• Resolved: 21/Oct/19 00:47

Description

The current archiving logic is not cleanly designed and implemented. Recently, we ran into the issue https://issues.apache.org/jira/browse/HUDI-162?filter=-2 . One of the main factor for this is the archiving logic. Currently, the archiving logic retains last N commits for each action. This could result in gaps in the active timeline for certain actions when  considering all timeline for all actions.

 

We need to re-design the archiving policy