[HUDI-6252] Deduce archival config based on hours derived from cleaner hours based config

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: archiving, cleaning, table-service
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 23/May/23 13:57
• Updated: 29/Mar/24 02:37

Description

We have hours based policy/configs for cleaning, but archival only has num_commits based configs. So, as of now, user/dev has to ensure they both align well together. We should derive archival configs automatically from cleaner configs. 

 

for eg, 

if cleaner configs is 48 hours, 

we can add 10% more for archival min commits and 20% for max commits. 

So, archival window can be 53 hours -> 58 hours. we can expose configs for these percentage if need be.