[HUDI-349] Make cleaner retention based on time period to account for higher deviations in ingestion runs

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: cleaning, writer-core
• Labels: core-flow-ds new-to-hudi pull-request-available sev:high
• Story Points: 1
• Assignee: sivabalan narayanan
• Reporter: Balaji Varadarajan
• Created: 19/Nov/19 17:39
• Updated: 22/Feb/22 13:43
• Resolved: 22/Feb/22 13:43

Description

Cleaner by commits is based on number of commits to be retained.  Ingestion time could vary across runs due to various factors. For providing a bound on the maximum running time for a query and for providing consistent retention period, it is better to use a retention config based on time (e:g 12h) 

Issue Links

• links to: GitHub Pull Request #3646