[HUDI-7104] Cleaner could miss to clean up some files w/ savepoint interplay

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0, 1.0.0
• Component/s: cleaning, savepoint, table-service
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 16/Nov/23 01:52
• Updated: 07/Jun/24 02:21
• Resolved: 07/Jun/24 02:21

Description

Lets say partitioning is day based and is based on created date. So, older partitions generally does not get any new data after few days. 

 

Lets say we have savepoints added to a day and later removed. 

day 1: cleaned up. 

day2: savepoint added. and so cleaner ignord. 

day3: cleaned up 

day4: earliest commit to retain based on cleaner configs. 

 

So, w/ this table/timeline state, if we remove the savepointed commit, data pertaining to day2 will never be cleaned by the cleaner since its lesser than the earliest commit to retain. 

 

Issue Links

• links to: GitHub Pull Request #10651