[HUDI-6352] KEEP_LATEST_BY_HOURS should consider modified time instead of commit time while setting earliestCommitToRetain value

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: cleaning, table-service
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Surya Prasanna Yalla
• Created: 10/Jun/23 19:31
• Updated: 18/Sep/24 01:18

Description

In CleanPlanner, KEEP_LATEST_BY_HOURS is setting earliestCommitToRetain value by consider timestamp directly, this will introduce bug if there are out of order commits where commit with lower timestamp is completed much later than commits with higher timestamps.

This policy's implementation needs to be revisit.

It should basically store the timestamp until which it cleaned let this be t1. Next cleaner instant should consider all the partitions and files that are modified from the point of t1 until (currentime-x) hours. Whichever files are not valid they should be removed.

Issue Links

• links to: GitHub Pull Request #9226