[HUDI-7226] Clean by hour does not respect lastVersionBeforeEarliestCommitToRetain

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.14.1
• Component/s: cleaning
• Labels: pull-request-available
• Assignee: Timothy Brown Assign to me
• Reporter: Shiyan Xu
• Created: 12/Dec/23 18:13
• Updated: 26/Mar/24 07:03
• Resolved: 26/Mar/24 07:03

Description

org.apache.hudi.table.action.clean.CleanPlanner#getFilesToCleanKeepingLatestCommits(java.lang.String, int, org.apache.hudi.common.model.HoodieCleaningPolicy)

lastVersionBeforeEarliestCommitToRetain is not honored by KEEP_LATEST_BY_HOURS policy. This essentially makes cleaner to remove the file slice when it becomes non-latest. This could fail long-running queries in a race condition:

1. timeline contains a t0.deltacommit (not cleaned because it's latest)
2. a snapshot query starts and running
3. compaction runs and creates t1.commit
4. cleaner runs and remove t0 (because now t1.commit is the latest)
5. the query failed due to a log file belongs to t0.deltacommit is not found

Issue Links

• links to: GitHub Pull Request #10321