[HUDI-6155] Fix cleaner based on hours for earliest commit to retain

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: cleaning, table-service
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: sivabalan narayanan
• Created: 29/Apr/23 00:41
• Updated: 29/Mar/24 02:37

Description

When cleaner is based on hours, we estimate the earliest commit to retain based on current time zone and not UTC or the timezone used to generate the commit time. so, there could be some mis-calculations and lead to deleting additional slices. 

 

Ref: https://github.com/apache/hudi/blob/c6760772f8dc62eb44c45b022ed07858d895d804/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java#L511

 

else if (config.getCleanerPolicy() == HoodieCleaningPolicy.KEEP_LATEST_BY_HOURS) {
  Instant instant = Instant.now();
  ZonedDateTime currentDateTime = ZonedDateTime.ofInstant(instant, ZoneId.systemDefault());
  String earliestTimeToRetain = HoodieActiveTimeline.formatDate(Date.from(currentDateTime.minusHours(hoursRetained).toInstant()));
  earliestCommitToRetain = Option.fromJavaOptional(commitTimeline.getInstantsAsStream().filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(),
          HoodieTimeline.GREATER_THAN_OR_EQUALS, earliestTimeToRetain)).findFirst());
} 

 

 

Potential fixes:

• Fix the time based on time zone set in table config. 

• Fetch the latest completed commit and decide the earliest commit based on that.

Issue Links

• links to: GitHub Pull Request #8659