[HBASE-28287] MOB HFiles are expired earlier than their reference data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.6.0, 2.4.18, 2.5.8, 3.0.0-beta-2
• Component/s: mob
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: WenjingLiu
• Reporter: WenjingLiu
• Created: 25/Dec/23 14:28
• Updated: 15/Jan/24 03:21
• Resolved: 13/Jan/24 09:59

Description

I have observed that mob HFiles are expired earlier than their reference data. Upon reviewing the relevant code, it was observed that the standard expired timestamp is not accurately set. The issue arises from the fact that if we do not set the calendar to the MILLISECOND level, a three-digit random number is appended to the end of the expired timestamp, leading to inaccuracies.

For instance, if we write mob data to a Time-To-Live (1 day) table at 22:00 on 12/20/2023, and the ExpiredMobFileCleaner begins to work at 20:00 on 12/21/2023, the timestamp of the mob HFile will be parsed as 1703001600000, and the StandardExpiredTS is calculated as 1703001600314 with the three-digit random number as MILLISECOND. Consequently, the StandardExpiredTS >= MobHfileTS, thus indicating that the mob HFile should be expired. However, it is observed that this mob HFile only exists for 22 hours, which is less than the specified one-day-time threshold.

Issue Links

• links to: GitHub Pull Request #5597
• links to: GitHub Pull Request #5599