[HBASE-21070] SnapshotFileCache won't update for snapshots stored in S3

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.1.1, 1.4.7
• Fix Version/s: 3.0.0-alpha-1, 1.5.0, 2.3.0
• Component/s: snapshots
• Labels: FSRedo s3
• Hadoop Flags: Reviewed
• Assignee: Zach York
• Reporter: Zach York
• Created: 17/Aug/18 22:41
• Updated: 07/May/19 21:29
• Resolved: 07/May/19 00:20

Description

The SnapshotFileCache depends on last modified time to determine whether to update the Snapshot HFile cache. However, in S3, real 'folders' don't exist. S3 filesystems create a dummy file in place of a folder, but the dummy file last modified time is not updated when files are changed 'under' it. This means that the SnapshotFileCache doesn't pick up new snapshot HFiles and these files aren't removed from the HFileCleaner and can be eligible for deletion.

 

My patch removes the lastmodified assumption.

Issue Links

• is related to: HBASE-20429 Support for mixed or write-heavy workloads on non-HDFS filesystems
• relates to: HBASE-22190 SnapshotFileCache may fail to load the correct snapshot file list when there is an on-going snapshot operation
• links to: GitHub Pull Request #209
• links to: Review Board (master)