[HBASE-18399] Files in a snapshot can go missing even after the snapshot is taken successfully

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: regionserver, Scanners
• Assignee: Unassigned
• Reporter: Ashu Pachauri
• Created: 17/Jul/17 21:42
• Updated: 01/Jul/22 20:27
• Resolved: 11/Aug/17 20:03

Description

Files missing after the snapshot is taken (only applicable when the TTL for the TimeToLiveHFileCleaner is small, like the default 5 mins)

• SnapshotManifest#addRegion visits store_file_A, but is yet to write it to the manifest.
• store_file_A is marked as compacted away and HFileArchiver moves the file to archive.
• HFileCleaner comes in and sees the store_file_A in archive. It adds the file to the list of files that might need to be cleaned up.
• HFileCleaner's SnapshotHFileCleaner plugin is kicked in.
• SnapshotFileCache#getUnreferencedFiles also says that store_file_A is unreferenced and should be cleaned up (It has not yet been written to the manifest).
• SnapshotHFileCleaner is still going through rest of the files in archive.
• store_file_A reference is created and written to snapshot manifest.
• Snapshot verification runs and sees the store_file_A is present in archive, and thus the verification passes.
• Now, the SnapshotHFileCleaner finishes and TimeToLiveHFileCleaner is triggered. If TTL has passed since the store_file_A was moved to archive (SnapshotHFileCleaner could take easily several minutes to go through rest of the files), the TimeToLiveHFileCleaner also marks the file as deletable.
• Since all cleaner plugins marked file as deletable, the store_file_A is deleted.

Issue Links

• is related to: HBASE-18398 Snapshot operation fails with FileNotFoundException