[HBASE-26718] HFileArchiver can remove referenced StoreFiles from the archive

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.95.0, 1.0.0, 3.0.0-alpha-1, 2.0.0
• Fix Version/s: 2.5.0, 3.0.0-alpha-3, 2.4.12
• Component/s: Compaction, HFile, snapshots
• Hadoop Flags: Reviewed
• Assignee: David Manning
• Reporter: David Manning
• Created: 28/Jan/22 00:46
• Updated: 29/Mar/22 20:22
• Resolved: 28/Mar/22 18:58

Description

There is a comment in HFileArchiver#resolveAndArchiveFile:

// if the file already exists in the archive, move that one to a timestamped backup. This is a
// really, really unlikely situtation, where we get the same name for the existing file, but
// is included just for that 1 in trillion chance.

In reality, we did encounter this frequently enough to cause problems. More details will be included and linked in a separate issue.

But regardless of how we get into this situation, we can consider a different approach to solving it. If we assume store files are immutable, and a store file with the same name and location already exists in the archive, then it can be safer to assume the file was already archived successfully, and react accordingly.

Issue Links

• relates to: HBASE-26722 Snapshot is corrupted due to interaction between move, warmupRegion, compaction, and HFileArchiver
• relates to: HBASE-26726 Allow disable of region warmup before graceful move
• requires: HBASE-5547 Don't delete HFiles when in "backup mode"
• links to: GitHub Pull Request #4274