- The master have an hfile cleaner thread (that is responsible for cleaning the /hbase/.archive dir)
- if the family/region/family directory is empty the cleaner removes it
- The master can archive files (from another thread, e.g. DeleteTableHandler)
- The region can archive files (from another server/process, e.g. compaction)
The simplified file archiving code looks like this:
Since there's no synchronization between HFileArchiver.resolveAndArchive() and the cleaner run (different process, thread, ...) you can end up in the situation where you are moving something in a directory that doesn't exists.
The bad thing of deleting the file without archiving is that if you've a snapshot that relies on the file to be present, or you've a clone table that relies on that file is that you're losing data.
- Create a ZooKeeper lock, to notify the master ("Hey I'm archiving something, wait a bit")
- Add a RS -> Master call to let the master removes files and avoid this kind of situations
- Avoid to remove empty directories from the archive if the table exists or is not disabled
- Add a try catch around the fs.rename
The last one, the easiest one, looks like: