[HIVE-13429] Tool to remove dangling scratch dir

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.1.0
• Target Version/s: 1.3.0, 2.1.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 05/Apr/16 22:43
• Updated: 04/May/23 17:30
• Resolved: 11/Apr/16 22:14

Description

We have seen in some cases, user will leave the scratch dir behind, and eventually eat out hdfs storage. This could happen when vm restarts and leave no chance for Hive to run shutdown hook. This is applicable for both HiveCli and HiveServer2. Here we provide an external tool to clear dead scratch dir as needed.

We need a way to identify which scratch dir is in use. We will rely on HDFS write lock for that. Here is how HDFS write lock works:
1. A HDFS client open HDFS file for write and only close at the time of shutdown
2. Cleanup process can try to open HDFS file for write. If the client holding this file is still running, we will get exception. Otherwise, we know the client is dead
3. If the HDFS client dies without closing the HDFS file, NN will reclaim the lease after 10 min, ie, the HDFS file hold by the dead client is writable again after 10 min

So here is how we remove dangling scratch directory in Hive:
1. HiveCli/HiveServer2 opens a well-named lock file in scratch directory and only close it when we about to drop scratch directory
2. A command line tool cleardanglingscratchdir will check every scratch directory and try open the lock file for write. If it does not get exception, meaning the owner is dead and we can safely remove the scratch directory
3. The 10 min window means it is possible a HiveCli/HiveServer2 is dead but we still cannot reclaim the scratch directory for another 10 min. But this should be tolerable

Issue Links

• is related to: HIVE-13498 cleardanglingscratchdir does not work if scratchdir is not on defaultFs
• is related to: HIVE-14690 Query fail when hive.exec.parallel=true, with conflicting session dir
• is related to: HIVE-27317 Temporary (local) session files cleanup improvements
• relates to: HIVE-10415 hive.start.cleanup.scratchdir configuration is not taking effect
• relates to: HIVE-13513 cleardanglingscratchdir does not work in some version of HDFS
• relates to: HIVE-13514 TestClearDanglingScratchDir fail on branch-1
• relates to: HIVE-13551 Make cleardanglingscratchdir work on Windows
• relates to: HIVE-2181 Clean up the scratch.dir (tmp/hive-root) while restarting Hive server.
• links to: RB