Spark's HiveServer2 registers a shutdown hook with the JVM Runtime.addShutdownHook() which can happen in parallel with the ShutdownHookManager sequence of spark & Hadoop, which operate the shutdowns in an ordered sequence.
This has some risks
- FS shutdown before rename of logs completes,
- Delays of rename on object stores may block FS close operation, which, on clusters with timeouts hooks (
HADOOP-12950) of FileSystem.closeAll() can force a kill of that shutdown hook and other problems.
General outcome: logs aren't present.
- register hook with org.apache.spark.util.ShutdownHookManager
HADOOP-15679to make shutdown wait time configurable, so O(data) renames don't trigger timeouts.