[SPARK-25183] Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 22/Aug/18 00:56
• Updated: 31/Aug/18 06:46
• Resolved: 31/Aug/18 06:46

Description

Spark's HiveServer2 registers a shutdown hook with the JVM Runtime.addShutdownHook() which can happen in parallel with the ShutdownHookManager sequence of spark & Hadoop, which operate the shutdowns in an ordered sequence.

This has some risks

• FS shutdown before rename of logs completes, SPARK-6933
• Delays of rename on object stores may block FS close operation, which, on clusters with timeouts hooks (HADOOP-12950) of FileSystem.closeAll() can force a kill of that shutdown hook and other problems.

General outcome: logs aren't present.

Proposed fix:

• register hook with org.apache.spark.util.ShutdownHookManager
• HADOOP-15679 to make shutdown wait time configurable, so O(data) renames don't trigger timeouts.

Issue Links

• relates to: HADOOP-15679 ShutdownHookManager shutdown time needs to be configurable & extended
• relates to: SPARK-6014 java.io.IOException: Filesystem is thrown when ctrl+c or ctrl+d spark-sql on YARN
• relates to: SPARK-6933 Thrift Server couldn't strip .inprogress suffix after being stopped
• relates to: HADOOP-12950 ShutdownHookManager should have a timeout for each of the Registered shutdown hook
• links to: [Github] Pull Request #22186 (steveloughran)