[HADOOP-15679] ShutdownHookManager shutdown time needs to be configurable & extended

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0
• Fix Version/s: 2.9.2, 3.0.4, 3.1.2, 2.8.6
• Component/s: util
• Target Version/s: 2.10.0, 2.9.2, 3.0.3, 2.8.5
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 16/Aug/18 05:06
• Updated: 24/Jul/19 21:53
• Resolved: 11/Oct/18 18:59

Description

HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging shutdowns. But the timeout is too short for applications where a large flush of data is needed on shutdown.

A key example of this is Spark apps which save their history to object stores, where the file close() call triggers an upload of the final local cached block of data (could be 32+MB), and then execute the final mutipart commit.

Proposed

1. make the default sleep time 30s, not 10s
2. make it configurable with a time duration property (with minimum time of 1s.?)

Issue Links

• is caused by: HADOOP-12950 ShutdownHookManager should have a timeout for each of the Registered shutdown hook
• is related to: HADOOP-16341 ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
• is related to: HADOOP-16461 Regression: FileSystem cache lock parses XML within the lock
• is related to: SPARK-24523 InterruptedException when closing SparkContext
• is related to: SPARK-25183 Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise