[SPARK-16416] Logging in shutdown hook does not work properly with Log4j 2.x

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.2
• Fix Version/s: 2.1.0
• Component/s: Spark Core
• Assignee: Mikael Ståldal
• Reporter: Mikael Ståldal
• Created: 07/Jul/16 12:09
• Updated: 26/Jul/16 09:56
• Resolved: 24/Jul/16 10:16

Description

Spark registers some shutdown hooks, and they log messages during shutdown:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L58

Since the Logging trait creates SLF4J loggers lazily:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/Logging.scala#L47

a SLF4J logger is created during the execution of the shutdown hook.

This does not work when Log4j 2.x is used as SLF4J implementation:
https://issues.apache.org/jira/browse/LOG4J2-1222

Even though Log4j 2.6 handles this more gracefully than before, it still does emit a warning and will not be able to process the log message properly.

Proposed solution: make sure to eagerly create the SLF4J logger to be used in shutdown hooks when registering the hook.

Issue Links

• relates to: SPARK-6305 Add support for log4j 2.x to Spark
• links to: [Github] Pull Request #14320 (mikaelstaldal)