[SPARK-14703] Spark uses SLF4J, but actually relies quite heavily on Log4J

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.6.0
• Component/s: Spark Core, YARN
• Labels: log4j logback logging slf4j

Environment

1.6.0-cdh5.7.0, logback 1.1.3, yarn

• Flags: Patch
• Assignee: Unassigned
• Reporter: Matthew Byng-Maddick
• Created: 18/Apr/16 15:36
• Updated: 03/Nov/17 21:21
• Resolved: 17/Aug/16 08:21

Description

We've built a version of Hadoop CDH-5.7.0 in house with logback as the SLF4J provider, in order to send hadoop logs straight to logstash (to handle with logstash/elasticsearch), on top of our existing use of the logback backend.

In trying to start spark-shell I discovered several points where the fact that we weren't quite using a real L4J caused the sc not to be created or the YARN module not to exist. There are many more places where we should probably be wrapping the logging more sensibly, but I have a basic patch that fixes some of the worst offenders (at least the ones that stop the sparkContext being created properly).

I'm prepared to accept that this is not a good solution and there probably needs to be some sort of better wrapper, perhaps in the Logging.scala class which handles this properly.