[SPARK-11929] spark-shell log level customization is lost if user provides a log4j.properties file

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core, Spark Shell
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 23/Nov/15 17:33
• Updated: 17/May/16 11:18
• Resolved: 24/Nov/15 21:09

Description

Logging.scala has code that defines the default log level for the spark-shell to WARN, to avoid lots of noise in the output.

But if a user provides a log4j.properies file in the Spark configuration, that customization is lost. That means that without a log4j.properties, there are two different configurations (one for regular apps, one for the shell). But if you have a custom file, you lose the ability to easily differentiate between those two, and you're stuck with a single config for both.

It would be nice to allow different configurations also in the second case.

Issue Links

• links to: [Github] Pull Request #9816 (vanzin)