[SPARK-2007] Spark on YARN picks up hadoop log4j.properties even if SPARK_LOG4J_CONF is set

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Component/s: YARN
• Assignee: Unassigned
• Reporter: Issac Buenrostro
• Created: 03/Jun/14 21:12
• Updated: 25/Nov/14 12:44
• Resolved: 25/Nov/14 12:44

Description

When running Spark on YARN, with SPARK_LOG4J_CONF environment variable set, containers might still pick up a different log4j configuration.

spark-yarn will create a local resource in YARN called log4j.properties, which is included in the classpath. However, there might be other log4j.properties files in the classpath which might be chosen instead.