[SPARK-2913] Spark's log4j.properties should always appear ahead of Hadoop's on classpath

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.0.0, 1.0.2, 1.1.0
• Component/s: Deploy
• Labels: bulk-closed
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 08/Aug/14 00:02
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

In the current compute-classpath scripts, the Hadoop conf directory may appear before Spark's conf directory in the computed classpath. This leads to Hadoop's log4j.properties being used instead of Spark's, preventing users from easily changing Spark's logging settings.

To fix this, we should add a new classpath entry for Spark's log4j.properties file.

Issue Links

• is duplicated by: SPARK-2027 spark-ec2 puts Hadoop's log4j ahead of Spark's in classpath
• is duplicated by: SPARK-4997 Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)
• is related to: SPARK-2858 Default log4j configuration no longer seems to work
• links to: [Github] Pull Request #1844 (JoshRosen)