[SPARK-9826] Cannot use custom classes in log4j.properties

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: Spark Core
• Labels: regression
• Target Version/s: 1.4.2, 1.5.0
• External issue URL: http://apache-spark-user-list.1001560.n3.nabble.com/log4j-custom-appender-ClassNotFoundException-with-spark-1-4-1-tt24159.html
• Assignee: Michel Lemay
• Reporter: Michel Lemay
• Created: 11/Aug/15 16:09
• Updated: 14/Aug/15 11:19
• Resolved: 13/Aug/15 00:40

Description

log4j is initialized before spark class loader is set on the thread context.
Therefore it cannot use classes embedded in fat-jars submitted to spark.

While parsing arguments, spark calls methods on Utils class and triggers ShutdownHookManager static initialization. This then leads to log4j being initialized before spark gets the chance to specify custom class MutableURLClassLoader on the thread context.

See detailed explanation here:
http://apache-spark-user-list.1001560.n3.nabble.com/log4j-custom-appender-ClassNotFoundException-with-spark-1-4-1-tt24159.html

Issue Links

• links to: [Github] Pull Request #8109 (michellemay)