[SPARK-5358] spark.files.userClassPathFirst doesn't work correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.0, 1.2.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Matthew Whelan
• Created: 21/Jan/15 21:12
• Updated: 16/Feb/15 23:23
• Resolved: 16/Feb/15 23:23

Description

org.apache.spark.executor.ChildExecutorURLClassLoader#findClass delegates to two different classloaders: a parent-less URL classloader, and the parent classloader, as a fallback. The delegation is broken such that calling loadClass twice in succession with the same parameters will fail the second time.

The delegation to the userClassLoader calls findClass directly, which bypasses the classloader's cache. So userClassLoader will attempt to define the same class multiple times, throwing LinkageErrors after the first time.

The canonical way to change the default delegation scheme is to override loadClass, rather than just findClass. It also might be sufficient to have userClassLoader.findClass call super.loadClass.

Issue Links

• duplicates: SPARK-4877 userClassPathFirst doesn't handle user classes inheriting from parent
• duplicates: SPARK-1863 Allowing user jars to take precedence over Spark jars does not work as expected
• duplicates: SPARK-2996 Standalone and Yarn have different settings for adding the user classpath first
• is related to: SPARK-2996 Standalone and Yarn have different settings for adding the user classpath first
• relates to: SPARK-4739 spark.files.userClassPathFirst does not work in local[*] mode
• relates to: SPARK-4877 userClassPathFirst doesn't handle user classes inheriting from parent
• relates to: SPARK-1863 Allowing user jars to take precedence over Spark jars does not work as expected
• links to: [Github] Pull Request #4165 (MattWhelan)
• links to: [Github] Pull Request #4166 (MattWhelan)