[SPARK-939] Allow user jars to take precedence over Spark jars, if desired

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Labels: starter
• Assignee: Holden Karau
• Reporter: Patrick Wendell
• Created: 19/Oct/13 14:54
• Updated: 23/Nov/15 13:14
• Resolved: 09/Apr/14 05:31

Description

Sometimes a user may want to include their own version of a jar that spark itself uses. For example, if their code requires a newer version of that jar than Spark offers. It would be good to have an option to give the users dependencies precedence over Spark. This options should be disabled by default, since it could lead to some odd behavior (e.g. parts of Spark not working). But I think we should have it.

From an implementation perspective, this would require modifying the way we do class loading inside of an Executor. The default behavior of the URLClassLoader is to delegate to it's parent first and, if that fails, to find a class locally. We want to have the opposite behavior. This is sometimes referred to as "parent-last" (as opposed to "parent-first") class loading precedence. There is an example of how to do this here:

http://stackoverflow.com/questions/5445511/how-do-i-create-a-parent-last-child-first-classloader-in-java-or-how-to-overr

We should write a similar class which can encapsulate a URL classloader and change the delegation order. Or if possible, maybe we could find a more elegant way to do this. See relevant discussion on the user list here:

https://groups.google.com/forum/#!topic/spark-users/b278DW3e38g

Also see the corresponding option in Hadoop:
https://issues.apache.org/jira/browse/MAPREDUCE-4521

Some other relevant Hadoop JIRA's:
https://issues.apache.org/jira/browse/MAPREDUCE-1700
https://issues.apache.org/jira/browse/MAPREDUCE-1938