[MAPREDUCE-1700] User supplied dependencies may conflict with MapReduce system JARs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.3-alpha
• Component/s: task
• Hadoop Flags: Reviewed
• Assignee: Thomas White
• Reporter: Thomas White
• Created: 14/Apr/10 04:10
• Updated: 09/Dec/16 03:31
• Resolved: 09/Jan/13 16:24

Description

If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported here.

The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this.

Issue Links

• depends upon: YARN-286 Add a YARN ApplicationClassLoader
• is related to: PIG-3043 Modify the UrlClassloader in PigContext so that classes from the same classloader are used first instead of the parent
• relates to: MAPREDUCE-1938 Ability for having user's classes take precedence over the system classes for tasks' classpath
• relates to: HADOOP-10893 isolated classloader on the client side
• relates to: MAPREDUCE-5802 Provide an inclusion list mapreduce.job.classloader.job.classes for the class loading isolation