[OOZIE-2714] Detect conflicting resources during class loading

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: core
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 19/Oct/16 09:54
• Updated: 30/Jan/18 18:26

Description

There are a bunch of issues in Oozie which are related to class loading.

The main problem is that the classpath is constructed in a way which is very specific to Oozie:

• Hadoop lib jars
• Sharelib jars
• User-defined jars

Sometimes there is a conflict between sharelib and hadoop lib version. Also, users can add their own jars which sometimes contain a different version of popular libraries such as Guava, Apache commons, etc.

We should be able to detect these conflicts and print exact error message so that Oozie users can take appropriate actions to resolve the problem.

A possible approach is the following:

• start the execution of an action on a different thread
• replace the thread's context classloader with a classloader which can detect conflicts
• when the JVM invokes the loadClass() method of the classloader, it scans through the jars (which are available as URLClassPath objects). If it finds the given resource in at least two jars, it can do different things depending on the setup:
  • throws an error immediately, mentioning the conflicting jars (this is probably too strict - but still an option)
  • loads the two resource into a byte array and compares them - it only throws an error if there is difference
  • compares the jars but only emits an error message if there is a conflict
  • something else (user defined action?)

Implementing such a classloader is not difficult and would greatly enhance the supportability of Oozie. It could work in multiple modes depending on the setup - perhaps being able to control it from a workflow config is desirable. If there's any problem, we should be able to turn it off completely, too.