Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-654

Provide a way to use 'uber' jars with Oozie MR actions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.2
    • Component/s: None
    • Labels:
      None

      Description

      Right now, say if you have a custom MR code in a jar that has a lib/ folder inside which carries more dependent jars (a structure known as 'uber' jars), and you submit the job via a regular 'hadoop jar' command, these lib/.jars get picked up by the framework because the supplied jar is specified explicitly via conf.setJarByClass or conf.setJar. That is, if this user uber jar goes to the JT as the mapred.jar, then it is handled by the framework properly and the lib/.jars are all considered and placed on the classpath.

      Distributed cache jars do not have this effect, and that is cause the MR framework does not consider them as uber jars and does not extract and use their internal lib/ directories.

      We should have a way in oozie to let users promote one of their jars as uber jars, as an option.

      Proposal: Have an optional oozie-prefixed config, or an optional element in the MR action XML, that lets user specify what class should be loaded to be set as setJarByClass(...). This will have to be a class available in the higher level of the uber jar (not under lib/) but can be any class inside the targeted jar really (just not from a jar under lib/). We then set this as jobConf.setJarByClass(loadedCls), and then run the job.

      Thoughts?

        Attachments

        1. OOZIE-654.patch
          33 kB
          Robert Kanter
        2. OOZIE-654.patch
          27 kB
          Robert Kanter
        3. OOZIE-654.patch
          17 kB
          Robert Kanter

          Issue Links

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                qwertymaniac Harsh J
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: