Hadoop Common
  1. Hadoop Common
  2. HADOOP-189

Add job jar lib, classes, etc. to CLASSPATH when in standalone mode

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.

        Activity

        Hide
        Doug Cutting added a comment -

        Okay, this time it works in both standalone and distributed modes.

        Show
        Doug Cutting added a comment - Okay, this time it works in both standalone and distributed modes.
        Hide
        Doug Cutting added a comment -

        I just reverted this, since it broke distributed operation.

        Show
        Doug Cutting added a comment - I just reverted this, since it broke distributed operation.
        Hide
        Doug Cutting added a comment -

        I just committed this.

        Show
        Doug Cutting added a comment - I just committed this.
        Hide
        stack added a comment -

        Ok. This patch works really well (Looks to be spin on unbundling before invocation of Main-Class). Thanks for all your work on this.

        Show
        stack added a comment - Ok. This patch works really well (Looks to be spin on unbundling before invocation of Main-Class). Thanks for all your work on this.
        Hide
        Doug Cutting added a comment -

        Here's a patch I've actually tested!

        Show
        Doug Cutting added a comment - Here's a patch I've actually tested!
        Hide
        Doug Cutting added a comment -

        > Tracing, I see that any new class will have null for ClassLoader

        For this patch to work your, e.g., Mapper class needs to only be in the job jar, not also on the classpath built by bin/hadoop. If you are able to instantiate classes in the job jar, and they show up with the bootstrap class loader, then that means the jar file is somehow on the bootstrap classpath, which it should not be after the changes to 'bin/hadoop jar' made in the patch. Is that how you are starting things?

        Another potential problem could be that the URLClassloader is not correctly constructed. The urls added must be either a jar file or end in slashes. It might be useful to log the urls added to the path in LocalJobRunner to check this.

        Show
        Doug Cutting added a comment - > Tracing, I see that any new class will have null for ClassLoader For this patch to work your, e.g., Mapper class needs to only be in the job jar, not also on the classpath built by bin/hadoop. If you are able to instantiate classes in the job jar, and they show up with the bootstrap class loader, then that means the jar file is somehow on the bootstrap classpath, which it should not be after the changes to 'bin/hadoop jar' made in the patch. Is that how you are starting things? Another potential problem could be that the URLClassloader is not correctly constructed. The urls added must be either a jar file or end in slashes. It might be useful to log the urls added to the path in LocalJobRunner to check this.
        Hide
        stack added a comment -

        The patch seems to have no effect. When my map goes to run, unless the classes I want to instantiate are in the root of the job jar, they are not found (ClassNotFoundException). Nothing under the job jar /lib nor /classes directory is ever found.

        Tracing, I see that any new class will have null for ClassLoader – i.e. the "bootstrap class loader" – never the URLClassLoader created by this patch.

        Seems like this patch runs too late in the game to be of use? In order to create a job, I need to be able to refer to the implementing class (and possibly configurations) BEFORE LocalJobRunner has had a chance to do its job jar unrolling machinations. To be able to refer to implementing classes, they need to be on the immediate CLASSPATH.

        Seems like any messings w/ classloading populating the CLASSPATH needs to happen before we run Main-Class. This would be most useful I'd imagine. Would a java program that did nothing but read hadoop config. for hadoop local and system dir and then unrolled the job jar finishing by emitting Main-Class and CLASSPATH additions for the hadoop script to exec online #156 be too ugly? It would have the advantage of resemblng how job jars are unrolled out on slaves.

        I can work up such a patch if amenable.

        Show
        stack added a comment - The patch seems to have no effect. When my map goes to run, unless the classes I want to instantiate are in the root of the job jar, they are not found (ClassNotFoundException). Nothing under the job jar /lib nor /classes directory is ever found. Tracing, I see that any new class will have null for ClassLoader – i.e. the "bootstrap class loader" – never the URLClassLoader created by this patch. Seems like this patch runs too late in the game to be of use? In order to create a job, I need to be able to refer to the implementing class (and possibly configurations) BEFORE LocalJobRunner has had a chance to do its job jar unrolling machinations. To be able to refer to implementing classes, they need to be on the immediate CLASSPATH. Seems like any messings w/ classloading populating the CLASSPATH needs to happen before we run Main-Class. This would be most useful I'd imagine. Would a java program that did nothing but read hadoop config. for hadoop local and system dir and then unrolled the job jar finishing by emitting Main-Class and CLASSPATH additions for the hadoop script to exec online #156 be too ugly? It would have the advantage of resemblng how job jars are unrolled out on slaves. I can work up such a patch if amenable.
        Hide
        Doug Cutting added a comment -

        Tell me if this fixes things for you. Thanks!

        Show
        Doug Cutting added a comment - Tell me if this fixes things for you. Thanks!

          People

          • Assignee:
            Doug Cutting
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development