Hadoop Common
  1. Hadoop Common
  2. HADOOP-7374

Don't add tools.jar to the classpath when running Hadoop

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The scripts that run Hadoop no longer automatically add tools.jar from the JDK to the classpath (if it is present). If your job depends on tools.jar in the JDK you will need to add this dependency in your job.

      Description

      bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. Executing Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

        Activity

        Eli Collins created issue -
        Hide
        Eli Collins added a comment -

        Patch attached. Ran the hdfs and mr daemons and looked at the web UIs for sanity.

        Show
        Eli Collins added a comment - Patch attached. Ran the hdfs and mr daemons and looked at the web UIs for sanity.
        Eli Collins made changes -
        Field Original Value New Value
        Attachment hadoop-7374-1.patch [ 12481985 ]
        Hide
        Owen O'Malley added a comment -

        The thing to test is that the tools still run: distcp, har, etc.

        Show
        Owen O'Malley added a comment - The thing to test is that the tools still run: distcp, har, etc.
        Hide
        Eli Collins added a comment -

        Thanks Owen. I tested distcp and har. I also grepped the sources for the package names in tools.jar and that didn't turn up anything so I think the code is clear. The only dependency looks like Jasper to compile the jsps, but that's a build-time dependency.

        Attached (trivial) patch look good?

        Show
        Eli Collins added a comment - Thanks Owen. I tested distcp and har. I also grepped the sources for the package names in tools.jar and that didn't turn up anything so I think the code is clear. The only dependency looks like Jasper to compile the jsps, but that's a build-time dependency. Attached (trivial) patch look good?
        Eli Collins made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481985/hadoop-7374-1.patch
        against trunk revision 1133125.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481985/hadoop-7374-1.patch against trunk revision 1133125. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        +1

        Show
        Todd Lipcon added a comment - +1
        Hide
        Eli Collins added a comment -

        Thanks Todd! I've committed this.

        Show
        Eli Collins added a comment - Thanks Todd! I've committed this.
        Eli Collins made changes -
        Summary hadoop-config.sh shouldn't add tools.jar to the classpath Don't add tools.jar to the classpath when running Hadoop
        Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
        Release Note The scripts that run Hadoop no longer automatically add tools.jar from the JDK to the classpath (if it is present). If your job depends on tools.jar in the JDK you will need to add this dependency in your job.
        Fix Version/s 0.23.0 [ 12315569 ]
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #649 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/649/)
        HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/bin/hadoop-config.sh
        • /hadoop/common/trunk/bin/rcc
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #649 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/649/ ) HADOOP-7374 . Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/bin/hadoop-config.sh /hadoop/common/trunk/bin/rcc
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-maven #13 (See https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/)
        HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/bin/hadoop-config.sh
        • /hadoop/common/trunk/bin/rcc
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-maven #13 (See https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/ ) HADOOP-7374 . Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/bin/hadoop-config.sh /hadoop/common/trunk/bin/rcc
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #717 (See https://builds.apache.org/job/Hadoop-Common-trunk/717/)
        HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/bin/hadoop-config.sh
        • /hadoop/common/trunk/bin/rcc
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #717 (See https://builds.apache.org/job/Hadoop-Common-trunk/717/ ) HADOOP-7374 . Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/bin/hadoop-config.sh /hadoop/common/trunk/bin/rcc
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development