Hadoop Common
  1. Hadoop Common
  2. HADOOP-6152

Hadoop scripts do not correctly put jars on the classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-hdfs.jar, etc, rather than the hadoop-hdfs.jar that is actually created by the 'ant jar' and 'ant package' targets.

      1. HADOOP-6152.patch
        2 kB
        Aaron Kimball
      2. HADOOP-6152.2.patch
        2 kB
        Aaron Kimball
      3. HADOOP-6152.3.patch
        3 kB
        Aaron Kimball
      4. HADOOP-6152.4.patch
        3 kB
        Aaron Kimball
      5. HADOOP-6152.5.patch
        4 kB
        Aaron Kimball

        Activity

        Hide
        Aaron Kimball added a comment -

        This patch fixes the various bin/* scripts.

        Show
        Aaron Kimball added a comment - This patch fixes the various bin/* scripts.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12413596/HADOOP-6152.patch
        against trunk revision 793987.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413596/HADOOP-6152.patch against trunk revision 793987. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        No tests added because we don't have a test framework that works around the end-user package target.

        Show
        Aaron Kimball added a comment - No tests added because we don't have a test framework that works around the end-user package target.
        Hide
        Jay Booth added a comment -

        Seems like hadoop-config.sh doesn't include the contrib jars from hdfs or mapred.. is this by design, are they listed elsewhere?

        Show
        Jay Booth added a comment - Seems like hadoop-config.sh doesn't include the contrib jars from hdfs or mapred.. is this by design, are they listed elsewhere?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The patch cannot be applied. Could you update it?

        Show
        Tsz Wo Nicholas Sze added a comment - The patch cannot be applied. Could you update it?
        Hide
        Aaron Kimball added a comment -

        Attaching new resync'd patch.

        Show
        Aaron Kimball added a comment - Attaching new resync'd patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416220/HADOOP-6152.2.patch
        against trunk revision 803296.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416220/HADOOP-6152.2.patch against trunk revision 803296. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Look like that the jar file for tools is in mapreduce but not common. Could you check it?

        -for f in $HADOOP_CORE_HOME/hadoop-*-tools.jar; do
        +for f in $HADOOP_CORE_HOME/hadoop-tools-*.jar; do
           TOOL_PATH=${TOOL_PATH}:$f;
         done
        -for f in $HADOOP_CORE_HOME/build/hadoop-*-tools.jar; do
        +for f in $HADOOP_CORE_HOME/build/hadoop-tools-*.jar; do
           TOOL_PATH=${TOOL_PATH}:$f;
         done
        
        Show
        Tsz Wo Nicholas Sze added a comment - Look like that the jar file for tools is in mapreduce but not common. Could you check it? - for f in $HADOOP_CORE_HOME/hadoop-*-tools.jar; do + for f in $HADOOP_CORE_HOME/hadoop-tools-*.jar; do TOOL_PATH=${TOOL_PATH}:$f; done - for f in $HADOOP_CORE_HOME/build/hadoop-*-tools.jar; do + for f in $HADOOP_CORE_HOME/build/hadoop-tools-*.jar; do TOOL_PATH=${TOOL_PATH}:$f; done
        Hide
        Aaron Kimball added a comment -

        Good catch. Adding a new patch that handles this case.

        Show
        Aaron Kimball added a comment - Good catch. Adding a new patch that handles this case.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416278/HADOOP-6152.3.patch
        against trunk revision 803296.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416278/HADOOP-6152.3.patch against trunk revision 803296. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The tools jar file name should be something like hadoop-mapred-tools-0.21.0-dev.jar.

        +for f in $HADOOP_MAPRED_HOME/hadoop-tools-*.jar; do
        +  TOOL_PATH=${TOOL_PATH}:$f;
        +done
        +for f in $HADOOP_MAPRED_HOME/build/hadoop-tools-*.jar; do
        +  TOOL_PATH=${TOOL_PATH}:$f;
        +done
        
        Show
        Tsz Wo Nicholas Sze added a comment - The tools jar file name should be something like hadoop-mapred-tools-0.21.0-dev.jar. + for f in $HADOOP_MAPRED_HOME/hadoop-tools-*.jar; do + TOOL_PATH=${TOOL_PATH}:$f; +done + for f in $HADOOP_MAPRED_HOME/build/hadoop-tools-*.jar; do + TOOL_PATH=${TOOL_PATH}:$f; +done
        Hide
        Aaron Kimball added a comment -

        Fixed mapred-tools jar path; attached new patch.

        Show
        Aaron Kimball added a comment - Fixed mapred-tools jar path; attached new patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417192/HADOOP-6152.4.patch
        against trunk revision 806368.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417192/HADOOP-6152.4.patch against trunk revision 806368. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good

        > -1 tests included. The patch doesn't appear to include any new or modified tests.
        We are currently doing manual tests for scripts. Could you post the manual test results and "Also please list what manual steps were performed to verify this patch"?

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good > -1 tests included. The patch doesn't appear to include any new or modified tests. We are currently doing manual tests for scripts. Could you post the manual test results and "Also please list what manual steps were performed to verify this patch"?
        Hide
        Aaron Kimball added a comment -

        Fixed more bugs in the script. Attaching (hopefully!) final version.

        My test procedure:

        • I have checked out the projects in ~/src/git/hadoop-common, ~/src/git/hdfs, and ~/src/git/mapred.
        • Create directory links:
          $ cd ~/src/git/hadoop-common
          $ ln -s ~/src/git/hdfs hdfs
          $ ln -s ~/src/git/mapred mapred
          
        • Build all three projects:
          $ cd ~/src/git/hadoop-common
          $ ant clean jar
          
        • Repeat in hdfs, mapred
        • If you want version info to be accurate, run 'ant jar' again in all three directories.
        • Configured a pseudo-distributed cluster in hadoop-common/conf/mapred-site.xml, hdfs-site.xml, core-site.xml
        • Start services:
          $ cd ~/src/git/hadoop-common
          $ bin/hdfs namenode -format
          $ bin/start-dfs.sh
          $ bin/start-mapred.sh
          
        • Prove that it works:
          $ bin/hadoop version
          Hadoop 0.21.0-dev
          Subversion git://jargon/home/aaron/src/git/hadoop-common -r 2480316d211ac22679400f000ad6c04c48358e71
          Compiled by aaron on Fri Aug 21 15:21:35 PDT 2009
          From source with checksum b5412529439930b062e5e706c0bf2456
          $ bin/hadoop fs -ls /
          Found 1 items
          drwxr-xr-x   - aaron supergroup          0 2009-08-21 15:16 /tmp
          
        Show
        Aaron Kimball added a comment - Fixed more bugs in the script. Attaching (hopefully!) final version. My test procedure: I have checked out the projects in ~/src/git/hadoop-common , ~/src/git/hdfs , and ~/src/git/mapred . Create directory links: $ cd ~/src/git/hadoop-common $ ln -s ~/src/git/hdfs hdfs $ ln -s ~/src/git/mapred mapred Build all three projects: $ cd ~/src/git/hadoop-common $ ant clean jar Repeat in hdfs, mapred If you want version info to be accurate, run 'ant jar' again in all three directories. Configured a pseudo-distributed cluster in hadoop-common/conf/mapred-site.xml, hdfs-site.xml, core-site.xml Start services: $ cd ~/src/git/hadoop-common $ bin/hdfs namenode -format $ bin/start-dfs.sh $ bin/start-mapred.sh Prove that it works: $ bin/hadoop version Hadoop 0.21.0-dev Subversion git: //jargon/home/aaron/src/git/hadoop-common -r 2480316d211ac22679400f000ad6c04c48358e71 Compiled by aaron on Fri Aug 21 15:21:35 PDT 2009 From source with checksum b5412529439930b062e5e706c0bf2456 $ bin/hadoop fs -ls / Found 1 items drwxr-xr-x - aaron supergroup 0 2009-08-21 15:16 /tmp
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12417331/HADOOP-6152.5.patch
        against trunk revision 806745.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417331/HADOOP-6152.5.patch against trunk revision 806745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > Prove that it works: ...
        Hi Aaron, I tried an earlier version of the patch (with some additional fixes). Everything works fine except for the jsp web pages (e.g. http://namenode:50070/dfshealth.jsp). Have you checked it with your latest patch?

        +1 on the latest patch which fixes many problems. We may commit it first and fix the jsp problem, if there is any, in a new jira.

        Show
        Tsz Wo Nicholas Sze added a comment - > Prove that it works: ... Hi Aaron, I tried an earlier version of the patch (with some additional fixes). Everything works fine except for the jsp web pages (e.g. http://namenode:50070/dfshealth.jsp ). Have you checked it with your latest patch? +1 on the latest patch which fixes many problems. We may commit it first and fix the jsp problem, if there is any, in a new jira.
        Hide
        Aaron Kimball added a comment -

        This doesn't fix the JSP issue. I think that'll require a separate issue. I've gotten closer to fixing some JSP build-process stuff in my MAPREDUCE-679 and HDFS-453 patches which are as-yet uncommitted. But they still don't totally fix the issue. (To test them, I am manually copying the build/webapps/ and build/classes/ hierarchies on top of one another.)

        Show
        Aaron Kimball added a comment - This doesn't fix the JSP issue. I think that'll require a separate issue. I've gotten closer to fixing some JSP build-process stuff in my MAPREDUCE-679 and HDFS-453 patches which are as-yet uncommitted. But they still don't totally fix the issue. (To test them, I am manually copying the build/webapps/ and build/classes/ hierarchies on top of one another.)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Aaron!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Aaron!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #70 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/70/)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #70 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/70/ )

          People

          • Assignee:
            Aaron Kimball
            Reporter:
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development