Hadoop Common
  1. Hadoop Common
  2. HADOOP-6123

hdfs script does not work after project split.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are problems running hdfs script from common.

      1. Usage message for hdfs does not have "fs" option anymore.
      2. There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
      3. Same with other options e.g. name-node.

      May I am missing something here. How do we call ls these days?
      Do we need to move hdfs script to hdfs project.

      1. 6123_v1.patch
        0.7 kB
        Sharad Agarwal

        Activity

        Hide
        Sharad Agarwal added a comment -

        The scripts won't run as it is for common and hdfs as the ivy lib path has changed. Classpath in hadoop-config.sh needs to include new paths.

        Show
        Sharad Agarwal added a comment - The scripts won't run as it is for common and hdfs as the ivy lib path has changed. Classpath in hadoop-config.sh needs to include new paths.
        Hide
        Sharad Agarwal added a comment -

        This patch updates the classpath after the project split.
        Apply this patch and copy the bin folder in hdfs project. All scripts should work as it is. This eventually has to be fixed in build file of mapred and hdfs which can extract the bin folder out of the common jar.

        Show
        Sharad Agarwal added a comment - This patch updates the classpath after the project split. Apply this patch and copy the bin folder in hdfs project. All scripts should work as it is. This eventually has to be fixed in build file of mapred and hdfs which can extract the bin folder out of the common jar.
        Hide
        Suresh Srinivas added a comment -

        I tried the change. It works.

        I noticed that unlike before I have to setup JAVA_HOME in hadoop-env.sh. This is not picked up from JAVA_HOME environment variable from my shell.

        Show
        Suresh Srinivas added a comment - I tried the change. It works. I noticed that unlike before I have to setup JAVA_HOME in hadoop-env.sh. This is not picked up from JAVA_HOME environment variable from my shell.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        +if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then
        +for f in $HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar; do
        +  CLASSPATH=${CLASSPATH}:$f;
        +done
        +fi
        

        The for-loop above can be eliminated by keeping the wild card (shown below) as what we does in the bin/hadoop script.

        if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then
          CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar;
        fi
        

        Try also the new "bin/hadoop classpath" command committed (HADOOP-5976) recently.

        Show
        Tsz Wo Nicholas Sze added a comment - + if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then + for f in $HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar; do + CLASSPATH=${CLASSPATH}:$f; +done +fi The for-loop above can be eliminated by keeping the wild card (shown below) as what we does in the bin/hadoop script. if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar; fi Try also the new "bin/hadoop classpath" command committed ( HADOOP-5976 ) recently.
        Hide
        Suresh Srinivas added a comment -

        when is this change going to be committed?

        Show
        Suresh Srinivas added a comment - when is this change going to be committed?
        Hide
        Hong Tang added a comment -

        You never mark the jira "patch available".

        Show
        Hong Tang added a comment - You never mark the jira "patch available".
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Sharad, do you think my previous comment making sense?

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Sharad, do you think my previous comment making sense?
        Hide
        Sharad Agarwal added a comment -

        do you think my previous comment making sense?

        I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH.

        Show
        Sharad Agarwal added a comment - do you think my previous comment making sense? I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH.
        In the java man page, it says,

        -classpath classpath
        -cp classpath

        ... As a special convenience, a class path element containing a basename of * is considered equivalent to specifying a list of all the files in the directory with the extension .jar or .JAR (a java program cannot tell the difference between the two invocations).

        For example, if directory foo contains a.jar and b.JAR, then the class path element foo/* is expanded to a A.jar:b.JAR, except that the order of jar files is unspecified. ...

        Show
        Tsz Wo Nicholas Sze added a comment - > I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH. In the java man page , it says, -classpath classpath -cp classpath ... As a special convenience, a class path element containing a basename of * is considered equivalent to specifying a list of all the files in the directory with the extension .jar or .JAR (a java program cannot tell the difference between the two invocations). For example, if directory foo contains a.jar and b.JAR, then the class path element foo/* is expanded to a A.jar:b.JAR, except that the order of jar files is unspecified. ...
        Hide
        Sharad Agarwal added a comment -

        @Tsz When I tried, it didn't work for me. May be I am missing something. Have you tried it? If it works for you, can you post a patch?

        Show
        Sharad Agarwal added a comment - @Tsz When I tried, it didn't work for me. May be I am missing something. Have you tried it? If it works for you, can you post a patch?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I tried your for-loop patch. It does not work.

        bash-3.2$ ./bin/hdfs namenode
        java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/server/namenode/NameNode
        ...
        

        > ... Have you tried it? If it works for you, can you post a patch?
        I have not tried it.

        Show
        Tsz Wo Nicholas Sze added a comment - I tried your for-loop patch. It does not work. bash-3.2$ ./bin/hdfs namenode java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/server/namenode/NameNode ... > ... Have you tried it? If it works for you, can you post a patch? I have not tried it.
        Hide
        Sharad Agarwal added a comment -

        I tried your for-loop patch. It does not work.

        It works for me with latest trunk. By any chance you are running the command in common itself ? It should be run in HDFS. See my comment above.

        Show
        Sharad Agarwal added a comment - I tried your for-loop patch. It does not work. It works for me with latest trunk. By any chance you are running the command in common itself ? It should be run in HDFS. See my comment above.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12412236/6123_v1.patch
        against trunk revision 798093.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412236/6123_v1.patch against trunk revision 798093. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Tried again the patch. It worked fine. I probably did something wrong last time.

        Since the for-loop approach is already used in hadoop-config.sh, we should keep the new codes consistent with the existing codes.

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - Tried again the patch. It worked fine. I probably did something wrong last time. Since the for-loop approach is already used in hadoop-config.sh, we should keep the new codes consistent with the existing codes. +1 patch looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        No new tests since this is a script change.

        I have committed this. Thanks, Sharad!

        Show
        Tsz Wo Nicholas Sze added a comment - No new tests since this is a script change. I have committed this. Thanks, Sharad!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/37/)
        . Add missing classpaths in hadoop-config.sh. Contributed by Sharad Agarwal

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/37/ ) . Add missing classpaths in hadoop-config.sh. Contributed by Sharad Agarwal
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21. Just a bug.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21. Just a bug.

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development