Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2323

start-dfs.sh script fails for tarball install

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I build Common and HDFS tarballs from trunk then tried to start a cluster with start-dfs.sh, but I got the following error:

      Starting namenodes on [localhost ]
      sbin/start-dfs.sh: line 55: /Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or directory
      sbin/start-dfs.sh: line 68: /Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or directory
      Starting secondary namenodes [0.0.0.0 ]
      sbin/start-dfs.sh: line 88: /Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or directory
      

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #793 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/793/)
        HDFS-2323. start-dfs.sh script fails for tarball install

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #793 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/793/ ) HDFS-2323 . start-dfs.sh script fails for tarball install tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #829 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/829/)
        HDFS-2323. start-dfs.sh script fails for tarball install

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #829 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/829/ ) HDFS-2323 . start-dfs.sh script fails for tarball install tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #893 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/893/)
        HDFS-2323. start-dfs.sh script fails for tarball install

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #893 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/893/ ) HDFS-2323 . start-dfs.sh script fails for tarball install tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #881 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/881/)
        HDFS-2323. start-dfs.sh script fails for tarball install

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #881 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/881/ ) HDFS-2323 . start-dfs.sh script fails for tarball install tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.23.0 [ 12315571 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I've just committed this. (The unit test failures were not caused by this change.)

        Show
        Tom White added a comment - I've just committed this. (The unit test failures were not caused by this change.)
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/958/)
        HDFS-2323. start-dfs.sh script fails for tarball install

        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/958/ ) HDFS-2323 . start-dfs.sh script fails for tarball install tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170233 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-dfs.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-secure-dns.sh /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
        Hide
        Tom White added a comment -

        > the start-dfs.sh script will not work without first overlaying the Common and HDFS target directories

        Yes, this is because the scripts assume that there is a single HADOOP_PREFIX directory containing bin, conf, etc. It might be a good idea to loosen this at some point, but this bug fix isn't the place to do it.

        Show
        Tom White added a comment - > the start-dfs.sh script will not work without first overlaying the Common and HDFS target directories Yes, this is because the scripts assume that there is a single HADOOP_PREFIX directory containing bin, conf, etc. It might be a good idea to loosen this at some point, but this bug fix isn't the place to do it.
        Hide
        Aaron T. Myers added a comment -

        +1, the patch looks good to me. I reviewed the code and also tested the script per Tom's instructions, and it worked like a charm.

        It might be worth noting somewhere that even with this patch, the start-dfs.sh script will not work without first overlaying the Common and HDFS target directories.

        Show
        Aaron T. Myers added a comment - +1, the patch looks good to me. I reviewed the code and also tested the script per Tom's instructions, and it worked like a charm. It might be worth noting somewhere that even with this patch, the start-dfs.sh script will not work without first overlaying the Common and HDFS target directories.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12493689/HDFS-2323.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493689/HDFS-2323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//console This message is automatically generated.
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Tom White added a comment -

        To ran a pseudo distributed HDFS cluster I ran

        mvn clean package -Pdist -Dtar -DskipTests -P-cbuild
        

        from the top level, then from another directory

        mkdir hadoop
        cd hadoop
        tar zxf ~/workspace/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.24.0-SNAPSHOT.tar.gz --strip 1
        tar zxf ~/workspace/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.tar.gz --strip 1
        export HADOOP_COMMON_HOME=$(pwd)
        export HADOOP_HDFS_HOME=$(pwd)
        
        cat > $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml  << EOF
        <?xml version="1.0"?>
        <!-- core-site.xml -->
        <configuration>
          <property>
            <name>fs.default.name</name>
            <value>hdfs://localhost/</value>
          </property>
        </configuration>
        
        
        EOF
        
        sbin/start-dfs.sh 
        
        Show
        Tom White added a comment - To ran a pseudo distributed HDFS cluster I ran mvn clean package -Pdist -Dtar -DskipTests -P-cbuild from the top level, then from another directory mkdir hadoop cd hadoop tar zxf ~/workspace/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.24.0-SNAPSHOT.tar.gz --strip 1 tar zxf ~/workspace/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.tar.gz --strip 1 export HADOOP_COMMON_HOME=$(pwd) export HADOOP_HDFS_HOME=$(pwd) cat > $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml << EOF <?xml version="1.0"?> <!-- core-site.xml --> <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost/</value> </property> </configuration> EOF sbin/start-dfs.sh
        Tom White made changes -
        Field Original Value New Value
        Attachment HDFS-2323.patch [ 12493689 ]
        Hide
        Tom White added a comment -

        There are a couple of problems:

        1. The scripts refer to "bin/hadoop-daemons.sh" not "sbin/hadoop-daemons.sh".
        2. The secondary namenode should not be started at all since it's not configured to, but the script tries to start it since the test for "0.0.0.0" fails due to a trailing space.

        This patch fixes both of these.

        Show
        Tom White added a comment - There are a couple of problems: 1. The scripts refer to "bin/hadoop-daemons.sh" not "sbin/hadoop-daemons.sh". 2. The secondary namenode should not be started at all since it's not configured to, but the script tries to start it since the test for "0.0.0.0" fails due to a trailing space. This patch fixes both of these.
        Tom White created issue -

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development