Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9043

Doc updation for commands in HDFS Federation

    Details

    • Hadoop Flags:
      Reviewed

      Description

      1. command is wrong

       $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode <datanode_host_name>:<datanode_rpc_port>
      

      Correct command is : hdfs dfsadmin -refreshNameNode's'

      2.command is wrong

       $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script $HADOOP_PREFIX/bin/hdfs start balancer 
      

      Correct command is : start-balancer.sh -policy

      3. Reference link to balancer for further details is wrong

      Note that Balancer only balances the data and does not balance the namespace. For the complete command usage, see balancer.
      
      1. HDFS-9043-1.patch
        0.8 kB
        J.Andreina
      2. HDFS-9043-branch-2.7.0-1.patch
        2 kB
        J.Andreina
      3. HDFS-9043-branch-2-1.patch
        1 kB
        J.Andreina

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #404 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/404/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #404 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/404/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2342 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2342/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2342 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2342/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2369 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2369/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2369 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2369/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1163/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1163/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #431 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/431/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #431 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/431/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #423 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/423/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #423 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/423/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8499 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8499/)
        HDFS-9043. Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8499 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8499/ ) HDFS-9043 . Doc updation for commands in HDFS Federation (Contributed by J.Andreina) (vinayakumarb: rev 57003fa971658c8482240f70445a6822c7692844) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/Federation.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        vinayrpet Vinayakumar B added a comment -

        Committed to trunk, branch-2 and branch-2.7

        Thanks J.Andreina for the contribution.
        Thanks Allen Wittenauer for reviews and suggestions.

        Show
        vinayrpet Vinayakumar B added a comment - Committed to trunk, branch-2 and branch-2.7 Thanks J.Andreina for the contribution. Thanks Allen Wittenauer for reviews and suggestions.
        Hide
        vinayrpet Vinayakumar B added a comment -

        In trunk already command is updated correctly using hdfs

        And for above commands there is no alternative yet.

        Sounds good.

        Show
        vinayrpet Vinayakumar B added a comment - In trunk already command is updated correctly using hdfs And for above commands there is no alternative yet. Sounds good.
        Hide
        andreina J.Andreina added a comment -

        Thanks Allen Wittenauer for the comments.

        The given trunk patch appears to be missing the same fixes as the branch-2 patch

        In trunk already command is updated correctly using hdfs as :

        [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start balancer [-policy <policy>]
        

        Also, if we're going to bother with this, let's make it complete: all of the sbin commands should get replaced with their equivalent bin versions since sbin is mostly deprecated.

        Currently the sbin commands used are :

        sbin/refresh-namenodes.sh
        sbin/distribute-exclude.sh
        sbin/stop-dfs.sh
        sbin/start-dfs.sh
        

        And for above commands there is no alternative yet.

        Please correct me if iam wrong .

        Show
        andreina J.Andreina added a comment - Thanks Allen Wittenauer for the comments. The given trunk patch appears to be missing the same fixes as the branch-2 patch In trunk already command is updated correctly using hdfs as : [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start balancer [-policy <policy>] Also, if we're going to bother with this, let's make it complete: all of the sbin commands should get replaced with their equivalent bin versions since sbin is mostly deprecated. Currently the sbin commands used are : sbin/refresh-namenodes.sh sbin/distribute-exclude.sh sbin/stop-dfs.sh sbin/start-dfs.sh And for above commands there is no alternative yet. Please correct me if iam wrong .
        Hide
        aw Allen Wittenauer added a comment -

        The given trunk patch appears to be missing the same fixes as the branch-2 patch. Also, if we're going to bother with this, let's make it complete: all of the sbin commands should get replaced with their equivalent bin versions since sbin is mostly deprecated.

        Show
        aw Allen Wittenauer added a comment - The given trunk patch appears to be missing the same fixes as the branch-2 patch. Also, if we're going to bother with this, let's make it complete: all of the sbin commands should get replaced with their equivalent bin versions since sbin is mostly deprecated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        All changes are looks good to me. +1 for all.

        Allen Wittenauer, do you also want to take a look?

        Will wait for a day before commit.
        -Thanks

        Show
        vinayrpet Vinayakumar B added a comment - All changes are looks good to me. +1 for all. Allen Wittenauer , do you also want to take a look? Will wait for a day before commit. -Thanks
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 3m 38s Pre-patch branch-2 compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 8s Site still builds.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            7m 15s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12755362/HDFS-9043-branch-2-1.patch
        Optional Tests site
        git revision branch-2 / d582345
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12394/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 38s Pre-patch branch-2 compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 3m 8s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     7m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755362/HDFS-9043-branch-2-1.patch Optional Tests site git revision branch-2 / d582345 Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12394/console This message was automatically generated.
        Hide
        andreina J.Andreina added a comment -

        Thanks Allen Wittenauer for the comments.

        Attached separate patches for trunk , branch-2 and branch-2.7.0 (as the fixes are different in each branch).
        Please review.

        Show
        andreina J.Andreina added a comment - Thanks Allen Wittenauer for the comments. Attached separate patches for trunk , branch-2 and branch-2.7.0 (as the fixes are different in each branch). Please review.
        Hide
        aw Allen Wittenauer added a comment -

        Shall we update the command to start-balancer.sh -policy which would be common to both branch-2 and trunk

        No, because sbin/* is mostly deprecated in trunk. This needs to be two different patches for the two different branches.

        Show
        aw Allen Wittenauer added a comment - Shall we update the command to start-balancer.sh -policy which would be common to both branch-2 and trunk No, because sbin/* is mostly deprecated in trunk. This needs to be two different patches for the two different branches.
        Hide
        andreina J.Andreina added a comment -

        Thanks Allen Wittenauer for the comments.
        I agree with your point .
        Shall we update the command to start-balancer.sh -policy which would be common to both branch-2 and trunk

        Show
        andreina J.Andreina added a comment - Thanks Allen Wittenauer for the comments. I agree with your point . Shall we update the command to start-balancer.sh -policy which would be common to both branch-2 and trunk
        Hide
        aw Allen Wittenauer added a comment -

        Also:

        It's worth noting that your replacement balancer command does something slightly different. The previous version starts it as a daemon. If it was intentionally meant to be a daemon, then we need to add a --daemon start to the replacement.

        There's likely other fixes that need to go in place now that more of the shell rewrite is in place. (e.g., get rid of all of the hadoop-daemon commands)

        Show
        aw Allen Wittenauer added a comment - Also: It's worth noting that your replacement balancer command does something slightly different. The previous version starts it as a daemon. If it was intentionally meant to be a daemon, then we need to add a --daemon start to the replacement. There's likely other fixes that need to go in place now that more of the shell rewrite is in place. (e.g., get rid of all of the hadoop-daemon commands)
        Hide
        aw Allen Wittenauer added a comment -

        Be aware that this document is different in trunk due to the shell rewrite.

        Show
        aw Allen Wittenauer added a comment - Be aware that this document is different in trunk due to the shell rewrite.

          People

          • Assignee:
            andreina J.Andreina
            Reporter:
            andreina J.Andreina
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development