Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10576 DiskBalancer followup work items
  3. HDFS-10562

DiskBalancer: update documentation on how to report issues and debug

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: HDFS-1312
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: balancer & mover
    • Labels:
      None

      Description

      Add a section in the diskbalancer documentation on how to report issues and how to debug diskbalancer usage.

      1. HDFS-10562-HDFS-1312.001.patch
        7 kB
        Anu Engineer
      2. HDFS-10562.002.patch
        6 kB
        Anu Engineer
      3. HDFS-10562.003.patch
        6 kB
        Anu Engineer

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 40s HDFS-1312 passed
        +1 mvnsite 0m 54s HDFS-1312 passed
        +1 mvnsite 0m 50s the patch passed
        -1 whitespace 0m 0s The patch has 51 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        9m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c88012f
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812620/HDFS-10562-HDFS-1312.001.patch
        JIRA Issue HDFS-10562
        Optional Tests asflicense mvnsite
        uname Linux 248141b1e2be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision HDFS-1312 / 4f5beb6
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15879/artifact/patchprocess/whitespace-eol.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15879/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 40s HDFS-1312 passed +1 mvnsite 0m 54s HDFS-1312 passed +1 mvnsite 0m 50s the patch passed -1 whitespace 0m 0s The patch has 51 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 9m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:c88012f JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812620/HDFS-10562-HDFS-1312.001.patch JIRA Issue HDFS-10562 Optional Tests asflicense mvnsite uname Linux 248141b1e2be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-1312 / 4f5beb6 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15879/artifact/patchprocess/whitespace-eol.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15879/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Hi Anu Engineer, thanks for the doc improvements. One question regarding the bandwidth. Should the first sentence be reworded to say something like This is the maximum aggregate source disk bandwidth used by the balancer?

        223	*bandwidth* - From disk balancer's point of view this is the bandwidth of the 
        224	source disk. After moving a block disk balancer computes how many seconds it 
        225	should have taken to move that block with the specified bandwidth. If the 
        226	actual move took less time than expected, then disk balancer will sleep for 
        227	that duration. Please note that currently all moves are executed 
        228	sequentially by a single thread.  
        
        Show
        arpitagarwal Arpit Agarwal added a comment - Hi Anu Engineer , thanks for the doc improvements. One question regarding the bandwidth . Should the first sentence be reworded to say something like This is the maximum aggregate source disk bandwidth used by the balancer ? 223 *bandwidth* - From disk balancer's point of view this is the bandwidth of the 224 source disk. After moving a block disk balancer computes how many seconds it 225 should have taken to move that block with the specified bandwidth. If the 226 actual move took less time than expected, then disk balancer will sleep for 227 that duration. Please note that currently all moves are executed 228 sequentially by a single thread.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 2s Docker failed to build yetus/hadoop:85209cc.



        Subsystem Report/Notes
        JIRA Issue HDFS-10562
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812620/HDFS-10562-HDFS-1312.001.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16689/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 2s Docker failed to build yetus/hadoop:85209cc. Subsystem Report/Notes JIRA Issue HDFS-10562 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812620/HDFS-10562-HDFS-1312.001.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16689/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        anu Anu Engineer added a comment -

        Arpit Agarwal Thanks for the code review comments. I have updated the patch.

        Show
        anu Anu Engineer added a comment - Arpit Agarwal Thanks for the code review comments. I have updated the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 8m 50s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvnsite 0m 51s the patch passed
        -1 whitespace 0m 0s The patch has 66 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        11m 26s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10562
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828146/HDFS-10562.002.patch
        Optional Tests asflicense mvnsite
        uname Linux f16324da63f9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 72dfb04
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16723/artifact/patchprocess/whitespace-eol.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16723/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 8m 50s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvnsite 0m 51s the patch passed -1 whitespace 0m 0s The patch has 66 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 asflicense 0m 17s The patch does not generate ASF License warnings. 11m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10562 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828146/HDFS-10562.002.patch Optional Tests asflicense mvnsite uname Linux f16324da63f9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72dfb04 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16723/artifact/patchprocess/whitespace-eol.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16723/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1. A minor comment:

        212	 *max disk errors* - The configuration used for this move step. currently it will
        213	 report it as zero, since the user interface to control these values per step
        214	 is not in place. It is a future work item. The default or the command line 
        215	 value specified in plan command is used for this value. 
        

        The sample output shows "maxDiskErrors" : 5,. Should we fix the output to show zero to be consistent with the description?

        Show
        arpitagarwal Arpit Agarwal added a comment - +1. A minor comment: 212 *max disk errors* - The configuration used for this move step. currently it will 213 report it as zero, since the user interface to control these values per step 214 is not in place. It is a future work item. The default or the command line 215 value specified in plan command is used for this value. The sample output shows "maxDiskErrors" : 5, . Should we fix the output to show zero to be consistent with the description?
        Hide
        anu Anu Engineer added a comment -

        Thanks for catching that, will do.

        Show
        anu Anu Engineer added a comment - Thanks for catching that, will do.
        Hide
        anu Anu Engineer added a comment -

        Fixed the issue pointed out by Arpit Agarwal

        Show
        anu Anu Engineer added a comment - Fixed the issue pointed out by Arpit Agarwal
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 47s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvnsite 0m 48s the patch passed
        -1 whitespace 0m 0s The patch has 67 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        9m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10562
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828162/HDFS-10562.003.patch
        Optional Tests asflicense mvnsite
        uname Linux 137807f6aa70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 72dfb04
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16729/artifact/patchprocess/whitespace-eol.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16729/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 47s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvnsite 0m 48s the patch passed -1 whitespace 0m 0s The patch has 67 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 asflicense 0m 15s The patch does not generate ASF License warnings. 9m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10562 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828162/HDFS-10562.003.patch Optional Tests asflicense mvnsite uname Linux 137807f6aa70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72dfb04 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16729/artifact/patchprocess/whitespace-eol.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16729/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 thanks Anu Engineer.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 thanks Anu Engineer .
        Hide
        anu Anu Engineer added a comment -

        Arpit Agarwal Thanks for the code reviews and your comments. I have committed this to trunk.

        Show
        anu Anu Engineer added a comment - Arpit Agarwal Thanks for the code reviews and your comments. I have committed this to trunk.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10437 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10437/)
        HDFS-10562. DiskBalancer: update documentation on how to report issues (aengineer: rev 1c0d18f32289837b8981aed80e7bdcd360adfb85)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSDiskbalancer.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10437 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10437/ ) HDFS-10562 . DiskBalancer: update documentation on how to report issues (aengineer: rev 1c0d18f32289837b8981aed80e7bdcd360adfb85) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSDiskbalancer.md

          People

          • Assignee:
            anu Anu Engineer
            Reporter:
            anu Anu Engineer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development