Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10534

NameNode WebUI should display DataNode usage histogram

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In addition of Min/Median/Max, another meaningful metric for cluster balance is DN usage in histogram style.
      Since NN already has provided necessary information to calculate histogram of DN usage, it can be done in JS side.

      1. HDFS-10534.01.patch
        4 kB
        Kai Sasaki
      2. HDFS-10534.02.patch
        4 kB
        Kai Sasaki
      3. HDFS-10534.03.patch
        7 kB
        Kai Sasaki
      4. HDFS-10534.04.patch
        7 kB
        Kai Sasaki
      5. HDFS-10534.05.patch
        8 kB
        Kai Sasaki
      6. HDFS-10534.06.patch
        211 kB
        Kai Sasaki
      7. HDFS-10534.07.patch
        479 kB
        Kai Sasaki
      8. HDFS-10534.08.patch
        211 kB
        Kai Sasaki
      9. HDFS-10534.09.patch
        211 kB
        Kai Sasaki
      10. HDFS-10534.10.patch
        211 kB
        Kai Sasaki
      11. HDFS-10534.11.patch
        212 kB
        Zhe Zhang
      12. HDFS-10534-branch-2.7-01.patch
        212 kB
        Kai Sasaki
      13. Screen Shot 2016-06-23 at 6.25.50 AM.png
        31 kB
        Kai Sasaki
      14. Screen Shot 2016-07-07 at 23.29.14.png
        249 kB
        Kai Sasaki
      15. Screen Shot 2016-11-14 at 4.27.15 PM.png
        21 kB
        Zhe Zhang
      16. Screen Shot 2016-11-17 at 0.14.06.png
        347 kB
        Kai Sasaki
      17. Screen Shot 2017-01-26 at 9.29.11 AM.png
        23 kB
        Zhe Zhang
      18. table_histogram.html
        2 kB
        Kai Sasaki

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 34s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 45s trunk passed
        +1 javadoc 0m 54s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 196 unchanged - 1 fixed = 198 total (was 197)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        -1 whitespace 0m 1s The patch 1 line(s) with tabs.
        +1 findbugs 1m 52s the patch passed
        +1 javadoc 0m 52s the patch passed
        +1 unit 74m 48s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        94m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811103/HDFS-10534.01.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 92cd648227b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cc6c265
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15797/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15797/artifact/patchprocess/whitespace-tabs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15797/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15797/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 34s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 196 unchanged - 1 fixed = 198 total (was 197) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 9s the patch passed -1 whitespace 0m 1s The patch 1 line(s) with tabs. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 74m 48s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 94m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811103/HDFS-10534.01.patch JIRA Issue HDFS-10534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 92cd648227b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cc6c265 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15797/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15797/artifact/patchprocess/whitespace-tabs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15797/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15797/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai! Patch looks pretty good. However I think we should make the percentile configurable (right now fixed at 95). What's your thought?

        Show
        zhz Zhe Zhang added a comment - Thanks Kai! Patch looks pretty good. However I think we should make the percentile configurable (right now fixed at 95). What's your thought?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 35s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 43s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 58s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 196 unchanged - 1 fixed = 197 total (was 197)
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 1m 4s the patch passed
        -1 unit 82m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        102m 54s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811484/HDFS-10534.02.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f70c6b9eece3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0761379
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15818/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15818/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 35s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 196 unchanged - 1 fixed = 197 total (was 197) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 55s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 82m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 102m 54s Reason Tests Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811484/HDFS-10534.02.patch JIRA Issue HDFS-10534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f70c6b9eece3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0761379 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15818/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15818/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15818/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang Thanks for checking! It sounds reasonable. I think we can make percentile configurable like

        <property>
          <name>dfs.namenode.metrics.node-usage.percentile</name>
          <value>0.95</value>
          <description>
            This setting specifies the percentile value of disk usage of data nodes 
            which is sorted in ascending order. Default is 0.95.
          </description>
        </property>
        
        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang Thanks for checking! It sounds reasonable. I think we can make percentile configurable like <property> <name>dfs.namenode.metrics.node-usage.percentile</name> <value>0.95</value> <description> This setting specifies the percentile value of disk usage of data nodes which is sorted in ascending order. Default is 0.95. </description> </property>
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 49s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 10s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 1m 3s trunk passed
        -1 mvninstall 0m 31s hadoop-hdfs in the patch failed.
        -1 compile 0m 34s hadoop-hdfs in the patch failed.
        -1 javac 0m 34s hadoop-hdfs in the patch failed.
        -1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587)
        -1 mvnsite 0m 25s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        -1 findbugs 0m 24s hadoop-hdfs in the patch failed.
        +1 javadoc 0m 53s the patch passed
        -1 unit 0m 26s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        18m 31s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811711/HDFS-10534.03.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 18dfc493fdeb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0319d73
        Default Java 1.8.0_91
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15828/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15828/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 3s trunk passed -1 mvninstall 0m 31s hadoop-hdfs in the patch failed. -1 compile 0m 34s hadoop-hdfs in the patch failed. -1 javac 0m 34s hadoop-hdfs in the patch failed. -1 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587) -1 mvnsite 0m 25s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 0m 24s hadoop-hdfs in the patch failed. +1 javadoc 0m 53s the patch passed -1 unit 0m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 18m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811711/HDFS-10534.03.patch JIRA Issue HDFS-10534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 18dfc493fdeb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0319d73 Default Java 1.8.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15828/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15828/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 51s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 34s trunk passed
        +1 mvnsite 0m 58s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 1m 0s trunk passed
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 58s the patch passed
        +1 javac 0m 58s the patch passed
        -1 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 1m 0s the patch passed
        -1 unit 69m 15s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        91m 41s



        Reason Tests
        Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
          hadoop.hdfs.server.namenode.TestCacheDirectives



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812191/HDFS-10534.04.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 4248bfa141a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e15cd43
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15846/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15846/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 51s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 1m 0s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 0m 58s the patch passed +1 javac 0m 58s the patch passed -1 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 69m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 91m 41s Reason Tests Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812191/HDFS-10534.04.patch JIRA Issue HDFS-10534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4248bfa141a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e15cd43 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15846/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15846/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15846/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Though checkstyle seems to be broken before attaching this patch, I'll make new method to reduce a method lines.

        Show
        lewuathe Kai Sasaki added a comment - Though checkstyle seems to be broken before attaching this patch, I'll make new method to reduce a method lines.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang Since several configurations in FSNamesystem are final values, it is difficult to extract a method in one meaning because FSNameSystem constructor mainly does set final configurations. I think keeping it as it is can be an option, what do you think?

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang Since several configurations in FSNamesystem are final values, it is difficult to extract a method in one meaning because FSNameSystem constructor mainly does set final configurations. I think keeping it as it is can be an option, what do you think?
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai Sasaki! Latest patch LGTM overall. A few minors:

        1. Can we add a sanity check when loading the dfs.namenode.metrics.node-usage.percentile config? Assert that it is between 0 and 1.
        2. Minor suggestion on the description of dfs.namenode.metrics.node-usage.percentile below. This could be confusing so I think we should be a little more specific.
          This setting specifies the percentile level to report node usage metrics. For example 0.95 means reporting the node usage for the 95th percentile of all DataNodes. If this setting is at 0.95 and the reported node usage is 70%, it means 95% of DataNodes have a usage below 70%.
          
        3. To be consistent with the mathematical definition , maybe we should also called it xth percentile on the html.
          percentileInfo.put("name", StringUtils.format("%dth percentile",
          

        I'm trying to start a NameNode with the patch to verify the new HTML. If you have one ready it'd be great to add a screenshot to the JIRA.

        Thanks for the nice work!

        Show
        zhz Zhe Zhang added a comment - Thanks Kai Sasaki ! Latest patch LGTM overall. A few minors: Can we add a sanity check when loading the dfs.namenode.metrics.node-usage.percentile config? Assert that it is between 0 and 1. Minor suggestion on the description of dfs.namenode.metrics.node-usage.percentile below. This could be confusing so I think we should be a little more specific. This setting specifies the percentile level to report node usage metrics. For example 0.95 means reporting the node usage for the 95th percentile of all DataNodes. If this setting is at 0.95 and the reported node usage is 70%, it means 95% of DataNodes have a usage below 70%. To be consistent with the mathematical definition , maybe we should also called it xth percentile on the html. percentileInfo.put( "name" , StringUtils.format( "%dth percentile" , I'm trying to start a NameNode with the patch to verify the new HTML. If you have one ready it'd be great to add a screenshot to the JIRA. Thanks for the nice work!
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai Sasaki for the update! +1 on v5 patch after Jenkins.

        Show
        zhz Zhe Zhang added a comment - Thanks Kai Sasaki for the update! +1 on v5 patch after Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 48s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 0m 58s trunk passed
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 48s the patch passed
        +1 javac 0m 48s the patch passed
        -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 52s the patch passed
        -1 unit 61m 2s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        82m 6s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.TestRenameWhileOpen



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812826/HDFS-10534.05.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 7fdf97d478e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dca298d
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15896/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15896/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15896/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15896/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 48s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 2 fixed = 587 total (was 587) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 61m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 82m 6s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812826/HDFS-10534.05.patch JIRA Issue HDFS-10534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7fdf97d478e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dca298d Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15896/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15896/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15896/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15896/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai Sasaki for updating the patch and the screenshot. I verified the reported test failures and committed the patch to trunk. Good work!

        Show
        zhz Zhe Zhang added a comment - Thanks Kai Sasaki for updating the patch and the screenshot. I verified the reported test failures and committed the patch to trunk. Good work!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10016 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10016/)
        HDFS-10534. NameNode WebUI should display DataNode usage rate with a (zhz: rev 0424056a77002f4a2334ee2eb240fbc67b676471)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10016 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10016/ ) HDFS-10534 . NameNode WebUI should display DataNode usage rate with a (zhz: rev 0424056a77002f4a2334ee2eb240fbc67b676471) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Hide
        zhz Zhe Zhang added a comment -

        Should this (adding a metric and adding it to web UI) be considered an incompatible change? If not I'd like to backport to 2.6.

        Andrew Wang Vinod Kumar Vavilapalli Any advice? Thanks.

        Show
        zhz Zhe Zhang added a comment - Should this (adding a metric and adding it to web UI) be considered an incompatible change? If not I'd like to backport to 2.6. Andrew Wang Vinod Kumar Vavilapalli Any advice? Thanks.
        Hide
        andrew.wang Andrew Wang added a comment -

        Adding a new metric is compatible, but upon looking at this patch, I had a few questions.

        • Why not present a histogram rather than a single threshold like this? That way we don't add a new config, present more info, and don't require a restart to change this threshold.
        • This is also a metric that could be calculated in client-side JS from existing information.
        • Finally, the config says it's a percentile, but it's really a quantile. Percentile are 0-100, quantiles are 0-1.0. Seems like the precondition check should also be inclusive of 0 and 1.0, since they are valid quantiles.

        I'd prefer if we backed out this change while we thought about these issues. Thanks folks.

        Show
        andrew.wang Andrew Wang added a comment - Adding a new metric is compatible, but upon looking at this patch, I had a few questions. Why not present a histogram rather than a single threshold like this? That way we don't add a new config, present more info, and don't require a restart to change this threshold. This is also a metric that could be calculated in client-side JS from existing information. Finally, the config says it's a percentile, but it's really a quantile. Percentile are 0-100, quantiles are 0-1.0. Seems like the precondition check should also be inclusive of 0 and 1.0, since they are valid quantiles. I'd prefer if we backed out this change while we thought about these issues. Thanks folks.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Andrew. I just reverted the change.

        Why not present a histogram rather than a single threshold like this? That way we don't add a new config, present more info, and don't require a restart to change this threshold.

        In our case we are mostly interested in the 95th percentile because it serves as an alarm that 5% DNs are becoming hot nodes and will likely cause job failures. A histogram is a nice idea actually. We can think about an appropriate granularity (e.g. every 5%?) for it. The only drawback is that it will add more content to NN web UI and make it busier – I imagine it will a table.

        This is also a metric that could be calculated in client-side JS from existing information.

        True. But I think showing on NN web UI is more convenient for admins. We proposed the change because median (50th percentile) is actually a poor metric to illustrate imbalance level; especially in a busy cluster with say > 70% overall utilization. We therefore wanted a "better median".

        the config says it's a percentile, but it's really a quantile.

        Good catch. We could change the config to be a real percentile to be b/w 0 and 100. Per above, we could also show a histogram instead.

        So overall I like the histogram idea. Kai Sasaki What are you thoughts?

        Show
        zhz Zhe Zhang added a comment - Thanks Andrew. I just reverted the change. Why not present a histogram rather than a single threshold like this? That way we don't add a new config, present more info, and don't require a restart to change this threshold. In our case we are mostly interested in the 95th percentile because it serves as an alarm that 5% DNs are becoming hot nodes and will likely cause job failures. A histogram is a nice idea actually. We can think about an appropriate granularity (e.g. every 5%?) for it. The only drawback is that it will add more content to NN web UI and make it busier – I imagine it will a table. This is also a metric that could be calculated in client-side JS from existing information. True. But I think showing on NN web UI is more convenient for admins. We proposed the change because median (50th percentile) is actually a poor metric to illustrate imbalance level; especially in a busy cluster with say > 70% overall utilization. We therefore wanted a "better median". the config says it's a percentile, but it's really a quantile. Good catch. We could change the config to be a real percentile to be b/w 0 and 100. Per above, we could also show a histogram instead. So overall I like the histogram idea. Kai Sasaki What are you thoughts?
        Hide
        lewuathe Kai Sasaki added a comment -

        Thank you so much Andrew Wang and Zhe Zhang for reviewing!

        Histgram sounds good idea which enables us to make own metrics flexibly. But I think percentile metrics which is provided by NN is also useful because it is provided through JMX api. This way seems simple access to get node usage metrics when external system tries to get it.

        So I think we can make NN provides percentile metrics here (Of course I'll fix configuration naming issue) and also implement histogram UI in another JIRA. What do you think about?

        Show
        lewuathe Kai Sasaki added a comment - Thank you so much Andrew Wang and Zhe Zhang for reviewing! Histgram sounds good idea which enables us to make own metrics flexibly. But I think percentile metrics which is provided by NN is also useful because it is provided through JMX api. This way seems simple access to get node usage metrics when external system tries to get it. So I think we can make NN provides percentile metrics here (Of course I'll fix configuration naming issue) and also implement histogram UI in another JIRA. What do you think about?
        Hide
        andrew.wang Andrew Wang added a comment -

        This is also a metric that could be calculated in client-side JS from existing information.

        To clarify this a bit more, the NN webUI is mostly built in JS right now. Is it possible to calculate the histogram in JS as well? Then we don't need to add this new metric.

        I imagine that other monitoring tools consuming JMX are already doing similar things for their alerting thresholds, since the raw data is already available. The metric added in this patch doesn't expose more information the same way our RPC latency histograms do, which limits its utility. Monitoring tools probably already have business logic for defining these alerting thresholds, and doing it there is more expressive too.

        Show
        andrew.wang Andrew Wang added a comment - This is also a metric that could be calculated in client-side JS from existing information. To clarify this a bit more, the NN webUI is mostly built in JS right now. Is it possible to calculate the histogram in JS as well? Then we don't need to add this new metric. I imagine that other monitoring tools consuming JMX are already doing similar things for their alerting thresholds, since the raw data is already available. The metric added in this patch doesn't expose more information the same way our RPC latency histograms do, which limits its utility. Monitoring tools probably already have business logic for defining these alerting thresholds, and doing it there is more expressive too.
        Hide
        lewuathe Kai Sasaki added a comment -

        Andrew Wang Thanks for comment.
        I think it is possible to calculate the histogram because NN provides UI which shows the usage of each DN. So I'll try to update a patch to build histogram on JS side. And also I'll update the title of this JIRA to fit current motivation.

        Show
        lewuathe Kai Sasaki added a comment - Andrew Wang Thanks for comment. I think it is possible to calculate the histogram because NN provides UI which shows the usage of each DN. So I'll try to update a patch to build histogram on JS side. And also I'll update the title of this JIRA to fit current motivation.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang I implemented node usage histogram UI and attached screen shot of the view.
        Could you give me some feedback if you are available? Thanks.

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang I implemented node usage histogram UI and attached screen shot of the view. Could you give me some feedback if you are available? Thanks.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai Sasaki! Screenshot looks very nice.

        On the first glance the js file added in the patch is very large. Is it necessary? I haven't actually applied and reviewed it, just a quick comment.

        Show
        zhz Zhe Zhang added a comment - Thanks Kai Sasaki ! Screenshot looks very nice. On the first glance the js file added in the patch is very large. Is it necessary? I haven't actually applied and reviewed it, just a quick comment.
        Hide
        zhz Zhe Zhang added a comment -

        So the v6 patch doesn't apply on current trunk. I applied d3.min.js but it is not formatted. Also why is the name d3.min? Thanks.

        Show
        zhz Zhe Zhang added a comment - So the v6 patch doesn't apply on current trunk. I applied d3.min.js but it is not formatted. Also why is the name d3.min ? Thanks.
        Hide
        lewuathe Kai Sasaki added a comment -

        d3.min.js is required to render histogram. Since the file is minified, the name is d3.min.
        But if we can use non-minified version of d3.js, it can be formatted and will be applied. I'll try to replace it.

        Show
        lewuathe Kai Sasaki added a comment - d3.min.js is required to render histogram. Since the file is minified, the name is d3.min . But if we can use non-minified version of d3.js, it can be formatted and will be applied. I'll try to replace it.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang I updated the patch to use d3.js and confirmed it can be applied current trunk in my environment.
        Could you check this when you get a chance? Thanks.

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang I updated the patch to use d3.js and confirmed it can be applied current trunk in my environment. Could you check this when you get a chance? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 asflicense 0m 21s The patch generated 1 ASF License warnings.
        0m 49s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816827/HDFS-10534.07.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense
        uname Linux fa3e39e6ad19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 30ee57c
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16007/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16007/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. -1 asflicense 0m 21s The patch generated 1 ASF License warnings. 0m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816827/HDFS-10534.07.patch JIRA Issue HDFS-10534 Optional Tests asflicense uname Linux fa3e39e6ad19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 30ee57c asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16007/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16007/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Note that if we're adding another dependency, it needs to be added to LICENSE and NOTICE. We already have a copy of d3 in hadoop-sls, so this shouldn't be too bad. I'd also prefer that we use the minified version, and put the version in the filename.

        If we wanted to drop the d3 dependency, I think we could calculate the histogram ourselves and render it as a table. My preference is to keep the webUI lean if possible. It might be that bootstrap, jquery, or dust (our existing dependencies) also have nice ways of making a histogram.

        Show
        andrew.wang Andrew Wang added a comment - Note that if we're adding another dependency, it needs to be added to LICENSE and NOTICE. We already have a copy of d3 in hadoop-sls, so this shouldn't be too bad. I'd also prefer that we use the minified version, and put the version in the filename. If we wanted to drop the d3 dependency, I think we could calculate the histogram ourselves and render it as a table. My preference is to keep the webUI lean if possible. It might be that bootstrap, jquery, or dust (our existing dependencies) also have nice ways of making a histogram.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 asflicense 0m 23s The patch generated 1 ASF License warnings.
        0m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816827/HDFS-10534.07.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense
        uname Linux 9929880e0a8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 932aed6
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16013/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16013/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. -1 asflicense 0m 23s The patch generated 1 ASF License warnings. 0m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816827/HDFS-10534.07.patch JIRA Issue HDFS-10534 Optional Tests asflicense uname Linux 9929880e0a8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 932aed6 asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16013/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16013/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Andrew Wang Thanks for comment!
        I prototyped a histogram rendered by table (table_histogram.html, and it looks well.
        But when we need to render dynamically, it's a little hacky and hard to maintain because jQuery and dust does not support rendering histogram natively. We need to maintain table rendering logic for various type of browser platforms for future.
        So I think since d3.js is already added as dependency in hadoop-sls, we can use also it in hadoop-hdfs. And it will reduce the cost of maintenance and the risk of missing support of browser platform.

        Show
        lewuathe Kai Sasaki added a comment - Andrew Wang Thanks for comment! I prototyped a histogram rendered by table ( table_histogram.html , and it looks well. But when we need to render dynamically, it's a little hacky and hard to maintain because jQuery and dust does not support rendering histogram natively. We need to maintain table rendering logic for various type of browser platforms for future. So I think since d3.js is already added as dependency in hadoop-sls , we can use also it in hadoop-hdfs . And it will reduce the cost of maintenance and the risk of missing support of browser platform.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 asflicense 1m 53s The patch generated 1 ASF License warnings.
        2m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817415/HDFS-10534.08.patch
        JIRA Issue HDFS-10534
        Optional Tests asflicense
        uname Linux f2493c4d6319 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ad256be
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16034/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16034/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. -1 asflicense 1m 53s The patch generated 1 ASF License warnings. 2m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817415/HDFS-10534.08.patch JIRA Issue HDFS-10534 Optional Tests asflicense uname Linux f2493c4d6319 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad256be asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16034/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16034/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the work Kai Sasaki! I'm checking with the dust team at LinkedIn for advice of generating histogram. If it cannot be easily done in dust / jQuery, I think using d3 here is a reasonable option.

        Show
        zhz Zhe Zhang added a comment - Thanks for the work Kai Sasaki ! I'm checking with the dust team at LinkedIn for advice of generating histogram. If it cannot be easily done in dust / jQuery, I think using d3 here is a reasonable option.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang Oh, you are next to dust team! Thanks!

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang Oh, you are next to dust team! Thanks!
        Hide
        zhz Zhe Zhang added a comment - - edited

        Thanks Kai again for the work, and sorry for the late review. I actually didn't find an easier way to generate histogram from dust.

        The histogram looks pretty nice! I only have 2 minot concerns:

        1. It is a little too wide and requires scrolling to the right to view the full picture. Meanwhile, i think each bar can be narrower which should save space to display the whole picture.
        2. Can we have 10% etc. instead of 0.1 etc?

        +1 pending the above. Thanks.

        Show
        zhz Zhe Zhang added a comment - - edited Thanks Kai again for the work, and sorry for the late review. I actually didn't find an easier way to generate histogram from dust. The histogram looks pretty nice! I only have 2 minot concerns: It is a little too wide and requires scrolling to the right to view the full picture. Meanwhile, i think each bar can be narrower which should save space to display the whole picture. Can we have 10% etc. instead of 0.1 etc? +1 pending the above. Thanks.
        Hide
        zhz Zhe Zhang added a comment -

        Screenshot of the web UI. I have to scroll to the right to see the part beyond 0.7.

        Show
        zhz Zhe Zhang added a comment - Screenshot of the web UI. I have to scroll to the right to see the part beyond 0.7.
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang Thanks for checking. I updated the patch and screenshot. Could you review that again?

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang Thanks for checking. I updated the patch and screenshot. Could you review that again?
        Hide
        lewuathe Kai Sasaki added a comment - - edited

        Zhe Zhang sorry for bothering you. Could you check updated patch if you are available?

        Show
        lewuathe Kai Sasaki added a comment - - edited Zhe Zhang sorry for bothering you. Could you check updated patch if you are available?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 asflicense 1m 57s The patch generated 1 ASF License warnings.
        2m 27s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10534
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848516/HDFS-10534.10.patch
        Optional Tests asflicense
        uname Linux 7ce7d053787f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e015b56
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18225/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18225/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. -1 asflicense 1m 57s The patch generated 1 ASF License warnings. 2m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10534 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848516/HDFS-10534.10.patch Optional Tests asflicense uname Linux 7ce7d053787f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e015b56 asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18225/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18225/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 asflicense 1m 59s The patch generated 1 ASF License warnings.
        2m 40s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10534
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848516/HDFS-10534.10.patch
        Optional Tests asflicense
        uname Linux 964bb5cd1c21 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d79c645
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18228/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18228/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. -1 asflicense 1m 59s The patch generated 1 ASF License warnings. 2m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10534 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848516/HDFS-10534.10.patch Optional Tests asflicense uname Linux 964bb5cd1c21 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d79c645 asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18228/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18228/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai! And I apologize for the late response. +1 on v10 patch pending a valid Jenkins run. I think the issue was with the license of the js file. How do we usually handle the license of those 3rd party libraries?

        Show
        zhz Zhe Zhang added a comment - Thanks Kai! And I apologize for the late response. +1 on v10 patch pending a valid Jenkins run. I think the issue was with the license of the js file. How do we usually handle the license of those 3rd party libraries?
        Hide
        lewuathe Kai Sasaki added a comment -

        Zhe Zhang Thanks! 3rd JavaScript libraries used in HDFS are not modified about license header. So we can add d3.js file as it is with its header because it is released under modified BSD license.

        Show
        lewuathe Kai Sasaki added a comment - Zhe Zhang Thanks! 3rd JavaScript libraries used in HDFS are not modified about license header. So we can add d3.js file as it is with its header because it is released under modified BSD license .
        Hide
        zhz Zhe Zhang added a comment -

        Adding the js file to the pom file excluded list in v11 patch.

        Show
        zhz Zhe Zhang added a comment - Adding the js file to the pom file excluded list in v11 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 36s trunk passed
        +1 compile 0m 47s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 47s the patch passed
        +1 javac 0m 47s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 javadoc 0m 37s the patch passed
        +1 unit 77m 1s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        98m 19s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10534
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848984/HDFS-10534.11.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 289c1f15c7e1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ce83205
        Default Java 1.8.0_121
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18242/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18242/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 36s trunk passed +1 compile 0m 47s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 37s the patch passed +1 unit 77m 1s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 98m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10534 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848984/HDFS-10534.11.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 289c1f15c7e1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ce83205 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18242/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18242/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Pre-commit is clean. Andrew Wang Could you take a look? I plan to commit tomorrow.

        Show
        zhz Zhe Zhang added a comment - Pre-commit is clean. Andrew Wang Could you take a look? I plan to commit tomorrow.
        Hide
        zhz Zhe Zhang added a comment -

        I committed the patch to trunk, branch-2, and branch-2.8. Backport to branch-2.7 has conflicts due to HDFS-6407. I'm trying to figure out whether that is a valid improvement for branch-2.7.

        Show
        zhz Zhe Zhang added a comment - I committed the patch to trunk, branch-2, and branch-2.8. Backport to branch-2.7 has conflicts due to HDFS-6407 . I'm trying to figure out whether that is a valid improvement for branch-2.7.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11170 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11170/)
        HDFS-10534. NameNode WebUI should display DataNode usage histogram. (zhz: rev 18e1d6820926646999e7ec248c504b4145cf1a76)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-v4.1.1.min.js
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11170 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11170/ ) HDFS-10534 . NameNode WebUI should display DataNode usage histogram. (zhz: rev 18e1d6820926646999e7ec248c504b4145cf1a76) (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/hadoop.css (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js (add) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-v4.1.1.min.js
        Hide
        lewuathe Kai Sasaki added a comment - - edited

        Zhe Zhang Thanks for reviewing! It depends on DataNode metrics so I can rebase and resolve conflict if DataNode provides these metrics too in 2.7.x. And I think so.

        Show
        lewuathe Kai Sasaki added a comment - - edited Zhe Zhang Thanks for reviewing! It depends on DataNode metrics so I can rebase and resolve conflict if DataNode provides these metrics too in 2.7.x. And I think so.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai! If you could provide a branch-2.7 patch that'd be great. I tried backporting HDFS-6407 but that depends on HDFS-8816 and that's too much change to backport. If we can only have this histogram change in 2.7 that's ideal.

        Show
        zhz Zhe Zhang added a comment - Thanks Kai! If you could provide a branch-2.7 patch that'd be great. I tried backporting HDFS-6407 but that depends on HDFS-8816 and that's too much change to backport. If we can only have this histogram change in 2.7 that's ideal.
        Hide
        lewuathe Kai Sasaki added a comment -

        Sure, I'll try to create a patch for branch-2.7.

        Show
        lewuathe Kai Sasaki added a comment - Sure, I'll try to create a patch for branch-2.7.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 47s branch-2.7 passed
        +1 compile 1m 0s branch-2.7 passed with JDK v1.8.0_121
        +1 compile 1m 1s branch-2.7 passed with JDK v1.7.0_121
        +1 mvnsite 0m 59s branch-2.7 passed
        +1 mvneclipse 0m 16s branch-2.7 passed
        +1 javadoc 0m 58s branch-2.7 passed with JDK v1.8.0_121
        +1 javadoc 1m 39s branch-2.7 passed with JDK v1.7.0_121
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 57s the patch passed with JDK v1.8.0_121
        +1 javac 0m 57s the patch passed
        +1 compile 0m 59s the patch passed with JDK v1.7.0_121
        +1 javac 0m 59s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 3744 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 1m 19s The patch 52 line(s) with tabs.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121
        +1 javadoc 1m 36s the patch passed with JDK v1.7.0_121
        -1 unit 68m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
        -1 asflicense 0m 24s The patch generated 3 ASF License warnings.
        151m 44s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.web.TestWebHdfsFileSystemContract
          hadoop.hdfs.server.datanode.TestBlockReplacement
        JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestClientProtocolForPipelineRecovery



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue HDFS-10534
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849490/HDFS-10534-branch-2.7-01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux e5b004d613af 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / ab62484
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18271/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18271/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 47s branch-2.7 passed +1 compile 1m 0s branch-2.7 passed with JDK v1.8.0_121 +1 compile 1m 1s branch-2.7 passed with JDK v1.7.0_121 +1 mvnsite 0m 59s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed +1 javadoc 0m 58s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 1m 39s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 53s the patch passed +1 compile 0m 57s the patch passed with JDK v1.8.0_121 +1 javac 0m 57s the patch passed +1 compile 0m 59s the patch passed with JDK v1.7.0_121 +1 javac 0m 59s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 3744 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 19s The patch 52 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 36s the patch passed with JDK v1.7.0_121 -1 unit 68m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 24s The patch generated 3 ASF License warnings. 151m 44s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.web.TestWebHdfsFileSystemContract   hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestClientProtocolForPipelineRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10534 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849490/HDFS-10534-branch-2.7-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux e5b004d613af 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / ab62484 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18271/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18271/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18271/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Kai! The branch-2.7 patch LGTM. I just committed it. Attaching the branch-2.7 screenshot. I also verified all reported unit test failures; cannot reproduce locally.

        Show
        zhz Zhe Zhang added a comment - Thanks Kai! The branch-2.7 patch LGTM. I just committed it. Attaching the branch-2.7 screenshot. I also verified all reported unit test failures; cannot reproduce locally.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            lewuathe Kai Sasaki
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development