Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, ui
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      DataNode Web UI has been improved with new HTML5 page, showing useful information.

      Description

      At present, datanode web UI doesn't have much information except for node name and port. Propose to add more information similar to namenode UI, including,

      • Static info (version, block pool and cluster ID)
      • Block pools info (BP IDs, namenode address, actor states)
      • Storage info (Volumes, capacity used, reserved, left)
      • Utilities (logs)
      1. dn_web_ui_mockup.jpg
        85 kB
        Weiwei Yang
      2. datanode_utilities.001.jpg
        23 kB
        Weiwei Yang
      3. HDFS-10440.001.patch
        8 kB
        Weiwei Yang
      4. datanode_html.001.jpg
        44 kB
        Weiwei Yang
      5. datanode_loading_err.002.jpg
        17 kB
        Weiwei Yang
      6. datanode_2nns.html.002.jpg
        78 kB
        Weiwei Yang
      7. datanode_utilities.002.jpg
        31 kB
        Weiwei Yang
      8. HDFS-10440.002.patch
        12 kB
        Weiwei Yang
      9. nn_dfs_storage_types.jpg
        20 kB
        Weiwei Yang
      10. HDFS-10440.003.patch
        14 kB
        Weiwei Yang
      11. dn_web_ui.003.jpg
        68 kB
        Weiwei Yang
      12. HDFS-10440.004.patch
        15 kB
        Weiwei Yang
      13. HDFS-10440.005.patch
        16 kB
        Weiwei Yang
      14. HDFS-10440.006.patch
        17 kB
        Weiwei Yang
      15. HDFS-10440.007.patch
        17 kB
        Weiwei Yang
      16. HDFS-10440.008.patch
        17 kB
        Weiwei Yang
      17. HDFS-10440.009.patch
        17 kB
        Weiwei Yang

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          Running state (active, decommissioning, decommissioned or lost etc)

          These are only known to NN. I wouldn't try to make it known to DN just for this.
          It will be useful to display configured block pools along with each service actor's state (e.g. nnAddr, runningState, etc)
          Showing per storage status will be most important.

          Show
          kihwal Kihwal Lee added a comment - Running state (active, decommissioning, decommissioned or lost etc) These are only known to NN. I wouldn't try to make it known to DN just for this. It will be useful to display configured block pools along with each service actor's state (e.g. nnAddr, runningState, etc) Showing per storage status will be most important.
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks Kihwal Lee I agree. I just attached some UI mockups, let me know if you have any comments.

          Show
          cheersyang Weiwei Yang added a comment - Thanks Kihwal Lee I agree. I just attached some UI mockups, let me know if you have any comments.
          Hide
          vinayrpet Vinayakumar B added a comment -

          As pointed out by Kihwal Lee, I think you can put blockpools' information in a separate table, because There will be multiple block pools with their own namenode actor threads.

          Show
          vinayrpet Vinayakumar B added a comment - As pointed out by Kihwal Lee , I think you can put blockpools' information in a separate table, because There will be multiple block pools with their own namenode actor threads.
          Hide
          cheersyang Weiwei Yang added a comment -

          Vinayakumar B Sure, thanks. I just attached a new mockup dn_web_ui_mockup.jpg, please take a look if that looks OK?

          Show
          cheersyang Weiwei Yang added a comment - Vinayakumar B Sure, thanks. I just attached a new mockup dn_web_ui_mockup.jpg , please take a look if that looks OK?
          Hide
          cheersyang Weiwei Yang added a comment -

          I have a patch ready to add datanode UI with basic information, including block pools and storage. Please check datanode_html.001.jpg and datanode_utilities.001.jpg. The patch can be applied to both trunk and branch-2. This patch is created based on existing datanode JMX, I think it's a good place to start.

          Show
          cheersyang Weiwei Yang added a comment - I have a patch ready to add datanode UI with basic information, including block pools and storage. Please check datanode_html.001.jpg and datanode_utilities.001.jpg . The patch can be applied to both trunk and branch-2. This patch is created based on existing datanode JMX, I think it's a good place to start.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          0m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807072/HDFS-10440.001.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense
          uname Linux 6cce37dd588b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 93d8a7f
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15614/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 0m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807072/HDFS-10440.001.patch JIRA Issue HDFS-10440 Optional Tests asflicense uname Linux 6cce37dd588b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 93d8a7f modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15614/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment - - edited

          Thanks for the patch. I think it is showing the available information very well. Having said that, we can take this opportunity to expose more on the block pool via jmx. The namenode addresses are useful, but showing the service actor state will be even better. Sometimes datanodes have trouble talking to some namenodes, but not all. Verifying it usually involves looking at the log. Exposing individual BP service actor state through jmx and showing them through UI will be very helpful.

          For the storage section, VolumeInfo in trunk/2.9/2.8 already contains reservedSpaceForReplicas (HDFS-6955) and numBlocks (HDFS-9425). Please verify (screenshot?) they appear on the web ui.

          Show
          kihwal Kihwal Lee added a comment - - edited Thanks for the patch. I think it is showing the available information very well. Having said that, we can take this opportunity to expose more on the block pool via jmx. The namenode addresses are useful, but showing the service actor state will be even better. Sometimes datanodes have trouble talking to some namenodes, but not all. Verifying it usually involves looking at the log. Exposing individual BP service actor state through jmx and showing them through UI will be very helpful. For the storage section, VolumeInfo in trunk/2.9/2.8 already contains reservedSpaceForReplicas ( HDFS-6955 ) and numBlocks ( HDFS-9425 ). Please verify (screenshot?) they appear on the web ui.
          Hide
          cheersyang Weiwei Yang added a comment -

          Sure Kihwal Lee I will add Reserved Space for Replicas and Blocks in storage section. And I will add another column in Block Pool section, to indicate the actor state, display the state of BPServiceActor.RunningState. Will upload a patch with screen shots shortly after I tested on a trunk build.

          Thanks a lot for suggestions.

          Show
          cheersyang Weiwei Yang added a comment - Sure Kihwal Lee I will add Reserved Space for Replicas and Blocks in storage section. And I will add another column in Block Pool section, to indicate the actor state, display the state of BPServiceActor.RunningState. Will upload a patch with screen shots shortly after I tested on a trunk build. Thanks a lot for suggestions.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Patch updates:

          • Added ReservedSpaceForReplicas and NumBlocks in Volume Information section
          • Added per-BP-actor state in Block Pools section
          • Minor modification to DataNodeMXBean.getNamenodeAddresses to include actor states
          • Added error handling when it's unable to retrieve info from JMX call

          New screen shots attached, they are taken from latest trunk build, I setup 2 NNs and the datanode UI looks like datanode_2nns.html.002.jpg, when it fails to load info from datanode JMX, it shows like datanode_loading_err.002.jpg. And by clicking Utilities, it shows the link to view configurations and logs datanode_utilities.002.jpg.

          Please kindly review and comment, thank you.

          Show
          cheersyang Weiwei Yang added a comment - - edited Patch updates: Added ReservedSpaceForReplicas and NumBlocks in Volume Information section Added per-BP-actor state in Block Pools section Minor modification to DataNodeMXBean.getNamenodeAddresses to include actor states Added error handling when it's unable to retrieve info from JMX call New screen shots attached, they are taken from latest trunk build, I setup 2 NNs and the datanode UI looks like datanode_2nns.html.002.jpg , when it fails to load info from datanode JMX, it shows like datanode_loading_err.002.jpg . And by clicking Utilities, it shows the link to view configurations and logs datanode_utilities.002.jpg . Please kindly review and comment, thank you.
          Hide
          kihwal Kihwal Lee added a comment -

          The patch looks good, waiting for the precommit to come back. I flagged it as incompatible change as the jmx format changed.
          It will be nice if we add links to datanodes in the namenode datanodes page, but that can be done in a separate jira.

          Show
          kihwal Kihwal Lee added a comment - The patch looks good, waiting for the precommit to come back. I flagged it as incompatible change as the jmx format changed. It will be nice if we add links to datanodes in the namenode datanodes page, but that can be done in a separate jira.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 16s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 1m 9s trunk passed
          +1 mvninstall 1m 1s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 57s the patch passed
          +1 javadoc 1m 9s the patch passed
          -1 unit 79m 13s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          100m 55s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808395/HDFS-10440.002.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 44c7c9d3fa6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 35f255b
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15661/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15661/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15661/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15661/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 9s trunk passed +1 mvninstall 1m 1s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 1m 9s the patch passed -1 unit 79m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 100m 55s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808395/HDFS-10440.002.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 44c7c9d3fa6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 35f255b Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15661/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15661/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15661/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15661/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          I don't think the 2 failed tests are related, I can run them successfully locally,

          -------------------------------------------------------------------------------
          Test set: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          -------------------------------------------------------------------------------
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.011 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager

          -------------------------------------------------------------------------------
          Test set: org.apache.hadoop.hdfs.TestRollingUpgrade
          -------------------------------------------------------------------------------
          Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.713 sec - in org.apache.hadoop.hdfs.TestRollingUpgrade

          Found an opening JIRA that similar to 2nd issue : HDFS-9664.

          Show
          cheersyang Weiwei Yang added a comment - I don't think the 2 failed tests are related, I can run them successfully locally, ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager ------------------------------------------------------------------------------- Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.011 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hdfs.TestRollingUpgrade ------------------------------------------------------------------------------- Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.713 sec - in org.apache.hadoop.hdfs.TestRollingUpgrade Found an opening JIRA that similar to 2nd issue : HDFS-9664 .
          Hide
          cheersyang Weiwei Yang added a comment -

          Kihwal Lee Yep, that makes sense to me, I just created a new JIRA HDFS-10493 to add links (Like what old UI provides).

          Show
          cheersyang Weiwei Yang added a comment - Kihwal Lee Yep, that makes sense to me, I just created a new JIRA HDFS-10493 to add links (Like what old UI provides).
          Hide
          vinayrpet Vinayakumar B added a comment -

          Commenting after seeing the HTML, didnt look at the patch yet.

          1. HTML page looks nice.
          2. I think, it would be better to show the relative time of last heartbeat sent and absolute time of last block report sent per namenode.
          3. In Volume stats, "Configured Capacity" also could be added as first column.
          4. Redundent 'Capacity' word could be removed from remaining column headers to make the columns smaller. (I am also fine, if you insist to keep it)

          Show
          vinayrpet Vinayakumar B added a comment - Commenting after seeing the HTML, didnt look at the patch yet. 1. HTML page looks nice. 2. I think, it would be better to show the relative time of last heartbeat sent and absolute time of last block report sent per namenode. 3. In Volume stats, "Configured Capacity" also could be added as first column. 4. Redundent 'Capacity' word could be removed from remaining column headers to make the columns smaller. (I am also fine, if you insist to keep it)
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Shreyas Vinayakumar

          Thanks for your comments. Please see my response below

          it would be better to show the relative time of last heartbeat sent and absolute time of last block report sent per namenode.

          Yes, however they are not exposed via DN JMX now, so it would require more changes. If necessary, we can address that in another JIRA probably?

          "Configured Capacity" also could be added as first column.

          Similar to 2, "Configured Capacity" is not available from DN JMX yet.

          Redundent 'Capacity' word could be removed

          I'd prefer to keep it, to be consistent with namenode UI (DFS Storage Types)

          Let me know if this makes sense. Thank you.

          Show
          cheersyang Weiwei Yang added a comment - Hello Shreyas Vinayakumar Thanks for your comments. Please see my response below it would be better to show the relative time of last heartbeat sent and absolute time of last block report sent per namenode. Yes, however they are not exposed via DN JMX now, so it would require more changes. If necessary, we can address that in another JIRA probably? "Configured Capacity" also could be added as first column. Similar to 2, "Configured Capacity" is not available from DN JMX yet. Redundent 'Capacity' word could be removed I'd prefer to keep it, to be consistent with namenode UI (DFS Storage Types) Let me know if this makes sense. Thank you.
          Hide
          kihwal Kihwal Lee added a comment -

          Vinayakumar B, I agree showing the last heartbeat and block report time will be very useful. Weiwei Yang, this might not be too much work since you have already added getActorInfoMap().

          Show
          kihwal Kihwal Lee added a comment - Vinayakumar B , I agree showing the last heartbeat and block report time will be very useful. Weiwei Yang , this might not be too much work since you have already added getActorInfoMap() .
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Kihwal Lee Vinayakumar B

          I just uploaded 003 patch, this patch includes following updates

          • Record last heartbeat and block report time in BPServiceActor
          • Add last heartbeat and block report time in actor state table on DN UI
          • Use relative time and human readable format to display the time, for block report time, it shows from "a few seconds", "a minute", "n minutes", "n hours" ...

          The screen shot on latest trunk is dn_web_ui.003.jpg Please let me know if this looks good, thanks

          Show
          cheersyang Weiwei Yang added a comment - Hello Kihwal Lee Vinayakumar B I just uploaded 003 patch, this patch includes following updates Record last heartbeat and block report time in BPServiceActor Add last heartbeat and block report time in actor state table on DN UI Use relative time and human readable format to display the time, for block report time, it shows from "a few seconds", "a minute", "n minutes", "n hours" ... The screen shot on latest trunk is dn_web_ui.003.jpg Please let me know if this looks good, thanks
          Hide
          vinayrpet Vinayakumar B added a comment -

          I was thinking about incompatible change done in DataNodeMXBean#getNamenodeAddresses(). Patch returns more information than just addresses.
          Instead of breaking compatibility, how about adding one more metric entry, ex: getNameNodesInfo(), this way we would keep the compatibility also, i.e. Adding a new JMX entry is compatible mentioned here.
          Address information is redundant, but better than breaking compatibility.
          Kihwal Lee, what you think?

          Show
          vinayrpet Vinayakumar B added a comment - I was thinking about incompatible change done in DataNodeMXBean#getNamenodeAddresses() . Patch returns more information than just addresses. Instead of breaking compatibility, how about adding one more metric entry, ex: getNameNodesInfo() , this way we would keep the compatibility also, i.e. Adding a new JMX entry is compatible mentioned here . Address information is redundant, but better than breaking compatibility. Kihwal Lee , what you think?
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Vinayakumar B

          That's a good point, I just modified the patch to avoid of breaking compatibility, just uploaded v4 patch . I added a new entry getBPServiceActorInfo() in DatanodeMXBean, I tend to use this instead of getNameNodesInfo because it sounds more accurate of the information it returns, but I am open for suggestions . Please help to review and let me know if this is good to you.

          Thanks

          Show
          cheersyang Weiwei Yang added a comment - Hello Vinayakumar B That's a good point, I just modified the patch to avoid of breaking compatibility, just uploaded v4 patch . I added a new entry getBPServiceActorInfo() in DatanodeMXBean, I tend to use this instead of getNameNodesInfo because it sounds more accurate of the information it returns, but I am open for suggestions . Please help to review and let me know if this is good to you. Thanks
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 17s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 208 unchanged - 0 fixed = 213 total (was 208)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 52s the patch passed
          -1 unit 56m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          75m 0s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Unread field:BPServiceActor.java:[line 1022]
          Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811327/HDFS-10440.004.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f727a1380f11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 09e82ac
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15804/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15804/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 17s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 40s the patch passed -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 208 unchanged - 0 fixed = 213 total (was 208) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 52s the patch passed -1 unit 56m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 75m 0s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Unread field:BPServiceActor.java: [line 1022] Failed junit tests hadoop.hdfs.TestAsyncHDFSWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811327/HDFS-10440.004.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f727a1380f11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 09e82ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15804/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15804/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15804/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment - - edited

          Address information is redundant, but better than breaking compatibility.
          Kihwal Lee, what you think?

          I think that's safer and preferred.

          Looking at the patch, the organization looks good. Is the last block report time going to be in the local time of the browser?

          Show
          kihwal Kihwal Lee added a comment - - edited Address information is redundant, but better than breaking compatibility. Kihwal Lee, what you think? I think that's safer and preferred. Looking at the patch, the organization looks good. Is the last block report time going to be in the local time of the browser?
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Kihwal Lee

          JMX returns time passed from last block report in milliseconds, and then the JS calculates the relative time on browser. So even the clock is not sync between datanode and browser, it won't be an issue.

          I just uploaded v5 patch, including following updates

          • Fix checkstyle and findbug issues
          • Add a test to test the new JMX entry
          Show
          cheersyang Weiwei Yang added a comment - Hello Kihwal Lee JMX returns time passed from last block report in milliseconds, and then the JS calculates the relative time on browser. So even the clock is not sync between datanode and browser, it won't be an issue. I just uploaded v5 patch, including following updates Fix checkstyle and findbug issues Add a test to test the new JMX entry
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 29s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 25s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 58m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          76m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
            hadoop.hdfs.TestDFSClientExcludedNodes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811406/HDFS-10440.005.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 74359b20fccf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2800695
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15809/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15809/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 29s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 58m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 76m 48s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics   hadoop.hdfs.TestDFSClientExcludedNodes Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811406/HDFS-10440.005.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74359b20fccf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15809/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15809/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          The 2 failed tests are not related to this patch, Kihwal Lee can you help to review the patch? Many thanks.

          Show
          cheersyang Weiwei Yang added a comment - The 2 failed tests are not related to this patch, Kihwal Lee can you help to review the patch? Many thanks.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Kihwal Lee Vinayakumar B

          Appreciate if you guys can help to review this (v5 patch), thanks a lot.

          Show
          cheersyang Weiwei Yang added a comment - Hello Kihwal Lee Vinayakumar B Appreciate if you guys can help to review this (v5 patch), thanks a lot.
          Hide
          kihwal Kihwal Lee added a comment -

          It looks good to me. +1 pending Vinay's review since he asked for the last set of changes.

          Show
          kihwal Kihwal Lee added a comment - It looks good to me. +1 pending Vinay's review since he asked for the last set of changes.
          Hide
          vinayrpet Vinayakumar B added a comment -

          I was having some problem to connect to DataNode's HTTP port in my eclipse run in windows. Seems to be problem with PortUnificationHandler. I was able to connect to proxy port. Anyway this comes without patch also. That I will dig more and raise a Jira.

          About patch, some comments.

          1. Header DataNode on 127.0.0.1:54107 should contain 'Data port' not 'http port' as Data Port is used as the ID of the datanode everywhere.

          2. Namenode Address should contain port as well. Not just hostname. Its 'Namenode Address' not just 'NameNode Host'. Though earlier JMX was returning only Hostname, now its time to be right in new JMX api.

          Show
          vinayrpet Vinayakumar B added a comment - I was having some problem to connect to DataNode's HTTP port in my eclipse run in windows. Seems to be problem with PortUnificationHandler. I was able to connect to proxy port. Anyway this comes without patch also. That I will dig more and raise a Jira. About patch, some comments. 1. Header DataNode on 127.0.0.1:54107 should contain 'Data port' not 'http port' as Data Port is used as the ID of the datanode everywhere. 2. Namenode Address should contain port as well. Not just hostname. Its 'Namenode Address' not just 'NameNode Host'. Though earlier JMX was returning only Hostname, now its time to be right in new JMX api.
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks a lot for the comments, will upload a new patch to address them.

          Show
          cheersyang Weiwei Yang added a comment - Thanks a lot for the comments, will upload a new patch to address them.
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          I was having some problem to connect to DataNode's HTTP port in my eclipse run in windows. Seems to be problem with PortUnificationHandler. I was able to connect to proxy port. Anyway this comes without patch also. That I will dig more and raise a Jira.

          This was a problem with the classpath.
          ChannelInitializer class is loading from older version of netty-transport jar instead of latest netty-all jar.
          Will handle this in separate Jira.

          Show
          vinayrpet Vinayakumar B added a comment - - edited I was having some problem to connect to DataNode's HTTP port in my eclipse run in windows. Seems to be problem with PortUnificationHandler. I was able to connect to proxy port. Anyway this comes without patch also. That I will dig more and raise a Jira. This was a problem with the classpath. ChannelInitializer class is loading from older version of netty-transport jar instead of latest netty-all jar. Will handle this in separate Jira.
          Hide
          cheersyang Weiwei Yang added a comment -

          V6 patch has addressed Vinayakumar B's comments, details:

          1. DataNode on 9.30.150.129:9867 now uses IPC port instead of HTTP port, so user could read and use the address in places like "hdfs dfsadmin -getDatanodeInfo", "-triggerBlockReport" etc.
          2. Namenode Address now uses namenodeHostName:RpcPort, this correction is done in DN metrics NamenodeAddresses and BPServiceActorInfo
          Show
          cheersyang Weiwei Yang added a comment - V6 patch has addressed Vinayakumar B 's comments , details: DataNode on 9.30.150.129:9867 now uses IPC port instead of HTTP port, so user could read and use the address in places like "hdfs dfsadmin -getDatanodeInfo", "-triggerBlockReport" etc. Namenode Address now uses namenodeHostName:RpcPort , this correction is done in DN metrics NamenodeAddresses and BPServiceActorInfo
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 41s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 11s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 77m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          96m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813030/HDFS-10440.006.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 56717ccf2075 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6314843
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15903/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15903/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 41s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 77m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 96m 45s Reason Tests Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813030/HDFS-10440.006.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 56717ccf2075 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6314843 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15903/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15903/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          I meant 'Data port' Not IPC port.
          Since data port is the one which is used everywhere, in logs, in fsck, etc.

          Show
          vinayrpet Vinayakumar B added a comment - I meant 'Data port' Not IPC port. Since data port is the one which is used everywhere, in logs, in fsck, etc.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Vinayakumar B

          Thank you for the input.I thought IPC port would be useful because there are quite some dfsadmin command utilities using datanode_host:ipc_port as an identifier. But I also agree with you, because I noticed the Node entry on Namenode UI -> Datanodes lists datanode with its data port, so it's more consistent. I just uploaded v7 patch for this.

          Show
          cheersyang Weiwei Yang added a comment - Hello Vinayakumar B Thank you for the input.I thought IPC port would be useful because there are quite some dfsadmin command utilities using datanode_host:ipc_port as an identifier. But I also agree with you, because I noticed the Node entry on Namenode UI -> Datanodes lists datanode with its data port, so it's more consistent. I just uploaded v7 patch for this.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Vinayakumar B

          Can you please help to review v7 patch? Appreciate your help.

          Show
          cheersyang Weiwei Yang added a comment - Hello Vinayakumar B Can you please help to review v7 patch? Appreciate your help.
          Hide
          vinayrpet Vinayakumar B added a comment -

          1. I didn't mean to change existing DataNode#getNamenodeAddresses() output. It breaks the compatibility. So let it be just hostname. After this, BPSA#getNameNodeAddress() could be private.

                   for (BPServiceActor actor : bpos.getBPServiceActors()) {
          -          info.put(actor.getNNSocketAddress().getHostName(),
          +          info.put(actor.getNameNodeAddress(),
                         bpos.getBlockPoolId());

          2. I am not very sure about giving link to /conf is good idea here since other pages dont have this yet. May be this could be taken as a separate improvement, together for all HTML pages.

          <li><a href="conf">Configuration</a></li>

          Other changes are looks good. Thanks for updates.
          +1, once these are addressed.

          FYI, You dont need to change the Jira status to "In-Progress" and back to "Patch Available" every time.
          Jenkins will pick up directly once the latest patch file is updated.

          Show
          vinayrpet Vinayakumar B added a comment - 1. I didn't mean to change existing DataNode#getNamenodeAddresses() output. It breaks the compatibility. So let it be just hostname. After this, BPSA#getNameNodeAddress() could be private. for (BPServiceActor actor : bpos.getBPServiceActors()) { - info.put(actor.getNNSocketAddress().getHostName(), + info.put(actor.getNameNodeAddress(), bpos.getBlockPoolId()); 2. I am not very sure about giving link to /conf is good idea here since other pages dont have this yet. May be this could be taken as a separate improvement, together for all HTML pages. <li><a href= "conf" >Configuration</a></li> Other changes are looks good. Thanks for updates. +1, once these are addressed. FYI, You dont need to change the Jira status to "In-Progress" and back to "Patch Available" every time. Jenkins will pick up directly once the latest patch file is updated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks a lot, let me upload a new patch to address these. Also for the tip about Jenkins, it's smarter than I thought .

          Show
          cheersyang Weiwei Yang added a comment - Thanks a lot, let me upload a new patch to address these. Also for the tip about Jenkins, it's smarter than I thought .
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Uploaded v8 patch to address following updates

          • Keep DataNode#getNamenodeAddresses() unmodified for compatibility
          • Modify BPServiceActor#getNameNodeAddress() to private
          • Remove Utilities/conf link, a separated JIRA HDFS-10583 is created to address this
          Show
          cheersyang Weiwei Yang added a comment - - edited Uploaded v8 patch to address following updates Keep DataNode#getNamenodeAddresses() unmodified for compatibility Modify BPServiceActor#getNameNodeAddress() to private Remove Utilities/conf link, a separated JIRA HDFS-10583 is created to address this
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 34s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 219 unchanged - 0 fixed = 221 total (was 219)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 67m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          88m 50s



          Reason Tests
          Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814019/HDFS-10440.008.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b61daa4901b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4fd37ee
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15927/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15927/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15927/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15927/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 34s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 219 unchanged - 0 fixed = 221 total (was 219) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 67m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 88m 50s Reason Tests Failed junit tests hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814019/HDFS-10440.008.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b61daa4901b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4fd37ee Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15927/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15927/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15927/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15927/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Fixed check style warnings in v9 patch.

          Show
          cheersyang Weiwei Yang added a comment - Fixed check style warnings in v9 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 41s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 11s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 1m 0s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 218 unchanged - 1 fixed = 218 total (was 219)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 59m 8s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          78m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814056/HDFS-10440.009.patch
          JIRA Issue HDFS-10440
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7df31281f1fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4fd37ee
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15929/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15929/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15929/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 41s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 1m 0s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 218 unchanged - 1 fixed = 218 total (was 219) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 59m 8s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 78m 45s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814056/HDFS-10440.009.patch JIRA Issue HDFS-10440 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7df31281f1fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4fd37ee Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15929/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15929/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15929/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Latest patch looks good to me
          +1.
          Committing based on Kihwal Lee's earlier +1.

          Show
          vinayrpet Vinayakumar B added a comment - Latest patch looks good to me +1. Committing based on Kihwal Lee 's earlier +1.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk, branch-2 and branch-2.8.

          Thanks for the Contribution Weiwei Yang.
          Thanks for reviews Kihwal Lee.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks for the Contribution Weiwei Yang . Thanks for reviews Kihwal Lee .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10025/)
          HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) (vinayakumarb: rev 2a0082c51da7cbe2770eddb5f72cd7f8d72fa5f6)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/dn.js
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/index.html
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10025/ ) HDFS-10440 . Improve DataNode web UI (Contributed by Weiwei Yang) (vinayakumarb: rev 2a0082c51da7cbe2770eddb5f72cd7f8d72fa5f6) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/dn.js hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/index.html hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks a lot Vinayakumar B, Kihwal Lee

          Show
          cheersyang Weiwei Yang added a comment - Thanks a lot Vinayakumar B , Kihwal Lee

            People

            • Assignee:
              cheersyang Weiwei Yang
              Reporter:
              cheersyang Weiwei Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development