Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10838

Last full block report received time for each DN should be easily discoverable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: ui
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It should be easy for administrators to discover the time of last full block report from each DataNode.

      We can show it in the NameNode web UI or in the output of hdfs dfsadmin -report, or both.

      1. DFSAdmin-Report.png
        56 kB
        Surendra Singh Lilhore
      2. HDFS-10838.002.patch
        12 kB
        Surendra Singh Lilhore
      3. HDFS-10838.003.patch
        13 kB
        Surendra Singh Lilhore
      4. HDFS-10838.004.patch
        13 kB
        Surendra Singh Lilhore
      5. HDFS-10838.005.patch
        12 kB
        Surendra Singh Lilhore
      6. HDFS-10838.006.patch
        15 kB
        Surendra Singh Lilhore
      7. HDFS-10838-001.patch
        12 kB
        Surendra Singh Lilhore
      8. NN_UI_relative_time_minutes.png
        52 kB
        Surendra Singh Lilhore
      9. NN_UI_relative_time.png
        17 kB
        Surendra Singh Lilhore
      10. NN_UI.png
        18 kB
        Surendra Singh Lilhore

        Issue Links

          Activity

          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Added last block report time in NN UI and dfsadmin -report
          Attached initial patch.
          Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Added last block report time in NN UI and dfsadmin -report Attached initial patch. Please review..
          Hide
          vinayrpet Vinayakumar B added a comment -

          Changes looks fine to me,
          Was having one doubt about the compatibility of dfsadmin -report output.
          Allen Wittenauer, Can you confirm addition of new line "Last Block Report: ").append(new Date(lastBlockReportTime)) for each DN , whether this will be incompatible change? Asking this doubt, because its a new addition in separate line. No modifications to existing output lines.

          Show
          vinayrpet Vinayakumar B added a comment - Changes looks fine to me, Was having one doubt about the compatibility of dfsadmin -report output. Allen Wittenauer , Can you confirm addition of new line "Last Block Report: ").append(new Date(lastBlockReportTime)) for each DN , whether this will be incompatible change? Asking this doubt, because its a new addition in separate line. No modifications to existing output lines.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Nice, thank you for the quick patch Surendra Singh Lilhore!

          1. DatanodeInfoProto convert(DatanodeInfo info) should handle missing lastBlockReportTime for wire compatibility. Something like this should work.
            di.hasLastBlockReportTime() ? di.getLastBlockReportTime() : 0
            
          2. Also when the value is displayed in the web UI/command line output it should be checked whether it is equal to zero and if so something like 'never' should be displayed instead of the timestamp.
          3. Should the NN web UI to show relative time, just like last heartbeat time? I'm fine with either though.
          Show
          arpitagarwal Arpit Agarwal added a comment - Nice, thank you for the quick patch Surendra Singh Lilhore ! DatanodeInfoProto convert(DatanodeInfo info) should handle missing lastBlockReportTime for wire compatibility. Something like this should work. di.hasLastBlockReportTime() ? di.getLastBlockReportTime() : 0 Also when the value is displayed in the web UI/command line output it should be checked whether it is equal to zero and if so something like 'never' should be displayed instead of the timestamp. Should the NN web UI to show relative time, just like last heartbeat time? I'm fine with either though.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Vinayakumar B and Arpit Agarwal for review...

          Should the NN web UI to show relative time, just like last heartbeat time? I'm fine with either though.

          Yes, I think in UI we should show relative time.

          Vinayakumar B, please give your suggestion on this.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Vinayakumar B and Arpit Agarwal for review... Should the NN web UI to show relative time, just like last heartbeat time? I'm fine with either though. Yes, I think in UI we should show relative time. Vinayakumar B , please give your suggestion on this.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Attached updated patch.
          Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Attached updated patch. Please review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827429/HDFS-10838.002.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16659/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827429/HDFS-10838.002.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16659/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          v3 : Rebased patch.

          Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - v3 : Rebased patch. Please review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827441/HDFS-10838.003.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16662/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827441/HDFS-10838.003.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16662/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 7m 16s trunk passed
          +1 compile 1m 32s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 14s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 25s the patch passed
          +1 cc 1m 25s the patch passed
          +1 javac 1m 25s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 6 new + 422 unchanged - 3 fixed = 428 total (was 425)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 26s the patch passed
          +1 javadoc 1m 10s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 unit 59m 21s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          87m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827643/HDFS-10838.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux fd17c7ea3e5e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cae33118
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16678/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16678/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16678/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 7m 16s trunk passed +1 compile 1m 32s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 25s the patch passed +1 cc 1m 25s the patch passed +1 javac 1m 25s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 6 new + 422 unchanged - 3 fixed = 428 total (was 425) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 26s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 59m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 87m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827643/HDFS-10838.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux fd17c7ea3e5e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cae33118 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16678/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16678/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16678/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for the updated patch Surendra Singh Lilhore. Looks good, two more comments.

          1. Sorry for not catching this earlier - we should call node.setLastBlockReportTime(now()); only if lastStorageInRpc is true i.e. when all storages have been processed.
          2. Do you know what helper_relative_time_now will print if the value is zero? The current "a few seconds ago" looks great actually but I think we can print in the same format as heartbeat time so it remains numerically sortable.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for the updated patch Surendra Singh Lilhore . Looks good, two more comments. Sorry for not catching this earlier - we should call node.setLastBlockReportTime(now()); only if lastStorageInRpc is true i.e. when all storages have been processed. Do you know what helper_relative_time_now will print if the value is zero? The current "a few seconds ago" looks great actually but I think we can print in the same format as heartbeat time so it remains numerically sortable.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Arpit Agarwal for review.

          Do you know what helper_relative_time_now will print if the value is zero?

          It will print "a few seconds ago" only.

          I think we can print in the same format as heartbeat time so it remains numerically sortable.

          Heartbeat time only print in seconds.

          <td ng-value="{lastContact}">{lastContact}s</td>

          Do you want block report time in seconds ?

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Arpit Agarwal for review. Do you know what helper_relative_time_now will print if the value is zero? It will print "a few seconds ago" only. I think we can print in the same format as heartbeat time so it remains numerically sortable. Heartbeat time only print in seconds. <td ng-value= "{lastContact}" >{lastContact}s</td> Do you want block report time in seconds ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Surendra Singh Lilhore, I think either seconds or perhaps more practically minutes (since full block report interval is in hours the count in seconds could get rather large). That will allow sorting on the column numerically to quickly scan for outliers.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Surendra Singh Lilhore , I think either seconds or perhaps more practically minutes (since full block report interval is in hours the count in seconds could get rather large). That will allow sorting on the column numerically to quickly scan for outliers.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Arpit Agarwal for clarifying..

          Attached updated patch. Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Arpit Agarwal for clarifying.. Attached updated patch. Please review..
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 pending Jenkins. Thanks Surendra Singh Lilhore.

          One remaining point is treating 0 as special case in the web UI also, just like the command-line. It can be done in a followup Jira.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins. Thanks Surendra Singh Lilhore . One remaining point is treating 0 as special case in the web UI also, just like the command-line. It can be done in a followup Jira.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Arpit Agarwal.

          I think some problem in jenkins build, its treating image file as patch.

          Processing: HDFS-10838
          cut: invalid byte, character or field list
          Try 'cut --help' for more information.
          cut: invalid byte, character or field list
          Try 'cut --help' for more information.
          HDFS-10838 patch is being downloaded at Sun Sep 11 10:48:13 UTC 2016 from
            https://issues.apache.org/jira/secure/attachment/12827924/NN_UI_relative_time_minutes.png -> Downloaded
          ERROR: Unsure how to process HDFS-10838.
          
          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Arpit Agarwal . I think some problem in jenkins build, its treating image file as patch. Processing: HDFS-10838 cut: invalid byte , character or field list Try 'cut --help' for more information. cut: invalid byte , character or field list Try 'cut --help' for more information. HDFS-10838 patch is being downloaded at Sun Sep 11 10:48:13 UTC 2016 from https: //issues.apache.org/jira/secure/attachment/12827924/NN_UI_relative_time_minutes.png -> Downloaded ERROR: Unsure how to process HDFS-10838.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Resubmitting patch, hope it will take last uploaded patch

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Resubmitting patch, hope it will take last uploaded patch
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Surendra Singh Lilhore, you may have to reattach the patch with another name. Jenkins seems to be picking up the latest file attached without looking at the extension.

          I looked at this some more and the lastStorageInRpc check will be insufficient. We have to look into the BlockReportContext to determine when the last report has been received. e.g. this block of code earlier on in processReport..

                if (context != null) {
                  storageInfo.setLastBlockReportId(context.getReportId());
                  if (lastStorageInRpc) {
                    int rpcsSeen = node.updateBlockReportContext(context);
                    if (rpcsSeen >= context.getTotalRpcs()) {
                      long leaseId = blockReportLeaseManager.removeLease(node);
          

          I'll also take a look at this and help with updating the patch to ensure we only update this timestamp when we are sure we have processed all storage RPCs.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Surendra Singh Lilhore , you may have to reattach the patch with another name. Jenkins seems to be picking up the latest file attached without looking at the extension. I looked at this some more and the lastStorageInRpc check will be insufficient. We have to look into the BlockReportContext to determine when the last report has been received. e.g. this block of code earlier on in processReport .. if (context != null ) { storageInfo.setLastBlockReportId(context.getReportId()); if (lastStorageInRpc) { int rpcsSeen = node.updateBlockReportContext(context); if (rpcsSeen >= context.getTotalRpcs()) { long leaseId = blockReportLeaseManager.removeLease(node); I'll also take a look at this and help with updating the patch to ensure we only update this timestamp when we are sure we have processed all storage RPCs.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          By the way there may be some update needed in this logic, per HDFS-10301. Looking at that too.

          Show
          arpitagarwal Arpit Agarwal added a comment - By the way there may be some update needed in this logic, per HDFS-10301 . Looking at that too.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Was having one doubt about the compatibility of dfsadmin -report output.
          Allen Wittenauer, Can you confirm addition of new line "Last Block Report: ").append(new Date(lastBlockReportTime)) for each DN , whether this will be incompatible change? Asking this doubt, because its a new addition in separate line. No modifications to existing output lines.

          Related to compatibility of 'dfsadmin -report', we can mark this jira as incompatible as per docs
          or
          Can separate only 'dfsadmin -report' related changes to separate Jira which can be present in only trunk and push remaining changes to branch-2* as well.

          Show
          vinayrpet Vinayakumar B added a comment - Was having one doubt about the compatibility of dfsadmin -report output. Allen Wittenauer, Can you confirm addition of new line "Last Block Report: ").append(new Date(lastBlockReportTime)) for each DN , whether this will be incompatible change? Asking this doubt, because its a new addition in separate line. No modifications to existing output lines. Related to compatibility of 'dfsadmin -report', we can mark this jira as incompatible as per docs or Can separate only 'dfsadmin -report' related changes to separate Jira which can be present in only trunk and push remaining changes to branch-2* as well.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Updated patch based on new changes done in HDFS-10301

          Please review...

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Updated patch based on new changes done in HDFS-10301 Please review...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 1m 21s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 0m 59s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 19s the patch passed
          +1 cc 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 421 unchanged - 2 fixed = 423 total (was 423)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 15s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
          -1 unit 96m 36s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          123m 33s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840563/HDFS-10838.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 9eae98e6fb0e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07825f2
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17666/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17666/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17666/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 1m 21s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 0m 59s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 19s the patch passed +1 cc 1m 19s the patch passed +1 javac 1m 19s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 421 unchanged - 2 fixed = 423 total (was 423) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 15s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. -1 unit 96m 36s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 123m 33s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840563/HDFS-10838.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9eae98e6fb0e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07825f2 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17666/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17666/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17666/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Arpit Agarwal, Vinayakumar B

          Can you review the v5 patch ?

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Arpit Agarwal , Vinayakumar B Can you review the v5 patch ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Surendra Singh Lilhore, thanks for picking this up again now that HDFS-10301 is resolved. I will take a look later this week.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Surendra Singh Lilhore , thanks for picking this up again now that HDFS-10301 is resolved. I will take a look later this week.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Hi Surendra Singh Lilhore, I tried to test this patch and ran into an issue that I had also hit earlier. The NameNode web UI tabs don't load.
          Never mind, there was something wrong on my end.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Hi Surendra Singh Lilhore , I tried to test this patch and ran into an issue that I had also hit earlier. The NameNode web UI tabs don't load. Never mind, there was something wrong on my end.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840563/HDFS-10838.005.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18518/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDFS-10838 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840563/HDFS-10838.005.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18518/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The patch needs a trivial rebase to resolve a conflict in hdfs.proto. Overall the changes look great. I tried this out in a test cluster and the new column in the NN web ui is very helpful! Two comments:

          1. We should use monotonicNow to calculate the time difference for the web UI. DataNodeInfo has two fields for heartbeat - lastUpdate and lastUpdateMonotonic. We can use the same approach.
          2. Also we can add the new field(s) to JsonUtil#toJsonMap(DatanodeInfo).

          Sorry about the long delay in reviewing this.

          Show
          arpitagarwal Arpit Agarwal added a comment - The patch needs a trivial rebase to resolve a conflict in hdfs.proto. Overall the changes look great. I tried this out in a test cluster and the new column in the NN web ui is very helpful! Two comments: We should use monotonicNow to calculate the time difference for the web UI. DataNodeInfo has two fields for heartbeat - lastUpdate and lastUpdateMonotonic. We can use the same approach. Also we can add the new field(s) to JsonUtil#toJsonMap(DatanodeInfo) . Sorry about the long delay in reviewing this.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Arpit Agarwal for review..
          Attached updated patch, Please review..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Arpit Agarwal for review.. Attached updated patch, Please review..
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Surendra Singh Lilhore. +1 pending Jenkins.

          HDFS Pre-commit runs appear to be busted right now due to a bad node, we should retrigger the Jenkins runs when the node is fixed.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Surendra Singh Lilhore . +1 pending Jenkins. HDFS Pre-commit runs appear to be busted right now due to a bad node, we should retrigger the Jenkins runs when the node is fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 16m 21s trunk passed
          +1 compile 1m 51s trunk passed
          +1 checkstyle 0m 56s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 58s trunk passed
          +1 javadoc 1m 23s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 55s the patch passed
          +1 compile 1m 51s the patch passed
          +1 cc 1m 51s the patch passed
          +1 javac 1m 51s the patch passed
          -0 checkstyle 1m 1s hadoop-hdfs-project: The patch generated 2 new + 430 unchanged - 2 fixed = 432 total (was 432)
          +1 mvnsite 1m 56s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 23s the patch passed
          +1 javadoc 1m 10s the patch passed
          +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
          -1 unit 68m 36s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          112m 41s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10838
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856072/HDFS-10838.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 78709fe36bcb 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d9dc444
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18596/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18596/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 16m 21s trunk passed +1 compile 1m 51s trunk passed +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 58s trunk passed +1 javadoc 1m 23s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 1m 51s the patch passed +1 cc 1m 51s the patch passed +1 javac 1m 51s the patch passed -0 checkstyle 1m 1s hadoop-hdfs-project: The patch generated 2 new + 430 unchanged - 2 fixed = 432 total (was 432) +1 mvnsite 1m 56s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 23s the patch passed +1 javadoc 1m 10s the patch passed +1 unit 1m 14s hadoop-hdfs-client in the patch passed. -1 unit 68m 36s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 112m 41s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10838 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12856072/HDFS-10838.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 78709fe36bcb 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d9dc444 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18596/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18596/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Committed this to trunk and branch-2. Thanks for the contribution Surendra Singh Lilhore!

          (verified that the TestBalancer failure did not repro locally).

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Committed this to trunk and branch-2. Thanks for the contribution Surendra Singh Lilhore ! (verified that the TestBalancer failure did not repro locally).
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11358 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11358/)
          HDFS-10838. Last full block report received time for each DN should be (arp: rev b5adc5c3011f111f86d232cb33ec522547f68a95)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11358 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11358/ ) HDFS-10838 . Last full block report received time for each DN should be (arp: rev b5adc5c3011f111f86d232cb33ec522547f68a95) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Arpit Agarwal for review and commit ...

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Arpit Agarwal for review and commit ...

            People

            • Assignee:
              surendrasingh Surendra Singh Lilhore
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development