Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When you run fsck with "-files" or "-racks", you will get something like below if one of the replicas is decommissioned.

      blk_x len=y repl=3 [dn1, dn2, dn3, dn4]
      

      That is because in NamenodeFsck, the repl count comes from live replicas count; while the actual nodes come from LocatedBlock which include decommissioned nodes.

      Another issue in NamenodeFsck is BlockPlacementPolicy's verifyBlockPlacement verifies LocatedBlock that includes decommissioned nodes. However, it seems better to exclude the decommissioned nodes in the verification; just like how fsck excludes decommissioned nodes when it check for under replicated blocks.

      1. HDFS-7993.1.patch
        7 kB
        J.Andreina
      2. HDFS-7993.2.patch
        7 kB
        J.Andreina
      3. HDFS-7993.3.patch
        7 kB
        J.Andreina
      4. HDFS-7993.4.patch
        9 kB
        J.Andreina
      5. HDFS-7993.5.patch
        12 kB
        J.Andreina
      6. HDFS-7993.6.patch
        13 kB
        J.Andreina

        Issue Links

          Activity

          Hide
          andreina J.Andreina added a comment -

          I would like to work on this issue. If you have patch please feel free to assign to you.

          Show
          andreina J.Andreina added a comment - I would like to work on this issue. If you have patch please feel free to assign to you.
          Hide
          mingma Ming Ma added a comment -

          Thanks, J.Andreina. Please go ahead.

          We might want to clarify what the right descriptions should be. e.g. a) decommissioned nodes should be specifically mentioned in the above case, so people know the block has one decommission replica and 3 live replicas. b) verifyBlockPlacement should be verified for the live replicas only. After that, it will be really useful to add new unit tests.

          Show
          mingma Ming Ma added a comment - Thanks, J.Andreina . Please go ahead. We might want to clarify what the right descriptions should be. e.g. a) decommissioned nodes should be specifically mentioned in the above case, so people know the block has one decommission replica and 3 live replicas. b) verifyBlockPlacement should be verified for the live replicas only. After that, it will be really useful to add new unit tests.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I agree that this is confusing currently. But there are compatibility issues with changing the output.

          I wonder if we could have a "-replicaDetails" option that could print out more detail about each replica (whether it is decomissioning, corrupt, etc.). Knowing what each replica was up to would be much more useful than just knowing the number of nodes that are live.

          Show
          cmccabe Colin P. McCabe added a comment - I agree that this is confusing currently. But there are compatibility issues with changing the output. I wonder if we could have a "-replicaDetails" option that could print out more detail about each replica (whether it is decomissioning, corrupt, etc.). Knowing what each replica was up to would be much more useful than just knowing the number of nodes that are live.
          Hide
          andreina J.Andreina added a comment -

          Thanks Colin P. McCabe and Ming Ma for suggestions.

          I wonder if we could have a "-replicaDetails" option that could print out more detail about each replica

          I agree with this point. Have uploaded an initial patch for adding "-replicaDetails" option to fsck command.

          After applying patch the result would be like below

          > hdfs fsck /100 -files -blocks -replicaDetails

          /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s).
          0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK](LIVE Replica), DatanodeInfoWithStorage[10.19.92.114:50010,DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82,DISK](DECOMMISSIONING Replica)]

          Please review the patch .

          Show
          andreina J.Andreina added a comment - Thanks Colin P. McCabe and Ming Ma for suggestions. I wonder if we could have a "-replicaDetails" option that could print out more detail about each replica I agree with this point. Have uploaded an initial patch for adding "-replicaDetails" option to fsck command. After applying patch the result would be like below > hdfs fsck /100 -files -blocks -replicaDetails /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s). 0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [DatanodeInfoWithStorage [10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK] (LIVE Replica) , DatanodeInfoWithStorage [10.19.92.114:50010,DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82,DISK] (DECOMMISSIONING Replica) ] Please review the patch .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724131/HDFS-7993.1.patch
          against trunk revision b1e0590.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724131/HDFS-7993.1.patch against trunk revision b1e0590. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10226//console This message is automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Updated the patch after correcting findbug failures.
          Please review.

          Show
          andreina J.Andreina added a comment - Updated the patch after correcting findbug failures. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724199/HDFS-7993.2.patch
          against trunk revision 6495940.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10229//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10229//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724199/HDFS-7993.2.patch against trunk revision 6495940. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10229//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10229//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for taking a look at this, J.Andreina.

          /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s).
          0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK](LIVE Replica), DatanodeInfoWithStorage[10.19.92.114:50010,DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82,DISK](DECOMMISSIONING Replica)]
          

          Hmm. We don't need to keep printing "DatanodeInfoWithStorage" for each replica, and we should print "LIVE" instead of "LIVE replica", etc. I think separating the fields with slashes might help make this parseable as well. Maybe it would look kind of like this:

          /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s).
          0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [LIVE/10.19.92.116:50010/DS-13faa414-cf01-4ae7-b307-2ae87caa273c, DECOMMISSIONING/10.19.92.114:50010/DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82]
          

          Looks good aside from that.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for taking a look at this, J.Andreina . /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s). 0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK](LIVE Replica), DatanodeInfoWithStorage[10.19.92.114:50010,DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82,DISK](DECOMMISSIONING Replica)] Hmm. We don't need to keep printing "DatanodeInfoWithStorage" for each replica, and we should print "LIVE" instead of "LIVE replica", etc. I think separating the fields with slashes might help make this parseable as well. Maybe it would look kind of like this: /100 6619 bytes, 1 block(s): Under replicated BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s). 0. BP-1926123874-10.19.92.116-1428408820198:blk_1073741829_1005 len=6619 repl=1 [LIVE/10.19.92.116:50010/DS-13faa414-cf01-4ae7-b307-2ae87caa273c, DECOMMISSIONING/10.19.92.114:50010/DS-583ffc24-075f-4ad1-913b-45fe0f8a9c82] Looks good aside from that.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Colin P. McCabe

          We don't need to keep printing "DatanodeInfoWithStorage" for each replica, and we should print "LIVE" instead of "LIVE replica", etc

          I agree with this. It needs to be formatted to be simple and nice.

          LIVE/10.19.92.116:50010/DS-13faa414-cf01-4ae7-b307-2ae87caa273c

          Here, there are two more things to consider.
          1. When only "-locations" options given, then DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK] will be printed. If "-racks" option is given then it will be something like /default-rack/10.19.92.116:50010. In second case, separating with slashes may not look good. It will add to rack topology.
          2. Changing the original output (without -replicaDetails) will not be compatibility issue? Originally HDFS-7647 introduced DatanodeInfoWithStorage to add storage details.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Colin P. McCabe We don't need to keep printing "DatanodeInfoWithStorage" for each replica, and we should print "LIVE" instead of "LIVE replica", etc I agree with this. It needs to be formatted to be simple and nice. LIVE/10.19.92.116:50010/DS-13faa414-cf01-4ae7-b307-2ae87caa273c Here, there are two more things to consider. 1. When only "-locations" options given, then DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK] will be printed. If "-racks" option is given then it will be something like /default-rack/10.19.92.116:50010 . In second case, separating with slashes may not look good. It will add to rack topology. 2. Changing the original output (without -replicaDetails) will not be compatibility issue? Originally HDFS-7647 introduced DatanodeInfoWithStorage to add storage details.
          Hide
          mingma Ming Ma added a comment -

          HDFS-7933 has improved the replica reporting in the case of missing or under replicated block w.r.t. decommission. It appears we can use that work to address the reporting of fully replicated blocks.

          • Change from report.append(" repl=" + liveReplicas); to report.append(" repl=" + totalReplicas);
          • Instead of using DatanodeInfo to find replica details, we can use NumberReplicas instead. However, there are two types of "stale" definitions in NN. One is "stale datanode" when the datanode hasn't sent heartbeat for some time. Another one is "stale block content" when NN hasn't received block report from that DN after failover; that is what NumberReplicas#replicasOnStaleNodes is for. If we need to count "stale datanode", we can add another field to NumberReplicas for that.
          Show
          mingma Ming Ma added a comment - HDFS-7933 has improved the replica reporting in the case of missing or under replicated block w.r.t. decommission. It appears we can use that work to address the reporting of fully replicated blocks. Change from report.append(" repl=" + liveReplicas); to report.append(" repl=" + totalReplicas); Instead of using DatanodeInfo to find replica details, we can use NumberReplicas instead. However, there are two types of "stale" definitions in NN. One is "stale datanode" when the datanode hasn't sent heartbeat for some time. Another one is "stale block content" when NN hasn't received block report from that DN after failover; that is what NumberReplicas#replicasOnStaleNodes is for. If we need to count "stale datanode", we can add another field to NumberReplicas for that.
          Hide
          vinayrpet Vinayakumar B added a comment -

          patch needs rebase after HDFS-7933.

          Change from report.append(" repl=" + liveReplicas); to report.append(" repl=" + totalReplicas);

          In the current output, blk_x len=y repl=3 [dn1, dn2, dn3, dn4], the count repl=3 exactly gives the count of live replicas excluding decommission(ing/ed). So i think leaving it as is would be better.

          Instead of using DatanodeInfo to find replica details, we can use NumberReplicas instead.

          As discussed above, this jira is to add the detail/state about each replica, not just the overall count, which is not available in NumberReplicas.

          If we need to count "stale datanode", we can add another field to NumberReplicas for that.

          I think this count will be there for long time, since the block report interval is long. IMO If necessary may go in followup jira

          Show
          vinayrpet Vinayakumar B added a comment - patch needs rebase after HDFS-7933 . Change from report.append(" repl=" + liveReplicas); to report.append(" repl=" + totalReplicas); In the current output, blk_x len=y repl=3 [dn1, dn2, dn3, dn4] , the count repl=3 exactly gives the count of live replicas excluding decommission(ing/ed). So i think leaving it as is would be better. Instead of using DatanodeInfo to find replica details, we can use NumberReplicas instead. As discussed above, this jira is to add the detail/state about each replica, not just the overall count, which is not available in NumberReplicas . If we need to count "stale datanode", we can add another field to NumberReplicas for that. I think this count will be there for long time, since the block report interval is long. IMO If necessary may go in followup jira
          Hide
          andreina J.Andreina added a comment -

          Attached a patch with updated trunk code.
          Please review.

          Show
          andreina J.Andreina added a comment - Attached a patch with updated trunk code. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725215/HDFS-7993.3.patch
          against trunk revision b5a0b24.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10274//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10274//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725215/HDFS-7993.3.patch against trunk revision b5a0b24. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10274//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10274//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks, Vinay.

          In the current output, blk_x len=y repl=3 [dn1, dn2, dn3, dn4], the count repl=3 exactly gives the count of live replicas excluding decommission(ing/ed). So i think leaving it as is would be better.

          Maybe we can change the description from repl to live repl? It will address the confusion others might have.

          As discussed above, this jira is to add the detail/state about each replica, not just the overall count, which is not available in NumberReplicas.

          Good point.

          I think this count will be there for long time, since the block report interval is long. IMO If necessary may go in followup jira

          It will be useful to show "stale block content" replica. After NN failover if there is any over replication, it won't be counted as excess replicas until BR. So running fsck will show these to-be-excess replicas as "Live Replica"s.

          Show
          mingma Ming Ma added a comment - Thanks, Vinay. In the current output, blk_x len=y repl=3 [dn1, dn2, dn3, dn4] , the count repl=3 exactly gives the count of live replicas excluding decommission(ing/ed). So i think leaving it as is would be better. Maybe we can change the description from repl to live repl ? It will address the confusion others might have. As discussed above, this jira is to add the detail/state about each replica, not just the overall count, which is not available in NumberReplicas. Good point. I think this count will be there for long time, since the block report interval is long. IMO If necessary may go in followup jira It will be useful to show "stale block content" replica. After NN failover if there is any over replication, it won't be counted as excess replicas until BR. So running fsck will show these to-be-excess replicas as "Live Replica"s.
          Hide
          andreina J.Andreina added a comment -

          Thanks Ming Ma and Vinayakumar B for the review and comments.

          Maybe we can change the description from repl to live repl? It will address the confusion others might have.

          Agree with this point. Have updated the patch accordingly.

          It will be useful to show "stale block content" replica

          Updated the patch to provide information whether stale replica is due to either Stale Datanode or Stale block content.

          Please review the patch.

          Show
          andreina J.Andreina added a comment - Thanks Ming Ma and Vinayakumar B for the review and comments. Maybe we can change the description from repl to live repl? It will address the confusion others might have. Agree with this point. Have updated the patch accordingly. It will be useful to show "stale block content" replica Updated the patch to provide information whether stale replica is due to either Stale Datanode or Stale block content. Please review the patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725779/HDFS-7993.4.patch
          against trunk revision 1b89a3e.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10286//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10286//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725779/HDFS-7993.4.patch against trunk revision 1b89a3e. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10286//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10286//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Maybe we can change the description from repl to live repl? It will address the confusion others might have.

          Can we do that in a separate JIRA? Since it's an incompatible change we might want to do it only in Hadoop 3.0. There are a lot of people parsing fsck output (unfortunately).

          The rest looks good, if we can keep the existing output the same I would love to add the replicaDetails option.

          Show
          cmccabe Colin P. McCabe added a comment - - edited Maybe we can change the description from repl to live repl? It will address the confusion others might have. Can we do that in a separate JIRA? Since it's an incompatible change we might want to do it only in Hadoop 3.0. There are a lot of people parsing fsck output (unfortunately). The rest looks good, if we can keep the existing output the same I would love to add the replicaDetails option.
          Hide
          mingma Ming Ma added a comment -

          Can we do that in a separate JIRA?

          I am fine with that.

          Thanks, J.Andreina. The code looks good. Maybe some new unit tests will be useful?

          Show
          mingma Ming Ma added a comment - Can we do that in a separate JIRA? I am fine with that. Thanks, J.Andreina . The code looks good. Maybe some new unit tests will be useful?
          Hide
          andreina J.Andreina added a comment -

          Thanks Colin P. McCabe and Ming Ma for reviewing .

          I have updated the patch with testcase and retaining the description as "repl" .
          Please review the patch.

          Show
          andreina J.Andreina added a comment - Thanks Colin P. McCabe and Ming Ma for reviewing . I have updated the patch with testcase and retaining the description as "repl" . Please review the patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726133/HDFS-7993.5.patch
          against trunk revision 76e7264.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10298//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10298//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726133/HDFS-7993.5.patch against trunk revision 76e7264. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10298//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10298//console This message is automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Could the test fail if the node becomes decommissioned right after it checks isDecommissionInProgress? Otherwise, it looks good.

          if (!checkDecommissionInProgress && datanodeInfo != null
           && datanodeInfo.isDecommissionInProgress()) {
              String fsckOut = runFsck(conf, 0, true, testFile, "-files", "-blocks", "-replicaDetails");
              assertTrue(fsckOut.contains("(DECOMMISSIONING)"));
              checkDecommissionInProgress = true;
          }
          
          Show
          mingma Ming Ma added a comment - Could the test fail if the node becomes decommissioned right after it checks isDecommissionInProgress ? Otherwise, it looks good. if (!checkDecommissionInProgress && datanodeInfo != null && datanodeInfo.isDecommissionInProgress()) { String fsckOut = runFsck(conf, 0, true, testFile, "-files", "-blocks", "-replicaDetails"); assertTrue(fsckOut.contains("(DECOMMISSIONING)")); checkDecommissionInProgress = true; }
          Hide
          vinayrpet Vinayakumar B added a comment - - edited

          In the latest patch attached, the output will change. Individual storage informations will not be printed ex DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK]. Thats because iterating over storages directly, not on the locations returned from located block.

          To retain the old output, Just needs following change. provided getStorageID() and getStorageType() are public.

          -            if (showRacks)
          -              sb.append(NodeBase.getPath(locs[j]));
          -            else
          -              sb.append(locs[j]);
          +            if (showRacks) {
          +              sb.append(NodeBase.getPath(dnDesc));
          +            } else {
          +              sb.append(new DatanodeInfoWithStorage(dnDesc, storage
          +                  .getStorageID(), storage.getStorageType()));
          +            }
          
          Show
          vinayrpet Vinayakumar B added a comment - - edited In the latest patch attached, the output will change. Individual storage informations will not be printed ex DatanodeInfoWithStorage[10.19.92.116:50010,DS-13faa414-cf01-4ae7-b307-2ae87caa273c,DISK] . Thats because iterating over storages directly, not on the locations returned from located block. To retain the old output, Just needs following change. provided getStorageID() and getStorageType() are public. - if (showRacks) - sb.append(NodeBase.getPath(locs[j])); - else - sb.append(locs[j]); + if (showRacks) { + sb.append(NodeBase.getPath(dnDesc)); + } else { + sb.append( new DatanodeInfoWithStorage(dnDesc, storage + .getStorageID(), storage.getStorageType())); + }
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi andreina,
          Patch looks almost good with the updated test.
          One update required as mentioned in prev comment.

          Could the test fail if the node becomes decommissioned right after it checks isDecommissionInProgress? Otherwise, it looks good.

          yes, you are right Ming Ma, Since there are 2 DNs already available, by the time fsck executed and seen, decommissioned DN might be moved to DECOMMISSIONED soon.
          To slow it down, I recommend to start only one node at the beginning of cluster. And once the DECOMMISSIONING state is verified in fsck, start another datanode and verify for the DECOMMISSIONED.

          Few more nits to be fixed in test.
          1. Unnecessary assertion + assertNotNull("Failed Cluster Creation", cluster);, as if building fails, then it will throw exception directly.
          2. For the current usage of DFSTestUtil, need not build it using Builder. directly can use static methods.

          +    DFSTestUtil util =
          +        new DFSTestUtil.Builder().setName(getClass().getSimpleName()).setNumFiles(1).build();
          

          3. + int count = 0; is not used. Either this could should be used in while loop as a condition. Also I recommend adding @Timeout annotation to test.

          Show
          vinayrpet Vinayakumar B added a comment - Hi andreina, Patch looks almost good with the updated test. One update required as mentioned in prev comment. Could the test fail if the node becomes decommissioned right after it checks isDecommissionInProgress? Otherwise, it looks good. yes, you are right Ming Ma , Since there are 2 DNs already available, by the time fsck executed and seen, decommissioned DN might be moved to DECOMMISSIONED soon. To slow it down, I recommend to start only one node at the beginning of cluster. And once the DECOMMISSIONING state is verified in fsck, start another datanode and verify for the DECOMMISSIONED. Few more nits to be fixed in test. 1. Unnecessary assertion + assertNotNull("Failed Cluster Creation", cluster); , as if building fails, then it will throw exception directly. 2. For the current usage of DFSTestUtil, need not build it using Builder. directly can use static methods. + DFSTestUtil util = + new DFSTestUtil.Builder().setName(getClass().getSimpleName()).setNumFiles(1).build(); 3. + int count = 0; is not used. Either this could should be used in while loop as a condition. Also I recommend adding @Timeout annotation to test.
          Hide
          andreina J.Andreina added a comment -

          Thanks Ming Ma and Vinayakumar B for reviewing and correcting me.
          I have Updated the patch addressing all the comments.
          Please review .

          Show
          andreina J.Andreina added a comment - Thanks Ming Ma and Vinayakumar B for reviewing and correcting me. I have Updated the patch addressing all the comments. Please review .
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks J.Andreina for the latest patch.
          +1.
          Waiting for jenkins

          Show
          vinayrpet Vinayakumar B added a comment - Thanks J.Andreina for the latest patch. +1. Waiting for jenkins
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726802/HDFS-7993.6.patch
          against trunk revision d52de61.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10331//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10331//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726802/HDFS-7993.6.patch against trunk revision d52de61. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10331//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10331//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed trunk and branch-2.
          Thanks J.Andreina for the great contribution.
          Thanks Ming Ma and Colin P. McCabe for great suggestions and reviews.

          Show
          vinayrpet Vinayakumar B added a comment - Committed trunk and branch-2. Thanks J.Andreina for the great contribution. Thanks Ming Ma and Colin P. McCabe for great suggestions and reviews.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7623 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7623/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7623 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7623/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/)
          HDFS-7993. Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/ ) HDFS-7993 . Provide each Replica details in fsck (Contributed by J.Andreina) (vinayakumarb: rev 8ddbb8dd433862509bd9b222dddafe2c3a74778a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8548/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8548/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/)
          Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/ ) Addendum fix for HDFS-7749 to be compatible with HDFS-7993 (zhezhang: rev 9af1f4779b646fb2f09b5e36447c8b8abe920a7c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java

            People

            • Assignee:
              andreina J.Andreina
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development