Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7281

Missing block is marked as corrupted block

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The patch improves the reporting around missing blocks and corrupted blocks.
       
      1. A block is missing if and only if all DNs of its expected replicas are dead.
      2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.
      3. A new line is added to fsck output to display the corrupt block size per file.
      4. A new line is added to fsck output to display the number of missing blocks in the summary section.
      Show
      The patch improves the reporting around missing blocks and corrupted blocks.   1. A block is missing if and only if all DNs of its expected replicas are dead. 2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted. 3. A new line is added to fsck output to display the corrupt block size per file. 4. A new line is added to fsck output to display the number of missing blocks in the summary section.

      Description

      In the situation where the block lost all its replicas, fsck shows the block is missing as well as corrupted. Perhaps it is better not to mark the block corrupted in this case. The reason it is marked as corrupted is numCorruptNodes == numNodes == 0 in the following code.

      BlockManager
          final boolean isCorrupt = numCorruptNodes == numNodes;
      

      Would like to clarify if it is the intent to mark missing block as corrupted or it is just a bug.

      1. HDFS-7281.patch
        5 kB
        Ming Ma
      2. HDFS-7281-2.patch
        5 kB
        Ming Ma
      3. HDFS-7281-3.patch
        5 kB
        Ming Ma
      4. HDFS-7281-4.patch
        8 kB
        Ming Ma
      5. HDFS-7281-5.patch
        9 kB
        Ming Ma
      6. HDFS-7281-6.patch
        9 kB
        Ming Ma

        Activity

        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks reporting this issue Ming Ma. I happen to notice the same in a fsck report today. It's indeed confusing.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks reporting this issue Ming Ma . I happen to notice the same in a fsck report today. It's indeed confusing.
        Hide
        mingma Ming Ma added a comment -

        Thanks, Yongjun. Besides missing block is marked as corrupted block, corrupted block is marked as missing block; that is because corrupted block has no live replica. So it will be useful to clarify that is the intention as well.

        So here is the patch that separates missing block from corrupted block. Will appreciate others' opinions on the definition of missing and corrupted block.

        1. A block is missing if and only if all DNs of its expected replicas are dead.
        2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.

        Show
        mingma Ming Ma added a comment - Thanks, Yongjun. Besides missing block is marked as corrupted block, corrupted block is marked as missing block; that is because corrupted block has no live replica. So it will be useful to clarify that is the intention as well. So here is the patch that separates missing block from corrupted block. Will appreciate others' opinions on the definition of missing and corrupted block. 1. A block is missing if and only if all DNs of its expected replicas are dead. 2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming,

        Thanks for the patch. And the definition you gave for missing and corrupted block makes sense to me. Let's see what other folks say.

        I reviewed the patch, it looks good. I have one minor comment here:

        369	        if (numMissing.equals(Integer.toString(totalMissingBlocks)) &&
        370	            numCorrupt.equals(Integer.toString(0))) {
        

        It seems that we could move the checking numCorrupt.equals(Integer.toString(0)) from the if statement above, and make it an assertion statement before this block, because we expect the numCorrupt to be 0 for this test. Right?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming, Thanks for the patch. And the definition you gave for missing and corrupted block makes sense to me. Let's see what other folks say. I reviewed the patch, it looks good. I have one minor comment here: 369 if (numMissing.equals( Integer .toString(totalMissingBlocks)) && 370 numCorrupt.equals( Integer .toString(0))) { It seems that we could move the checking numCorrupt.equals(Integer.toString(0)) from the if statement above, and make it an assertion statement before this block, because we expect the numCorrupt to be 0 for this test. Right? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12676827/HDFS-7281.patch
        against trunk revision 57dec28.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestLeaseRecovery2
        org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8515//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8515//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676827/HDFS-7281.patch against trunk revision 57dec28. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8515//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8515//console This message is automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Thanks, Yongjun. Here is the updated patch to address your comment.

        Show
        mingma Ming Ma added a comment - Thanks, Yongjun. Here is the updated patch to address your comment.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12677079/HDFS-7281-2.patch
        against trunk revision 683897f.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8539//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677079/HDFS-7281-2.patch against trunk revision 683897f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8539//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12677436/HDFS-7281-2.patch
        against trunk revision a16d022.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestLeaseRecovery2

        The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestEncryptionZonesWithHA

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8558//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8558//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677436/HDFS-7281-2.patch against trunk revision a16d022. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRecovery2 The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZonesWithHA +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8558//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8558//console This message is automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        HI Ming Ma,

        Thanks for addressing my comments, the change looks good to me.

        About the test failure, I used the tool from HADOOP-11045 and found the following:

        ****Recently FAILED builds in url: https://builds.apache.org//job/PreCommit-Hdfs-Build
            THERE ARE 95 builds (out of 100) that have failed tests in the past 7 days, as listed below:
        ......
        Among 100 runs examined, all failed tests <#failedRuns: testName>:
            6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser
            6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery
            6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart
            5: org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode
            5: org.apache.hadoop.hdfs.TestLeaseRecovery2.testThreadName
            3: org.apache.hadoop.hdfs.TestDFSClientRetries.testFailuresArePerOperation
        ...
        

        So the TestLeaseReovery2 is not relevant to your change as we expected.

        I suggest that you run locally both the this test and the timeouted one TestEncryptionZonesWithHA and see if they pass with your patch, for completeness.

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - HI Ming Ma , Thanks for addressing my comments, the change looks good to me. About the test failure, I used the tool from HADOOP-11045 and found the following: ****Recently FAILED builds in url: https: //builds.apache.org//job/PreCommit-Hdfs-Build THERE ARE 95 builds (out of 100) that have failed tests in the past 7 days, as listed below: ...... Among 100 runs examined, all failed tests <#failedRuns: testName>: 6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser 6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery 6: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart 5: org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode 5: org.apache.hadoop.hdfs.TestLeaseRecovery2.testThreadName 3: org.apache.hadoop.hdfs.TestDFSClientRetries.testFailuresArePerOperation ... So the TestLeaseReovery2 is not relevant to your change as we expected. I suggest that you run locally both the this test and the timeouted one TestEncryptionZonesWithHA and see if they pass with your patch, for completeness. Thanks.
        Hide
        mingma Ming Ma added a comment -

        Thanks, Yongjun. HADOOP-11045 is useful. Both TestEncryptionZonesWithHA and TestLeaseRecovery2 pass with the local run.

        Show
        mingma Ming Ma added a comment - Thanks, Yongjun. HADOOP-11045 is useful. Both TestEncryptionZonesWithHA and TestLeaseRecovery2 pass with the local run.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Ming Ma.

        Hi Aaron T. Myers, the latest patch looks good to me. I wonder if you would have time to do a review here? thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Ming Ma . Hi Aaron T. Myers , the latest patch looks good to me. I wonder if you would have time to do a review here? thanks.
        Hide
        lohit Lohit Vijayarenu added a comment -

        Hi Ming Ma, These changes look good. This changes format of fsck output, which should be ok to change (unless someone scrapes output of fsck command). May be mark it as incompatible in release notes?

        Show
        lohit Lohit Vijayarenu added a comment - Hi Ming Ma , These changes look good. This changes format of fsck output, which should be ok to change (unless someone scrapes output of fsck command). May be mark it as incompatible in release notes?
        Hide
        yzhangal Yongjun Zhang added a comment -

        HI Lohit Vijayarenu,

        Thanks for your comments. Format-wise, Ming's change added one standalone line of "Missing blocks" to the report. As we don't consider it incompatible when we add new entries to jmx report, I guess we don't have to mark it as incompatible too. Do you agree?

        Hi Ming, sorry about the long delay. the patch doesn't apply anymore, would you please bring the patch up-to-date? I'd like to push it forward. thanks.

        Show
        yzhangal Yongjun Zhang added a comment - HI Lohit Vijayarenu , Thanks for your comments. Format-wise, Ming's change added one standalone line of "Missing blocks" to the report. As we don't consider it incompatible when we add new entries to jmx report, I guess we don't have to mark it as incompatible too. Do you agree? Hi Ming, sorry about the long delay. the patch doesn't apply anymore, would you please bring the patch up-to-date? I'd like to push it forward. thanks.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12677436/HDFS-7281-2.patch
        against trunk revision 6779467.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10305//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677436/HDFS-7281-2.patch against trunk revision 6779467. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10305//console This message is automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Thanks, Yongjun Zhang. Here is the rebased patch.

        Show
        mingma Ming Ma added a comment - Thanks, Yongjun Zhang . Here is the rebased patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12726301/HDFS-7281-3.patch
        against trunk revision 6779467.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10307//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10307//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726301/HDFS-7281-3.patch against trunk revision 6779467. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10307//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10307//console This message is automatically generated.
        Hide
        mingma Ming Ma added a comment -

        For the incompatibility issue Lohit mentioned, if some tools expect "Corrupt blocks" line follows right after "Average block replication" line, that could break. Do we have any definition for text output incompatibility? Thanks, Lohit Vijayarenu and Yongjun Zhang.

        Show
        mingma Ming Ma added a comment - For the incompatibility issue Lohit mentioned, if some tools expect "Corrupt blocks" line follows right after "Average block replication" line, that could break. Do we have any definition for text output incompatibility? Thanks, Lohit Vijayarenu and Yongjun Zhang .
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming Ma,

        Thanks for the updated patch, it looks good to me.

        I agree that

        if some tools expect "Corrupt blocks" line follows right after "Average block replication" line, that could break

        However, personally I think that the tool is too vulnerable if it counts on the exact sequence of the fsck report. I know that adding one entry in the jmx report is not considered incompatible.

        We can certainly add a description to the release notes about the newly added line.

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming Ma , Thanks for the updated patch, it looks good to me. I agree that if some tools expect "Corrupt blocks" line follows right after "Average block replication" line, that could break However, personally I think that the tool is too vulnerable if it counts on the exact sequence of the fsck report. I know that adding one entry in the jmx report is not considered incompatible. We can certainly add a description to the release notes about the newly added line. Thanks.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming,

        I noticed that we have "missize" to record size of all missing blocks for a given path, and "missingSize" for all missing blocks of all paths checked by fsck. Since now we distinguish missing blocks and corrupt blocks and report them separately, it seems reasonable to also have a similar recording about the size of data for corrupted blocks. What do you think?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming, I noticed that we have "missize" to record size of all missing blocks for a given path, and "missingSize" for all missing blocks of all paths checked by fsck. Since now we distinguish missing blocks and corrupt blocks and report them separately, it seems reasonable to also have a similar recording about the size of data for corrupted blocks. What do you think? Thanks.
        Hide
        mingma Ming Ma added a comment -

        Yongjun, that makes sense. Here is the updated patch.

        Show
        mingma Ming Ma added a comment - Yongjun, that makes sense. Here is the updated patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12726794/HDFS-7281-4.patch
        against trunk revision d52de61.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10329//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10329//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726794/HDFS-7281-4.patch against trunk revision d52de61. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10329//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10329//console This message is automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming Ma,

        Thanks for the new rev and sorry for being late. It looks good, only one comment here:

        We may change:

              if (missing > 0) {
                out.print(" MISSING " + missing + " blocks of total size " + missize + " B\n");
              }
              if (corrupt > 0) {
                out.print(" CORRUPT " + corrupt + " blocks of total size " +
                    corruptSize + " B\n");
              }
              if (underReplicatedPerFile == 0 && misReplicatedPerFile == 0) {
                out.print(" OK\n");
              }
        

        to

              if (missing > 0 || corrupt > 0) {
                if (missing > 0) {
                  out.print(" MISSING " + missing + " blocks of total size " + missize + " B\n");
                }
                if (corrupt > 0) {
                  out.print(" CORRUPT " + corrupt + " blocks of total size " +
                    corruptSize + " B\n");
                }
              } else if (underReplicatedPerFile == 0 && misReplicatedPerFile == 0) {
                out.print(" OK\n");
              }
        

        such that if there are missing/corrupt blocks, we don't report OK. Right?

        BTW, Would you please write a release notes about this fsck report change?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming Ma , Thanks for the new rev and sorry for being late. It looks good, only one comment here: We may change: if (missing > 0) { out.print( " MISSING " + missing + " blocks of total size " + missize + " B\n" ); } if (corrupt > 0) { out.print( " CORRUPT " + corrupt + " blocks of total size " + corruptSize + " B\n" ); } if (underReplicatedPerFile == 0 && misReplicatedPerFile == 0) { out.print( " OK\n" ); } to if (missing > 0 || corrupt > 0) { if (missing > 0) { out.print( " MISSING " + missing + " blocks of total size " + missize + " B\n" ); } if (corrupt > 0) { out.print( " CORRUPT " + corrupt + " blocks of total size " + corruptSize + " B\n" ); } } else if (underReplicatedPerFile == 0 && misReplicatedPerFile == 0) { out.print( " OK\n" ); } such that if there are missing/corrupt blocks, we don't report OK. Right? BTW, Would you please write a release notes about this fsck report change? Thanks.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Ming, Your earlier comment:

        https://issues.apache.org/jira/browse/HDFS-7281?focusedCommentId=14180898&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14180898

        already has a good description as 1 and 2. I think adding a third line on top of these describing the corruptSize reporting as a counterpart of missingSize would make a good release notes.

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Ming, Your earlier comment: https://issues.apache.org/jira/browse/HDFS-7281?focusedCommentId=14180898&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14180898 already has a good description as 1 and 2. I think adding a third line on top of these describing the corruptSize reporting as a counterpart of missingSize would make a good release notes. Thanks.
        Hide
        aw Allen Wittenauer added a comment -

        Yes, output changes to command lines tools are definitely incompatible changes. This is covered in http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI .

        Show
        aw Allen Wittenauer added a comment - Yes, output changes to command lines tools are definitely incompatible changes. This is covered in http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI .
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Allen Wittenauer and Chris Nauroth for addressing my questions in the email thread.

        What about adding new entries to jmx report? Somehow I had the impression that if we add new entries to it, it's not considered incompatible. Often within the same minor release, we want to add new info to jmx report instead of waiting for a major release.

        For CLI like fsck, maybe we can add a new command line option to enable the change, and if the command line option is not enabled, don't change the output, so we can still commit the change within the same release line?
        quote}

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Allen Wittenauer and Chris Nauroth for addressing my questions in the email thread. What about adding new entries to jmx report? Somehow I had the impression that if we add new entries to it, it's not considered incompatible. Often within the same minor release, we want to add new info to jmx report instead of waiting for a major release. For CLI like fsck, maybe we can add a new command line option to enable the change, and if the command line option is not enabled, don't change the output, so we can still commit the change within the same release line? quote} Thanks.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Chris Nauroth for his comment below

        Metrics/JMX is covered by our compatibility guidelines:

        http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Comp
        atibility.html#MetricsJMX

        Metrics/JMX is similar to our usage of Protocol Buffers/JSON that I
        mentioned. It supports backwards-compatible evolution if the change is
        done correctly. Adding new fields/beans is compatible. Changing the
        names or data types of existing fields/beans is incompatible. Deleting
        existing fields/beans is incompatible.

        --Chris Nauroth

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Chris Nauroth for his comment below Metrics/JMX is covered by our compatibility guidelines: http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Comp atibility.html#MetricsJMX Metrics/JMX is similar to our usage of Protocol Buffers/JSON that I mentioned. It supports backwards-compatible evolution if the change is done correctly. Adding new fields/beans is compatible. Changing the names or data types of existing fields/beans is incompatible. Deleting existing fields/beans is incompatible. --Chris Nauroth
        Hide
        aw Allen Wittenauer added a comment -

        It should be noted that if I had to wager, the fsck output is probably the second most automatically processed output for operations people. (With hdfs audit log being first.) Making changes in the code here is definitely disruptive.

        Show
        aw Allen Wittenauer added a comment - It should be noted that if I had to wager, the fsck output is probably the second most automatically processed output for operations people. (With hdfs audit log being first.) Making changes in the code here is definitely disruptive.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Allen Wittenauer, that's very informative.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Allen Wittenauer , that's very informative.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming Ma,

        I think we can get this fix to trunk targetting 3.0, and follow up with other improvement like Andrew Wang proposed in the email thread.

        Would you please take a look at comment at
        https://issues.apache.org/jira/browse/HDFS-7281?focusedCommentId=14510451&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14510451
        ?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming Ma , I think we can get this fix to trunk targetting 3.0, and follow up with other improvement like Andrew Wang proposed in the email thread. Would you please take a look at comment at https://issues.apache.org/jira/browse/HDFS-7281?focusedCommentId=14510451&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14510451 ? Thanks.
        Hide
        mingma Ming Ma added a comment -

        Yongjun Zhang, good catch. I have updated the patch and the release notes.

        Show
        mingma Ming Ma added a comment - Yongjun Zhang , good catch. I have updated the patch and the release notes.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Ming Ma,

        Thanks for the new rev and the release notes! One minor thing, I found three lines touched by this patch exceed 80 chars:

        • Line 852 of BlockManager
        • Line 575 and 703 of NamenodeFsck
          Really sorry I did not catch them in earlier rounds. +1 after that and jenkins.
        Show
        yzhangal Yongjun Zhang added a comment - Hi Ming Ma , Thanks for the new rev and the release notes! One minor thing, I found three lines touched by this patch exceed 80 chars: Line 852 of BlockManager Line 575 and 703 of NamenodeFsck Really sorry I did not catch them in earlier rounds. +1 after that and jenkins.
        Hide
        mingma Ming Ma added a comment -

        Thanks Yongjun Zhang. Here is the updated patch.

        Show
        mingma Ming Ma added a comment - Thanks Yongjun Zhang . Here is the updated patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 26s The applied patch generated 1 additional checkstyle issues.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 3m 7s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 226m 22s Tests failed in hadoop-hdfs.
            272m 36s  



        Reason Tests
        FindBugs module:hadoop-hdfs
          Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java:[lines 177-201]
        Failed unit tests hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.TestDeleteRace
          hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
          hadoop.hdfs.TestFileLengthOnClusterRestart
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestClose
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.server.datanode.TestBlockRecovery
          hadoop.hdfs.TestDFSOutputStream
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.TestDataTransferProtocol
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729274/HDFS-7281-6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3dd6395
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/checkstyle-result-diff.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10454/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10454/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 26s The applied patch generated 1 additional checkstyle issues. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 3m 7s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 226m 22s Tests failed in hadoop-hdfs.     272m 36s   Reason Tests FindBugs module:hadoop-hdfs   Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java: [lines 177-201] Failed unit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestQuota   hadoop.hdfs.TestClose   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestDFSOutputStream Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache   org.apache.hadoop.hdfs.TestDataTransferProtocol   org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729274/HDFS-7281-6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3dd6395 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10454/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10454/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10454/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 25s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 29s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 3m 58s The applied patch generated 1 additional checkstyle issues.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 3m 9s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 225m 9s Tests failed in hadoop-hdfs.
            269m 26s  



        Reason Tests
        FindBugs module:hadoop-hdfs
          Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java:[lines 177-201]
        Failed unit tests hadoop.hdfs.server.datanode.TestBlockRecovery
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestFileLengthOnClusterRestart
          hadoop.hdfs.TestDFSOutputStream
          hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
          hadoop.hdfs.server.namenode.TestDeleteRace
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.TestClose
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.TestDataTransferProtocol
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729274/HDFS-7281-6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e2e8f77
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/checkstyle-result-diff.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10485/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10485/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 29s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 58s The applied patch generated 1 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 3m 9s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 225m 9s Tests failed in hadoop-hdfs.     269m 26s   Reason Tests FindBugs module:hadoop-hdfs   Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java: [lines 177-201] Failed unit tests hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestQuota   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.cli.TestHDFSCLI   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestClose Timed out tests org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache   org.apache.hadoop.hdfs.TestDataTransferProtocol   org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729274/HDFS-7281-6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e2e8f77 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10485/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10485/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10485/console This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        The test failure is unrelated, I reran all failed tests successfully at local machine. I do see that the failed tests appear in multiple jenkins tests for different jiras, something about the test env.

        +1. I will commit by tomorrow.

        Show
        yzhangal Yongjun Zhang added a comment - The test failure is unrelated, I reran all failed tests successfully at local machine. I do see that the failed tests appear in multiple jenkins tests for different jiras, something about the test env. +1. I will commit by tomorrow.
        Hide
        yzhangal Yongjun Zhang added a comment -

        I committed to trunk.

        Thanks Ming Ma for the contribution, Lohit Vijayarenu, Allen Wittenauer, Chris Nauroth and Andrew Wang et al for the compatibility discussion.

        Show
        yzhangal Yongjun Zhang added a comment - I committed to trunk. Thanks Ming Ma for the contribution, Lohit Vijayarenu , Allen Wittenauer , Chris Nauroth and Andrew Wang et al for the compatibility discussion.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #7712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7712/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7712/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        mingma Ming Ma added a comment - Thanks Yongjun Zhang and Lohit Vijayarenu , Allen Wittenauer , Chris Nauroth , Andrew Wang .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/181/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/181/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #915 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/915/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #915 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/915/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/172/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/172/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2113/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2113/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/182/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/182/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2131/)
        HDFS-7281. Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2131/ ) HDFS-7281 . Missing block is marked as corrupted block (Ming Ma via Yongjun Zhang) (yzhang: rev 279958b772c25e0633bd967828b7d27d5c0a6a56) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java

          People

          • Assignee:
            mingma Ming Ma
            Reporter:
            mingma Ming Ma
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development