Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9205

Do not schedule corrupt blocks for replication

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Corrupted blocks by definition are blocks cannot be read. As a consequence, they cannot be replicated. In UnderReplicatedBlocks, there is a queue for QUEUE_WITH_CORRUPT_BLOCKS and chooseUnderReplicatedBlocks may choose blocks from it. It seems that scheduling corrupted block for replication is wasting resource and potentially slow down replication for the higher priority blocks.

      1. h9205_20151007.patch
        9 kB
        Tsz Wo Nicholas Sze
      2. h9205_20151007b.patch
        12 kB
        Tsz Wo Nicholas Sze
      3. h9205_20151008.patch
        32 kB
        Tsz Wo Nicholas Sze
      4. h9205_20151009.patch
        32 kB
        Tsz Wo Nicholas Sze
      5. h9205_20151009b.patch
        32 kB
        Tsz Wo Nicholas Sze
      6. h9205_20151013.patch
        32 kB
        Tsz Wo Nicholas Sze
      7. h9205_20151015.patch
        32 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151007.patch: do not choose corrupted blocks.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151007.patch: do not choose corrupted blocks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 5s There were no new javac warning messages.
        +1 javadoc 10m 26s There were no new javadoc warning messages.
        -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 24s The applied patch generated 5 new checkstyle issues (total was 198, now 200).
        -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 189m 36s Tests failed in hadoop-hdfs.
            236m 0s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765341/h9205_20151007.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3112f26
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12832/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12832/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 5s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 24s The applied patch generated 5 new checkstyle issues (total was 198, now 200). -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 189m 36s Tests failed in hadoop-hdfs.     236m 0s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765341/h9205_20151007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3112f26 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12832/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12832/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151007b.patch: fixes TestReplicationPolicy.

        The other failed tests do not seem related.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151007b.patch: fixes TestReplicationPolicy. The other failed tests do not seem related.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 19s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 58s There were no new javac warning messages.
        +1 javadoc 11m 56s There were no new javadoc warning messages.
        -1 release audit 0m 23s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 198, now 199).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 46s mvn install still works.
        +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
        +1 findbugs 2m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 39s Pre-build of native portion
        -1 hdfs tests 232m 18s Tests failed in hadoop-hdfs.
            284m 38s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          hadoop.hdfs.TestDataTransferKeepalive
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765399/h9205_20151007b.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 99e5204
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12835/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12835/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 58s There were no new javac warning messages. +1 javadoc 11m 56s There were no new javadoc warning messages. -1 release audit 0m 23s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 198, now 199). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 2m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 39s Pre-build of native portion -1 hdfs tests 232m 18s Tests failed in hadoop-hdfs.     284m 38s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765399/h9205_20151007b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 99e5204 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12835/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12835/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        I think it makes sense to skip scheduling replication for corrupted blocks. For the patch, one comment is that in the following code the for loop in next actually should be in hasNext method. Other than this the patch looks good to me.

              @Override
              public BlockInfo next() {
                for(; !b.hasNext() && q.hasNext(); ) {
                  b = q.next().iterator();
                }
                return b.next();
              }
        
              @Override
              public boolean hasNext() {
                return b.hasNext();
              }
        
        Show
        jingzhao Jing Zhao added a comment - I think it makes sense to skip scheduling replication for corrupted blocks. For the patch, one comment is that in the following code the for loop in next actually should be in hasNext method. Other than this the patch looks good to me. @Override public BlockInfo next() { for (; !b.hasNext() && q.hasNext(); ) { b = q.next().iterator(); } return b.next(); } @Override public boolean hasNext() { return b.hasNext(); }
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        You are right. Both hasNext() and next() need to advance the iterators.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - You are right. Both hasNext() and next() need to advance the iterators.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        The failure of TestReadOnlySharedStorage actually is related – the current implementation of read-only storage breaks the corrupt block definition. It treats blocks with read-only replicas but no normal replicas as corrupt replicas.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - The failure of TestReadOnlySharedStorage actually is related – the current implementation of read-only storage breaks the corrupt block definition. It treats blocks with read-only replicas but no normal replicas as corrupt replicas.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151008.patch: treats block with readonly replicas but no normal replicas as highest priority for replication.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151008.patch: treats block with readonly replicas but no normal replicas as highest priority for replication.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Nicholas for the work. A few comments:

        1. As a consequence, they cannot be replicated

          Just to clarify, do you mean that even without the patch, those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)?

        2. I agree that corrupt blocks are unreadable by HDFS client. But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? For example, there's a chance (although very rare) that the replica is intact and only the metadata file is corrupt.
        3. If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether?

        Nit:

        1. This line of comment should be updated:
          // and 5 blocks from QUEUE_WITH_CORRUPT_BLOCKS.
          
        Show
        zhz Zhe Zhang added a comment - Thanks Nicholas for the work. A few comments: As a consequence, they cannot be replicated Just to clarify, do you mean that even without the patch, those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)? I agree that corrupt blocks are unreadable by HDFS client. But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? For example, there's a chance (although very rare) that the replica is intact and only the metadata file is corrupt. If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether? Nit: This line of comment should be updated: // and 5 blocks from QUEUE_WITH_CORRUPT_BLOCKS.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 26m 26s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 13m 11s There were no new javac warning messages.
        +1 javadoc 14m 50s There were no new javadoc warning messages.
        -1 release audit 0m 28s The applied patch generated 1 release audit warnings.
        -1 checkstyle 2m 19s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 2m 13s mvn install still works.
        +1 eclipse:eclipse 0m 54s The patch built with eclipse:eclipse.
        +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 4m 57s Pre-build of native portion
        -1 hdfs tests 127m 24s Tests failed in hadoop-hdfs.
            196m 27s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
        Timed out tests org.apache.hadoop.hdfs.server.namenode.TestINodeFile
          org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765475/h9205_20151008.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / fde729f
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12845/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12845/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 26m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 13m 11s There were no new javac warning messages. +1 javadoc 14m 50s There were no new javadoc warning messages. -1 release audit 0m 28s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 19s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 2m 13s mvn install still works. +1 eclipse:eclipse 0m 54s The patch built with eclipse:eclipse. +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 4m 57s Pre-build of native portion -1 hdfs tests 127m 24s Tests failed in hadoop-hdfs.     196m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks Timed out tests org.apache.hadoop.hdfs.server.namenode.TestINodeFile   org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765475/h9205_20151008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fde729f Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12845/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12845/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Thanks Zhe for the comments.

        > ... those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)?

        Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.)

        > ... But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? ...

        This patch does not prevent that.

        > If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether?

        The block priority could possibly be updated.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Zhe for the comments. > ... those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)? Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.) > ... But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? ... This patch does not prevent that. > If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether? The block priority could possibly be updated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151009.patch: removes a comment in a test for addressing Zhe's review comment.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151009.patch: removes a comment in a test for addressing Zhe's review comment.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151009b.patch: fixes TestAddOverReplicatedStripedBlocks.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151009b.patch: fixes TestAddOverReplicatedStripedBlocks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 21m 17s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 55s There were no new javac warning messages.
        +1 javadoc 10m 28s There were no new javadoc warning messages.
        -1 release audit 0m 21s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 15s Pre-build of native portion
        -1 hdfs tests 187m 57s Tests failed in hadoop-hdfs.
            238m 27s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.web.TestWebHDFSOAuth2



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765722/h9205_20151009.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e1bf8b3
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12878/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12878/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 55s There were no new javac warning messages. +1 javadoc 10m 28s There were no new javadoc warning messages. -1 release audit 0m 21s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 187m 57s Tests failed in hadoop-hdfs.     238m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765722/h9205_20151009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e1bf8b3 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12878/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12878/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 19s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 11s There were no new javac warning messages.
        +1 javadoc 10m 16s There were no new javadoc warning messages.
        -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 184m 37s Tests failed in hadoop-hdfs.
            231m 3s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
          hadoop.hdfs.TestDFSShell
        Timed out tests org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.TestDecommission



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e1bf8b3
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12880/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12880/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 11s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 184m 37s Tests failed in hadoop-hdfs.     231m 3s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestDFSShell Timed out tests org.apache.hadoop.hdfs.TestReplication   org.apache.hadoop.hdfs.TestDecommission Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e1bf8b3 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12880/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12880/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Nicholas.

        Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.)

        Right, those blocks only have corrupt replicas. Before trying to replicate a block replica DN validates it based on almost the same conditions as NN's corrupt replica logic, with the following exception:

        // DataNode#transferBlock
            } catch (EOFException e) {
              lengthTooShort = true;
        

        Basically, DN skips a replica only if it's too short, while NN considers a replica as corrupt when the size is different (larger or smaller) than the NN's copy.

        The above is a very rare corner case, and I agree this is a good change to cut unnecessary NN=>DN traffic for tasks that will be filtered out later anyway.

        Show
        zhz Zhe Zhang added a comment - Thanks Nicholas. Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.) Right, those blocks only have corrupt replicas. Before trying to replicate a block replica DN validates it based on almost the same conditions as NN's corrupt replica logic, with the following exception: // DataNode#transferBlock } catch (EOFException e) { lengthTooShort = true ; Basically, DN skips a replica only if it's too short, while NN considers a replica as corrupt when the size is different (larger or smaller) than the NN's copy. The above is a very rare corner case, and I agree this is a good change to cut unnecessary NN=>DN traffic for tasks that will be filtered out later anyway.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 8s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 10m 17s There were no new javadoc warning messages.
        -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 22s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 10s Pre-build of native portion
        -1 hdfs tests 188m 20s Tests failed in hadoop-hdfs.
            234m 3s  



        Reason Tests
        Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / db93047
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12925/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12925/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 17s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 22s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 188m 20s Tests failed in hadoop-hdfs.     234m 3s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / db93047 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12925/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12925/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -
        1. Nit: need to fix the javadoc of UnderReplicatedBlocks, "getPriority(BlockInfo, int, int, int)" should be updated to "getPriority(BlockInfo, int, int, int, int)".
        2. Minor: since the iterator of the LightWeightLinkedSet already correctly throws NoSuchElementException when there is no next element, it may not be necessary to do the hasNext check.
                public BlockInfo next() {
                  if (!hasNext()) {
                    throw new NoSuchElementException();
                  }
                  return b.next();
                }
          

        +1 after addressing these.

        Show
        jingzhao Jing Zhao added a comment - Nit: need to fix the javadoc of UnderReplicatedBlocks , "getPriority(BlockInfo, int, int, int)" should be updated to "getPriority(BlockInfo, int, int, int, int)". Minor: since the iterator of the LightWeightLinkedSet already correctly throws NoSuchElementException when there is no next element, it may not be necessary to do the hasNext check. public BlockInfo next() { if (!hasNext()) { throw new NoSuchElementException(); } return b.next(); } +1 after addressing these.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151013.patch: addresses Jing's comments.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151013.patch: addresses Jing's comments.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 30s There were no new javac warning messages.
        +1 javadoc 10m 22s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 25s The applied patch generated 7 new checkstyle issues (total was 202, now 205).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 187m 5s Tests failed in hadoop-hdfs.
            234m 5s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.web.TestWebHDFSOAuth2



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766253/h9205_20151013.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / c60a16f
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12946/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12946/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 30s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 7 new checkstyle issues (total was 202, now 205). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 187m 5s Tests failed in hadoop-hdfs.     234m 5s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766253/h9205_20151013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c60a16f checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12946/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12946/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        h9205_20151015.patch: updates with trunk.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151015.patch: updates with trunk.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 50s There were no new javac warning messages.
        +1 javadoc 11m 14s There were no new javadoc warning messages.
        -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 201, now 204).
        -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 2m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 25s Pre-build of native portion
        -1 hdfs tests 65m 45s Tests failed in hadoop-hdfs.
            116m 36s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.fs.TestGlobPaths



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766712/h9205_20151015.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / c80b3a8
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12998/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12998/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 50s There were no new javac warning messages. +1 javadoc 11m 14s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 201, now 204). -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 25s Pre-build of native portion -1 hdfs tests 65m 45s Tests failed in hadoop-hdfs.     116m 36s   Reason Tests Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.fs.TestGlobPaths Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766712/h9205_20151015.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c80b3a8 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12998/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12998/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        The failed tests are not related.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - The failed tests are not related.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8641 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8641/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8641 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8641/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Thanks Jing for reviewing the patch.

        I have committed this.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jing for reviewing the patch. I have committed this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8642/)
        Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8642/ ) Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1271 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1271/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1271 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1271/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/535/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/535/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #548 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/548/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #548 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/548/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2484 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2484/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2484 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2484/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2438 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2438/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2438 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2438/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/501/)
        HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/501/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Merged this to branch-2.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Merged this to branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8650/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8650/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1278/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1278/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #557 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/557/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #557 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/557/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/542/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/542/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2491/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2491/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2443 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2443/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2443 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2443/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/506/)
        Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/506/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks for the change Nicholas! Should this line be modified? https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java#L62 .

        I think most often this queue had missing blocks, so it didn't really make sense to re-replicate missing blocks anyway. We should be careful about removing this queue though, because its where the count of missing blocks is taken from

        Show
        raviprak Ravi Prakash added a comment - Thanks for the change Nicholas! Should this line be modified? https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java#L62 . I think most often this queue had missing blocks, so it didn't really make sense to re-replicate missing blocks anyway. We should be careful about removing this queue though, because its where the count of missing blocks is taken from

          People

          • Assignee:
            szetszwo Tsz Wo Nicholas Sze
            Reporter:
            szetszwo Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development