Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9205

Do not schedule corrupt blocks for replication

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Corrupted blocks by definition are blocks cannot be read. As a consequence, they cannot be replicated. In UnderReplicatedBlocks, there is a queue for QUEUE_WITH_CORRUPT_BLOCKS and chooseUnderReplicatedBlocks may choose blocks from it. It seems that scheduling corrupted block for replication is wasting resource and potentially slow down replication for the higher priority blocks.

      1. h9205_20151007.patch
        9 kB
        Tsz Wo Nicholas Sze
      2. h9205_20151007b.patch
        12 kB
        Tsz Wo Nicholas Sze
      3. h9205_20151008.patch
        32 kB
        Tsz Wo Nicholas Sze
      4. h9205_20151009.patch
        32 kB
        Tsz Wo Nicholas Sze
      5. h9205_20151009b.patch
        32 kB
        Tsz Wo Nicholas Sze
      6. h9205_20151013.patch
        32 kB
        Tsz Wo Nicholas Sze
      7. h9205_20151015.patch
        32 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          raviprak Ravi Prakash added a comment -

          Thanks for the change Nicholas! Should this line be modified? https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java#L62 .

          I think most often this queue had missing blocks, so it didn't really make sense to re-replicate missing blocks anyway. We should be careful about removing this queue though, because its where the count of missing blocks is taken from

          Show
          raviprak Ravi Prakash added a comment - Thanks for the change Nicholas! Should this line be modified? https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java#L62 . I think most often this queue had missing blocks, so it didn't really make sense to re-replicate missing blocks anyway. We should be careful about removing this queue though, because its where the count of missing blocks is taken from
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/506/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #506 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/506/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2443 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2443/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2443 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2443/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2491/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2491/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/542/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #542 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/542/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #557 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/557/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #557 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/557/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1278/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1278/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8650/)
          Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8650 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8650/ ) Revert "Move HDFS-9205 to trunk in CHANGES.txt." (szetszwo: rev a554701fe4402ae30461e2ef165cb60970a202a0) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Merged this to branch-2.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Merged this to branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/501/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/501/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2438 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2438/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2438 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2438/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2484 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2484/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2484 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2484/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #548 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/548/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #548 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/548/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/535/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #535 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/535/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1271 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1271/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
            Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1271 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1271/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8642/)
          Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8642/ ) Move HDFS-9205 to trunk in CHANGES.txt. (szetszwo: rev a49298d585f2cbd3bb81579f6e5d0d7b69126264) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Jing for reviewing the patch.

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jing for reviewing the patch. I have committed this.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8641 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8641/)
          HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8641 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8641/ ) HDFS-9205 . Do not schedule corrupt blocks for replication. (szetszwo) (szetszwo: rev 5411dc559d5f73e4153e76fdff94a26869c17a37) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlockQueues.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The failed tests are not related.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The failed tests are not related.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 50s There were no new javac warning messages.
          +1 javadoc 11m 14s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 201, now 204).
          -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 2m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 25s Pre-build of native portion
          -1 hdfs tests 65m 45s Tests failed in hadoop-hdfs.
              116m 36s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.fs.TestGlobPaths



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766712/h9205_20151015.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c80b3a8
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12998/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12998/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 50s There were no new javac warning messages. +1 javadoc 11m 14s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 201, now 204). -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 25s Pre-build of native portion -1 hdfs tests 65m 45s Tests failed in hadoop-hdfs.     116m 36s   Reason Tests Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.fs.TestGlobPaths Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766712/h9205_20151015.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c80b3a8 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12998/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12998/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12998/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151015.patch: updates with trunk.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151015.patch: updates with trunk.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 30s There were no new javac warning messages.
          +1 javadoc 10m 22s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 7 new checkstyle issues (total was 202, now 205).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 187m 5s Tests failed in hadoop-hdfs.
              234m 5s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.web.TestWebHDFSOAuth2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766253/h9205_20151013.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c60a16f
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12946/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12946/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 30s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 7 new checkstyle issues (total was 202, now 205). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 187m 5s Tests failed in hadoop-hdfs.     234m 5s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766253/h9205_20151013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c60a16f checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12946/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12946/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12946/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151013.patch: addresses Jing's comments.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151013.patch: addresses Jing's comments.
          Hide
          jingzhao Jing Zhao added a comment -
          1. Nit: need to fix the javadoc of UnderReplicatedBlocks, "getPriority(BlockInfo, int, int, int)" should be updated to "getPriority(BlockInfo, int, int, int, int)".
          2. Minor: since the iterator of the LightWeightLinkedSet already correctly throws NoSuchElementException when there is no next element, it may not be necessary to do the hasNext check.
                  public BlockInfo next() {
                    if (!hasNext()) {
                      throw new NoSuchElementException();
                    }
                    return b.next();
                  }
            

          +1 after addressing these.

          Show
          jingzhao Jing Zhao added a comment - Nit: need to fix the javadoc of UnderReplicatedBlocks , "getPriority(BlockInfo, int, int, int)" should be updated to "getPriority(BlockInfo, int, int, int, int)". Minor: since the iterator of the LightWeightLinkedSet already correctly throws NoSuchElementException when there is no next element, it may not be necessary to do the hasNext check. public BlockInfo next() { if (!hasNext()) { throw new NoSuchElementException(); } return b.next(); } +1 after addressing these.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 10m 17s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 10s Pre-build of native portion
          -1 hdfs tests 188m 20s Tests failed in hadoop-hdfs.
              234m 3s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / db93047
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12925/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12925/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 17s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 22s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 188m 20s Tests failed in hadoop-hdfs.     234m 3s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / db93047 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12925/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12925/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12925/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Nicholas.

          Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.)

          Right, those blocks only have corrupt replicas. Before trying to replicate a block replica DN validates it based on almost the same conditions as NN's corrupt replica logic, with the following exception:

          // DataNode#transferBlock
              } catch (EOFException e) {
                lengthTooShort = true;
          

          Basically, DN skips a replica only if it's too short, while NN considers a replica as corrupt when the size is different (larger or smaller) than the NN's copy.

          The above is a very rare corner case, and I agree this is a good change to cut unnecessary NN=>DN traffic for tasks that will be filtered out later anyway.

          Show
          zhz Zhe Zhang added a comment - Thanks Nicholas. Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.) Right, those blocks only have corrupt replicas. Before trying to replicate a block replica DN validates it based on almost the same conditions as NN's corrupt replica logic, with the following exception: // DataNode#transferBlock } catch (EOFException e) { lengthTooShort = true ; Basically, DN skips a replica only if it's too short, while NN considers a replica as corrupt when the size is different (larger or smaller) than the NN's copy. The above is a very rare corner case, and I agree this is a good change to cut unnecessary NN=>DN traffic for tasks that will be filtered out later anyway.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 11s There were no new javac warning messages.
          +1 javadoc 10m 16s There were no new javadoc warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 184m 37s Tests failed in hadoop-hdfs.
              231m 3s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.TestDFSShell
          Timed out tests org.apache.hadoop.hdfs.TestReplication
            org.apache.hadoop.hdfs.TestDecommission



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e1bf8b3
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12880/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12880/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 11s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 28s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 184m 37s Tests failed in hadoop-hdfs.     231m 3s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestDFSShell Timed out tests org.apache.hadoop.hdfs.TestReplication   org.apache.hadoop.hdfs.TestDecommission Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765729/h9205_20151009b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e1bf8b3 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12880/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12880/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12880/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 17s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 55s There were no new javac warning messages.
          +1 javadoc 10m 28s There were no new javadoc warning messages.
          -1 release audit 0m 21s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 187m 57s Tests failed in hadoop-hdfs.
              238m 27s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            hadoop.hdfs.web.TestWebHDFS
            hadoop.hdfs.web.TestWebHDFSOAuth2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765722/h9205_20151009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e1bf8b3
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12878/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12878/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 55s There were no new javac warning messages. +1 javadoc 10m 28s There were no new javadoc warning messages. -1 release audit 0m 21s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 30s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 187m 57s Tests failed in hadoop-hdfs.     238m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765722/h9205_20151009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e1bf8b3 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12878/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12878/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12878/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151009b.patch: fixes TestAddOverReplicatedStripedBlocks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151009b.patch: fixes TestAddOverReplicatedStripedBlocks.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151009.patch: removes a comment in a test for addressing Zhe's review comment.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151009.patch: removes a comment in a test for addressing Zhe's review comment.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Zhe for the comments.

          > ... those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)?

          Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.)

          > ... But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? ...

          This patch does not prevent that.

          > If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether?

          The block priority could possibly be updated.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Zhe for the comments. > ... those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)? Those blocks have zero replicas so that it is impossible to replicate them. (Let's ignore read-only storage here since it is an incomplete feature.) > ... But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? ... This patch does not prevent that. > If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether? The block priority could possibly be updated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 26m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 13m 11s There were no new javac warning messages.
          +1 javadoc 14m 50s There were no new javadoc warning messages.
          -1 release audit 0m 28s The applied patch generated 1 release audit warnings.
          -1 checkstyle 2m 19s The applied patch generated 8 new checkstyle issues (total was 203, now 207).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 2m 13s mvn install still works.
          +1 eclipse:eclipse 0m 54s The patch built with eclipse:eclipse.
          +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 4m 57s Pre-build of native portion
          -1 hdfs tests 127m 24s Tests failed in hadoop-hdfs.
              196m 27s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          Timed out tests org.apache.hadoop.hdfs.server.namenode.TestINodeFile
            org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765475/h9205_20151008.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fde729f
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12845/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12845/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 26m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 13m 11s There were no new javac warning messages. +1 javadoc 14m 50s There were no new javadoc warning messages. -1 release audit 0m 28s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 19s The applied patch generated 8 new checkstyle issues (total was 203, now 207). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 2m 13s mvn install still works. +1 eclipse:eclipse 0m 54s The patch built with eclipse:eclipse. +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 4m 57s Pre-build of native portion -1 hdfs tests 127m 24s Tests failed in hadoop-hdfs.     196m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks Timed out tests org.apache.hadoop.hdfs.server.namenode.TestINodeFile   org.apache.hadoop.hdfs.server.datanode.TestTriggerBlockReport Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765475/h9205_20151008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fde729f Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12845/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12845/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12845/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Nicholas for the work. A few comments:

          1. As a consequence, they cannot be replicated

            Just to clarify, do you mean that even without the patch, those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)?

          2. I agree that corrupt blocks are unreadable by HDFS client. But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? For example, there's a chance (although very rare) that the replica is intact and only the metadata file is corrupt.
          3. If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether?

          Nit:

          1. This line of comment should be updated:
            // and 5 blocks from QUEUE_WITH_CORRUPT_BLOCKS.
            
          Show
          zhz Zhe Zhang added a comment - Thanks Nicholas for the work. A few comments: As a consequence, they cannot be replicated Just to clarify, do you mean that even without the patch, those blocks won't be re-replicated, even though chooseUnderReplicatedBlocks returns them? Or they are re-replicated in the current logic, but they should not be (IIUC that's the case)? I agree that corrupt blocks are unreadable by HDFS client. But is there a use case for an admin to list corrupt blocks and reason about them by accessing the local blk_ (and metadata) files? For example, there's a chance (although very rare) that the replica is intact and only the metadata file is corrupt. If we do want to save the replication work for corrupt blocks, should we get rid of QUEUE_WITH_CORRUPT_BLOCKS altogether? Nit: This line of comment should be updated: // and 5 blocks from QUEUE_WITH_CORRUPT_BLOCKS.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151008.patch: treats block with readonly replicas but no normal replicas as highest priority for replication.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151008.patch: treats block with readonly replicas but no normal replicas as highest priority for replication.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The failure of TestReadOnlySharedStorage actually is related – the current implementation of read-only storage breaks the corrupt block definition. It treats blocks with read-only replicas but no normal replicas as corrupt replicas.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The failure of TestReadOnlySharedStorage actually is related – the current implementation of read-only storage breaks the corrupt block definition. It treats blocks with read-only replicas but no normal replicas as corrupt replicas.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          You are right. Both hasNext() and next() need to advance the iterators.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - You are right. Both hasNext() and next() need to advance the iterators.
          Hide
          jingzhao Jing Zhao added a comment -

          I think it makes sense to skip scheduling replication for corrupted blocks. For the patch, one comment is that in the following code the for loop in next actually should be in hasNext method. Other than this the patch looks good to me.

                @Override
                public BlockInfo next() {
                  for(; !b.hasNext() && q.hasNext(); ) {
                    b = q.next().iterator();
                  }
                  return b.next();
                }
          
                @Override
                public boolean hasNext() {
                  return b.hasNext();
                }
          
          Show
          jingzhao Jing Zhao added a comment - I think it makes sense to skip scheduling replication for corrupted blocks. For the patch, one comment is that in the following code the for loop in next actually should be in hasNext method. Other than this the patch looks good to me. @Override public BlockInfo next() { for (; !b.hasNext() && q.hasNext(); ) { b = q.next().iterator(); } return b.next(); } @Override public boolean hasNext() { return b.hasNext(); }
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 19s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 58s There were no new javac warning messages.
          +1 javadoc 11m 56s There were no new javadoc warning messages.
          -1 release audit 0m 23s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 198, now 199).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 46s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          +1 findbugs 2m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 39s Pre-build of native portion
          -1 hdfs tests 232m 18s Tests failed in hadoop-hdfs.
              284m 38s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765399/h9205_20151007b.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 99e5204
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12835/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12835/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 58s There were no new javac warning messages. +1 javadoc 11m 56s There were no new javadoc warning messages. -1 release audit 0m 23s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 35s The applied patch generated 4 new checkstyle issues (total was 198, now 199). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 2m 58s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 39s Pre-build of native portion -1 hdfs tests 232m 18s Tests failed in hadoop-hdfs.     284m 38s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765399/h9205_20151007b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 99e5204 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12835/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12835/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12835/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151007b.patch: fixes TestReplicationPolicy.

          The other failed tests do not seem related.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151007b.patch: fixes TestReplicationPolicy. The other failed tests do not seem related.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 5s There were no new javac warning messages.
          +1 javadoc 10m 26s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 24s The applied patch generated 5 new checkstyle issues (total was 198, now 200).
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 189m 36s Tests failed in hadoop-hdfs.
              236m 0s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.qjournal.client.TestQJMWithFaults
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765341/h9205_20151007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3112f26
          Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12832/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12832/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 5s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 24s The applied patch generated 5 new checkstyle issues (total was 198, now 200). -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 189m 36s Tests failed in hadoop-hdfs.     236m 0s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765341/h9205_20151007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3112f26 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12832/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12832/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12832/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h9205_20151007.patch: do not choose corrupted blocks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h9205_20151007.patch: do not choose corrupted blocks.

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development