Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8647

Abstract BlockManager's rack policy into BlockPlacementPolicy

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Sometimes we want to have namenode use alternative block placement policy such as upgrade domains in HDFS-7541.

      BlockManager has built-in assumption about rack policy in functions such as useDelHint, blockHasEnoughRacks. That means when we have new block placement policy, we need to modify BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.

      1. HDFS-8647-branch27.patch
        40 kB
        Xiao Chen
      2. HDFS-8647-branch26.patch
        38 kB
        Xiao Chen
      3. HDFS-8647-009.patch
        50 kB
        Brahma Reddy Battula
      4. HDFS-8647-008.patch
        50 kB
        Brahma Reddy Battula
      5. HDFS-8647-007.patch
        50 kB
        Brahma Reddy Battula
      6. HDFS-8647-006.patch
        48 kB
        Brahma Reddy Battula
      7. HDFS-8647-005.patch
        46 kB
        Brahma Reddy Battula
      8. HDFS-8647-004.patch
        33 kB
        Brahma Reddy Battula
      9. HDFS-8647-004.patch
        37 kB
        Brahma Reddy Battula
      10. HDFS-8647-003.patch
        29 kB
        Brahma Reddy Battula
      11. HDFS-8647-002.patch
        17 kB
        Brahma Reddy Battula
      12. HDFS-8647-001.patch
        17 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for confirming that Vinayakumar B and Brahma Reddy Battula.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for confirming that Vinayakumar B and Brahma Reddy Battula .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yes, Tested on the latest code. auto replication happens to new rack. No behaviour change in 2.6/2.7.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yes, Tested on the latest code. auto replication happens to new rack. No behaviour change in 2.6/2.7.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Brahma Reddy Battula, does this mean we have a change of behavior in 2.6/2.7 after HDFS-8647 until HDFS-9127 is fixed? Thanks.

          I think, this was for the patch v3. I have tested on latest code, auto replication happens to new rack.
          So HDFS-9127, is not needed. can be closed.

          Show
          vinayrpet Vinayakumar B added a comment - Hi Brahma Reddy Battula, does this mean we have a change of behavior in 2.6/2.7 after HDFS-8647 until HDFS-9127 is fixed? Thanks. I think, this was for the patch v3. I have tested on latest code, auto replication happens to new rack. So HDFS-9127 , is not needed. can be closed.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          a. After change, auto replication (if single rack have already enough replicas == RF) will not happen when the cluster is added with one more rack. Only this will be triggered only if RF changes on those blocks.

          Hi Brahma Reddy Battula, does this mean we have a change of behavior in 2.6/2.7 after HDFS-8647 until HDFS-9127 is fixed? Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - a. After change, auto replication (if single rack have already enough replicas == RF) will not happen when the cluster is added with one more rack. Only this will be triggered only if RF changes on those blocks. Hi Brahma Reddy Battula , does this mean we have a change of behavior in 2.6/2.7 after HDFS-8647 until HDFS-9127 is fixed? Thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot for the review and commit Zhe!

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot for the review and commit Zhe!
          Hide
          zhz Zhe Zhang added a comment -

          +1 on branch-2.6 patch as well. I just committed it. Great work Xiao!

          Show
          zhz Zhe Zhang added a comment - +1 on branch-2.6 patch as well. I just committed it. Great work Xiao!
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the work Xiao! +1 on the branch-2.7 patch. I just committed it to branch-2.7.

          Reviewing 2.6 patch now.

          Show
          zhz Zhe Zhang added a comment - Thanks for the work Xiao! +1 on the branch-2.7 patch. I just committed it to branch-2.7. Reviewing 2.6 patch now.
          Hide
          xiaochen Xiao Chen added a comment -

          Locally applied both patches to corresponding branch and verified compile ok, touched tests pass.

          Show
          xiaochen Xiao Chen added a comment - Locally applied both patches to corresponding branch and verified compile ok, touched tests pass.
          Hide
          xiaochen Xiao Chen added a comment -

          Summary from branch-2.7 backport to branch-2.6 below:

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
            (HDFS-7411 which is in 2.7 but not 2.6. Resolved by copying over 2.7 but removing changes related to HDFS-7411. Also multiple branch2.6 fixes: 1. Changed a couple of initializing statements to fix the error 'diamond operator is not supported in -source 1.6'. 2. Updated log statement in BlockManager.java#2928 to not use placeholder syntax.
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
            (import conflicts.)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
            (import conflicts. Also fixed diamond operators to compile.)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            (The test TestBalancer#testBalancerWithPinnedBlocks from HDFS-6133 in 2.7 but not in 2.6. Resolved by not bringing in.)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
            1 line conflict in HDFS-9083’s new test
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
            (import conflicts. Also fixed diamond operators to compile.)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
            (import conflicts.)
          Show
          xiaochen Xiao Chen added a comment - Summary from branch-2.7 backport to branch-2.6 below: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java ( HDFS-7411 which is in 2.7 but not 2.6. Resolved by copying over 2.7 but removing changes related to HDFS-7411 . Also multiple branch2.6 fixes: 1. Changed a couple of initializing statements to fix the error 'diamond operator is not supported in -source 1.6'. 2. Updated log statement in BlockManager.java#2928 to not use placeholder syntax. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java (import conflicts.) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts. Also fixed diamond operators to compile.) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java (The test TestBalancer#testBalancerWithPinnedBlocks from HDFS-6133 in 2.7 but not in 2.6. Resolved by not bringing in.) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java 1 line conflict in HDFS-9083 ’s new test hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (import conflicts. Also fixed diamond operators to compile.) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java (import conflicts.)
          Hide
          xiaochen Xiao Chen added a comment -

          Sorry about the flooding comments here...
          Talked with Zhe offline, the cherry-pick was from trunk - sadly I didn't grep out from branch-2! (lesson learned: double check before backporting - comments above also showed Ming did the backport...) So please ignore the above comments regarding the cherry-picks. Summary from branch-2 backport to branch-2.7 below:
          Conflicts:

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
            (HDFS-8823 and HDFS-8938, both in branch-2 but not in branch2.7. Several BlockInfo related conflicts resolved by using Block, and resolved some conflicts regarding HDFS-8938's new methods by not bringing the methods into this patch. HDFS-9083 is only in branch 2.6 and 2.7, and removed shouldCheckForEnoughRacks related stuff in BlockManager.java, so no change needed here.)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
            (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
            (not exist)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
            (not exist)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
            (Import conflicts. Also additional changes due to HDFS-9083 which is only in branch 2.6 and 2.7.)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
            (not exist)
          Show
          xiaochen Xiao Chen added a comment - Sorry about the flooding comments here... Talked with Zhe offline, the cherry-pick was from trunk - sadly I didn't grep out from branch-2! (lesson learned: double check before backporting - comments above also showed Ming did the backport...) So please ignore the above comments regarding the cherry-picks. Summary from branch-2 backport to branch-2.7 below: Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java ( HDFS-8823 and HDFS-8938 , both in branch-2 but not in branch2.7. Several BlockInfo related conflicts resolved by using Block, and resolved some conflicts regarding HDFS-8938 's new methods by not bringing the methods into this patch. HDFS-9083 is only in branch 2.6 and 2.7, and removed shouldCheckForEnoughRacks related stuff in BlockManager.java, so no change needed here.) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java (not exist) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java (not exist) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (Import conflicts. Also additional changes due to HDFS-9083 which is only in branch 2.6 and 2.7.) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java (not exist)
          Hide
          xiaochen Xiao Chen added a comment -

          Patch for 2.6 didn't seem to need any specific change for HDFS-7086. But I had to fix 2 things for it to compile:

          • Changed a couple of initializing statements to fix the error 'diamond operator is not supported in -source 1.6'
          • Updated log statement in BlockManager.java#2928 to not use placeholder syntax
          Show
          xiaochen Xiao Chen added a comment - Patch for 2.6 didn't seem to need any specific change for HDFS-7086 . But I had to fix 2 things for it to compile: Changed a couple of initializing statements to fix the error 'diamond operator is not supported in -source 1.6' Updated log statement in BlockManager.java#2928 to not use placeholder syntax
          Hide
          xiaochen Xiao Chen added a comment -

          Attached patch for branch 2.6. This is cherry-picked from the 2.7 patch and modified upon.
          Conflicts:

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java (The test is only in branch2.7 by HDFS-6133, so not changed here)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (1 line conflict in the branch-only test brought in by HDFS-9083)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java (import conflicts)

          Both the 2.6 patch and 2.7 patch are compiled locally, and passed the changed unit tests. Please let me know what further testing/verification is needed for them to qualify. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Attached patch for branch 2.6. This is cherry-picked from the 2.7 patch and modified upon. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java (The test is only in branch2.7 by HDFS-6133 , so not changed here) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (1 line conflict in the branch-only test brought in by HDFS-9083 ) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java (import conflicts) Both the 2.6 patch and 2.7 patch are compiled locally, and passed the changed unit tests. Please let me know what further testing/verification is needed for them to qualify. Thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Attached patch for branch2.7. Will work on 2.6 later.
          Conflict list and reasoning below.
          Conflicts:

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (HDFS-9083 and HDFS-8543. HDFS-9083 is only in branch 2.6 and 2.7. Removed shouldCheckForEnoughRacks related stuff in BlockManager.java, so not needed here. Also there's a new test in TestBlockManager.java so need to update that. HDFS-8543 is EC related so only in trunk.)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java (Not exist)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java (Not exist)
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java (1 line diff, due to HDFS-8186 - EC related, only in trunk.)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (import conflicts)
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java (Not exist)
          Show
          xiaochen Xiao Chen added a comment - Attached patch for branch2.7. Will work on 2.6 later. Conflict list and reasoning below. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java ( HDFS-9083 and HDFS-8543 . HDFS-9083 is only in branch 2.6 and 2.7. Removed shouldCheckForEnoughRacks related stuff in BlockManager.java, so not needed here. Also there's a new test in TestBlockManager.java so need to update that. HDFS-8543 is EC related so only in trunk.) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java (Not exist) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java (Not exist) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java (1 line diff, due to HDFS-8186 - EC related, only in trunk.) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java (import conflicts) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java (Not exist)
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang and Ming Ma, I'll provide patches for 2.6/2.7 and post here for review soon.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang and Ming Ma , I'll provide patches for 2.6/2.7 and post here for review soon.
          Hide
          mingma Ming Ma added a comment -

          I agree we don't want to push HDFS-8823 to branch-2.6/2.7. A special patch for 2.7 should be relatively straightforward; it just need to get block replication factor in the old way. A special patch for 2.6 might require more changes as it StorageType has been moved from hadoop-hdfs to hadoop-common by HDFS-7806.

          Show
          mingma Ming Ma added a comment - I agree we don't want to push HDFS-8823 to branch-2.6/2.7. A special patch for 2.7 should be relatively straightforward; it just need to get block replication factor in the old way. A special patch for 2.6 might require more changes as it StorageType has been moved from hadoop-hdfs to hadoop-common by HDFS-7806 .
          Hide
          zhz Zhe Zhang added a comment -

          Brahma Reddy Battula Ming Ma Walter Su I wonder whether we should consider pushing this to branch-2.7 and branch-2.6. Maybe after the currently planned 2.7.2 and 2.6.3 releases.

          Doing so will enable the inclusion of bug fixes HDFS-9313 and HDFS-9314 in 2.6.x and 2.7.x.

          I worked with Xiao Chen offline along this direction. The main challenge is from HDFS-8823, which should have been done in a feature branch but was committed to branch-2 – so I don't think we should push that one to branch-2.6/2.7. If we reach an agreement here we can create a branch-2.6/2.7 patch for this JIRA.

          Thanks.

          Show
          zhz Zhe Zhang added a comment - Brahma Reddy Battula Ming Ma Walter Su I wonder whether we should consider pushing this to branch-2.7 and branch-2.6. Maybe after the currently planned 2.7.2 and 2.6.3 releases. Doing so will enable the inclusion of bug fixes HDFS-9313 and HDFS-9314 in 2.6.x and 2.7.x. I worked with Xiao Chen offline along this direction. The main challenge is from HDFS-8823 , which should have been done in a feature branch but was committed to branch-2 – so I don't think we should push that one to branch-2.6/2.7. If we reach an agreement here we can create a branch-2.6/2.7 patch for this JIRA. Thanks.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #522 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/522/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #522 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/522/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2459 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2459/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2459 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2459/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2510/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2510/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/563/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/563/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1298 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1298/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1298 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1298/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #577 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/577/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #577 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/577/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma thanks for prompted reviews and commit..and thanks to others for additional review and suggestions..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma thanks for prompted reviews and commit..and thanks to others for additional review and suggestions..
          Hide
          mingma Ming Ma added a comment -

          +1 on the latest patch. I have committed it to trunk and branch-2. Thanks Brahma Reddy Battula for the great contribution; Walter Su for the initial patch; Vinayakumar B and Andrew Wang for the suggestion and code review.

          Show
          mingma Ming Ma added a comment - +1 on the latest patch. I have committed it to trunk and branch-2. Thanks Brahma Reddy Battula for the great contribution; Walter Su for the initial patch; Vinayakumar B and Andrew Wang for the suggestion and code review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8678 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8678/)
          HDFS-8647. Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8678 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8678/ ) HDFS-8647 . Abstract BlockManager's rack policy into (mingma: rev e27c2ae8bafc94f18eb38f5d839dcef5652d424e) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          TestRecoverStripedFile failure and Pre-patch Findbugs warnings are unrelated.

          Show
          brahmareddy Brahma Reddy Battula added a comment - TestRecoverStripedFile failure and Pre-patch Findbugs warnings are unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 26m 32s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 10m 30s There were no new javac warning messages.
          +1 javadoc 10m 53s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 34s There were no new checkstyle issues.
          -1 whitespace 0m 5s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 44s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 7m 32s Tests passed in hadoop-common.
          -1 hdfs tests 68m 38s Tests failed in hadoop-hdfs.
              134m 2s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestRecoverStripedFile



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767622/HDFS-8647-009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0c4af0f
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13099/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13099/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 26m 32s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 10m 30s There were no new javac warning messages. +1 javadoc 10m 53s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 34s There were no new checkstyle issues. -1 whitespace 0m 5s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 44s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 7m 32s Tests passed in hadoop-common. -1 hdfs tests 68m 38s Tests failed in hadoop-hdfs.     134m 2s   Reason Tests Failed unit tests hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767622/HDFS-8647-009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0c4af0f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13099/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13099/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13099/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          oops, I missed this.. Uploaded the patch for same..

          Show
          brahmareddy Brahma Reddy Battula added a comment - oops, I missed this.. Uploaded the patch for same..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 36m 34s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 16m 34s There were no new javac warning messages.
          +1 javadoc 19m 36s There were no new javadoc warning messages.
          +1 release audit 0m 48s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 5m 8s There were no new checkstyle issues.
          -1 whitespace 0m 12s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 2m 34s mvn install still works.
          +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse.
          +1 findbugs 5m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 16m 56s Tests failed in hadoop-common.
          -1 hdfs tests 74m 12s Tests failed in hadoop-hdfs.
              179m 25s  



          Reason Tests
          Failed unit tests hadoop.fs.shell.find.TestName
            hadoop.fs.shell.find.TestFind
            hadoop.ha.TestZKFailoverController
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestPrint
            hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.TestSetrepDecreasing
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          Timed out tests org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767528/HDFS-8647-008.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7e2837f
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13070/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13070/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 36m 34s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 16m 34s There were no new javac warning messages. +1 javadoc 19m 36s There were no new javadoc warning messages. +1 release audit 0m 48s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 8s There were no new checkstyle issues. -1 whitespace 0m 12s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 2m 34s mvn install still works. +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse. +1 findbugs 5m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 16m 56s Tests failed in hadoop-common. -1 hdfs tests 74m 12s Tests failed in hadoop-hdfs.     179m 25s   Reason Tests Failed unit tests hadoop.fs.shell.find.TestName   hadoop.fs.shell.find.TestFind   hadoop.ha.TestZKFailoverController   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestPrint   hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy Timed out tests org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767528/HDFS-8647-008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7e2837f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13070/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13070/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13070/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula. It seems you also need to change the while condition used in the function from candidates.size() - expectedNumOfReplicas > 0 to candidates.size() - expectedNumOfReplicas > excessReplicas.size().

          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula . It seems you also need to change the while condition used in the function from candidates.size() - expectedNumOfReplicas > 0 to candidates.size() - expectedNumOfReplicas > excessReplicas.size() .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma Agree with you,Even it will not cause any issue...Removed candidates.remove(cur); from BlockPlacementPolicyDefault#chooseReplicasToDelete and uploaded the patch again..And thanks for spending time on cheery-pick to branch-2..it should work.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma Agree with you,Even it will not cause any issue...Removed candidates.remove(cur); from BlockPlacementPolicyDefault#chooseReplicasToDelete and uploaded the patch again..And thanks for spending time on cheery-pick to branch-2..it should work.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula! It looks great. Sorry, there is one question I didn't mentioned last time.

          It seems the replica-to-be-deleted will be removed from the candidates in BlockPlacementPolicyDefault#chooseReplicasToDelete: candidates.remove(cur);. It will be removed again later in BlockManager#processChosenExcessReplica: nonExcess.remove(chosen);. Even though this doesn't cause any issue; it is better to keep the semantics of BlockPlacementPolicyDefault#chooseReplicasToDelete to not modify the candidates. What do you think? If you agree, after you update the patch and the patch passes jenkins, it is good to go.

          In addition, I have used the local repo to evaluate how hard it is to cherry-pick to branch-2. It turns out I had to spend some time manually resolving the issues. I have taken care of that and you don't need to provide a version for branch-2. The patch passed HDFS unit tests in branch-2.

          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula ! It looks great. Sorry, there is one question I didn't mentioned last time. It seems the replica-to-be-deleted will be removed from the candidates in BlockPlacementPolicyDefault#chooseReplicasToDelete : candidates.remove(cur); . It will be removed again later in BlockManager#processChosenExcessReplica : nonExcess.remove(chosen); . Even though this doesn't cause any issue; it is better to keep the semantics of BlockPlacementPolicyDefault#chooseReplicasToDelete to not modify the candidates. What do you think? If you agree, after you update the patch and the patch passes jenkins, it is good to go. In addition, I have used the local repo to evaluate how hard it is to cherry-pick to branch-2. It turns out I had to spend some time manually resolving the issues. I have taken care of that and you don't need to provide a version for branch-2. The patch passed HDFS unit tests in branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 22s There were no new javadoc warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 10s There were no new checkstyle issues.
          -1 whitespace 0m 5s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 10m 5s Tests failed in hadoop-common.
          -1 hdfs tests 50m 44s Tests failed in hadoop-hdfs.
              108m 48s  



          Reason Tests
          Failed unit tests hadoop.fs.TestLocalFsFCStatistics
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.blockmanagement.TestBlockManager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767395/HDFS-8647-007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6144e01
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13053/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13053/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 13s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 10s There were no new checkstyle issues. -1 whitespace 0m 5s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 10m 5s Tests failed in hadoop-common. -1 hdfs tests 50m 44s Tests failed in hadoop-hdfs.     108m 48s   Reason Tests Failed unit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767395/HDFS-8647-007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6144e01 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13053/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13053/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma

          BlockManager should swap addedNode and delNodeHint.,

          yes you are correct..

          After the fix, the only failed unit test left is TestBalancer#testBalancerWithPinnedBlocks. Can you please investigate if it is related to the patch?

          .

          cluster = new MiniDFSCluster.Builder(conf).numDataNodes(capacities.length)
            .hosts(new String[]{"localhost", "localhost"})
            .racks(new String[]{"rack0", "rack1"}).simulatedCapacities(capacities).build()

          2 DNs are started with "rack1". Ideally we should not create 2 DNs with the same hostname.And Pinning depends on favoredNodes.DFSClient#create(..) only uses host:port, if favoredNodes is created by new InetSocketAddress(ip, port)

          DFSClient will attempt a reverse lookup locally to get host:port, instead of sending ip:port directly to NameNode.
          .
          MiniDFSCluster use fake hostname "host1.foo.com" to start DataNodes.DFSClient doesn't use StaticMapping. So if DFSClient do reverse lookup, "127.0.0.1:8020" becomes "localhost:8020".

          public can be removed from public DatanodeStorageInfo chooseReplicaToDelete.

          it's overidden in test TestDnfencing#chooseReplicaToDelete .hence I make this method visible for test.

          Remaning changes are done and uploaded the patch....Hope I address all of your comments..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma BlockManager should swap addedNode and delNodeHint., yes you are correct.. After the fix, the only failed unit test left is TestBalancer#testBalancerWithPinnedBlocks . Can you please investigate if it is related to the patch? . cluster = new MiniDFSCluster.Builder(conf).numDataNodes(capacities.length) .hosts( new String []{ "localhost" , "localhost" }) .racks( new String []{ "rack0" , "rack1" }).simulatedCapacities(capacities).build() 2 DNs are started with "rack1". Ideally we should not create 2 DNs with the same hostname.And Pinning depends on favoredNodes.DFSClient#create(..) only uses host:port, if favoredNodes is created by new InetSocketAddress(ip, port) DFSClient will attempt a reverse lookup locally to get host:port, instead of sending ip:port directly to NameNode. . MiniDFSCluster use fake hostname "host1.foo.com" to start DataNodes.DFSClient doesn't use StaticMapping. So if DFSClient do reverse lookup, "127.0.0.1:8020" becomes "localhost:8020". public can be removed from public DatanodeStorageInfo chooseReplicaToDelete . it's overidden in test TestDnfencing#chooseReplicaToDelete .hence I make this method visible for test. Remaning changes are done and uploaded the patch....Hope I address all of your comments..
          Hide
          mingma Ming Ma added a comment -

          Most of the test failures are due to mismatch between BlockManager and function definition of chooseReplicasToDelete; BlockManager should swap addedNode and delNodeHint.

          After the fix, the only failed unit test left is TestBalancer#testBalancerWithPinnedBlocks. Can you please investigate if it is related to the patch?

          Nits:

          • public can be removed from public DatanodeStorageInfo chooseReplicaToDelete.
          • final List<DatanodeStorageInfo> moreThanOne = new ArrayList<DatanodeStorageInfo>(); Given hadoop uses JDK7, you can simplify this with type inference final List<DatanodeStorageInfo> moreThanOne = new ArrayList<>(); Same for other places in the patch.
          • chooseReplicaToDelete’s parameter Collection<DatanodeStorageInfo> first. How about name it moreThanOne instead? Similarly for second.
          Show
          mingma Ming Ma added a comment - Most of the test failures are due to mismatch between BlockManager and function definition of chooseReplicasToDelete ; BlockManager should swap addedNode and delNodeHint . After the fix, the only failed unit test left is TestBalancer#testBalancerWithPinnedBlocks . Can you please investigate if it is related to the patch? Nits: public can be removed from public DatanodeStorageInfo chooseReplicaToDelete . final List<DatanodeStorageInfo> moreThanOne = new ArrayList<DatanodeStorageInfo>(); Given hadoop uses JDK7, you can simplify this with type inference final List<DatanodeStorageInfo> moreThanOne = new ArrayList<>(); Same for other places in the patch. chooseReplicaToDelete ’s parameter Collection<DatanodeStorageInfo> first . How about name it moreThanOne instead? Similarly for second .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          TestBalancer fails on specific machines where hostname resolution have some difference..Not related this patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - TestBalancer fails on specific machines where hostname resolution have some difference..Not related this patch..
          Hide
          mingma Ming Ma added a comment -

          Brahma Reddy Battula, can you please check again if TestBalancer and maybe other test failures are related? Somehow these tests timed out with the patch. Thanks!

          Show
          mingma Ming Ma added a comment - Brahma Reddy Battula , can you please check again if TestBalancer and maybe other test failures are related? Somehow these tests timed out with the patch. Thanks!
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Test failures are unrelated,Ran all the impacted testcases all those are Passing.. Kindly review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Test failures are unrelated,Ran all the impacted testcases all those are Passing.. Kindly review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 14s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 26s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 4s There were no new checkstyle issues.
          -1 whitespace 0m 5s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 6m 42s Tests failed in hadoop-common.
          -1 hdfs tests 74m 39s Tests failed in hadoop-hdfs.
              131m 18s  



          Reason Tests
          Failed unit tests hadoop.net.TestDNS
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestInjectionForSimulatedStorage
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.TestLargeBlock
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          Timed out tests org.apache.hadoop.hdfs.TestReplication
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766598/HDFS-8647-006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0d77e85
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12983/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12983/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 14s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 4s There were no new checkstyle issues. -1 whitespace 0m 5s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 6m 42s Tests failed in hadoop-common. -1 hdfs tests 74m 39s Tests failed in hadoop-hdfs.     131m 18s   Reason Tests Failed unit tests hadoop.net.TestDNS   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestLargeBlock   hadoop.fs.TestGlobPaths   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Timed out tests org.apache.hadoop.hdfs.TestReplication   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766598/HDFS-8647-006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0d77e85 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12983/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12983/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12983/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 1s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 8m 2s There were no new javac warning messages.
          +1 javadoc 10m 26s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 11s There were no new checkstyle issues.
          -1 whitespace 0m 5s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 4m 8s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken.
          +1 findbugs 4m 8s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 0m 23s Tests failed in hadoop-common.
          -1 hdfs tests 0m 23s Tests failed in hadoop-hdfs.
              48m 18s  



          Reason Tests
          Failed build hadoop-common
            hadoop-hdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766598/HDFS-8647-006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 56dc777
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12985/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12985/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 1s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 11s There were no new checkstyle issues. -1 whitespace 0m 5s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 4m 8s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken. +1 findbugs 4m 8s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 0m 23s Tests failed in hadoop-common. -1 hdfs tests 0m 23s Tests failed in hadoop-hdfs.     48m 18s   Reason Tests Failed build hadoop-common   hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766598/HDFS-8647-006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 56dc777 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12985/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12985/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12985/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma and Vinayakumar B

          Uploaded patch to address following comments....

          The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks,

          It might be easier to not to pass isStriped to verifyBlockPlacement. Instead, have BlockPlacementPolicyRackFaultTolerant implement verifyBlockPlacement which will use numberOfReplicas as the minRacks. This also makes the patch more applicable to branch-2 which doesn't have striped EC

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma and Vinayakumar B Uploaded patch to address following comments.... The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks, It might be easier to not to pass isStriped to verifyBlockPlacement. Instead, have BlockPlacementPolicyRackFaultTolerant implement verifyBlockPlacement which will use numberOfReplicas as the minRacks. This also makes the patch more applicable to branch-2 which doesn't have striped EC
          Hide
          vinayrpet Vinayakumar B added a comment -

          I was trying to point out that cherry-pick likely won't work. It is just to make sure after I commit it to trunk, there is patch for reference in case I need to manually resolve it as part of cherry-pick.

          yes, correct.
          What i exactly meant is, currently there will be lot of difference, since EC is not merged to branch-2. after that it would be easy to merge. Anyway I am okay if its okay to go with different patch for branch-2.

          Show
          vinayrpet Vinayakumar B added a comment - I was trying to point out that cherry-pick likely won't work. It is just to make sure after I commit it to trunk, there is patch for reference in case I need to manually resolve it as part of cherry-pick. yes, correct. What i exactly meant is, currently there will be lot of difference, since EC is not merged to branch-2. after that it would be easy to merge. Anyway I am okay if its okay to go with different patch for branch-2.
          Hide
          mingma Ming Ma added a comment -

          Thanks Vinayakumar B!

          how about waiting for commit to branch-2. After that simple cherry-pick might work.

          I was trying to point out that cherry-pick likely won't work. It is just to make sure after I commit it to trunk, there is patch for reference in case I need to manually resolve it as part of cherry-pick.

          "# of racks >= # of data blocks" requirement is to ensure rackwise failure doesn't create any dataloss in case of EC'ed file.

          Is that to ensure no data loss or more for read reconstruction time optimization? Use the example of RS(6,3) and 5 racks, the blocks will be written successfully. If rack_0 fails, read needs to reconstruct data_0 and data_5, but there is no data loss. This is just for my own education, to clarify why the write policy is different from verification policy for EC scenario. Again, this is not related this jira.

          rack_0 rack_1 rack_2 rack_3 rack_4
          data_0 data_1 data_2 data_3 data_4
          data_5 prty_0 prty_1 prty_2
          Show
          mingma Ming Ma added a comment - Thanks Vinayakumar B ! how about waiting for commit to branch-2. After that simple cherry-pick might work. I was trying to point out that cherry-pick likely won't work. It is just to make sure after I commit it to trunk, there is patch for reference in case I need to manually resolve it as part of cherry-pick. "# of racks >= # of data blocks" requirement is to ensure rackwise failure doesn't create any dataloss in case of EC'ed file. Is that to ensure no data loss or more for read reconstruction time optimization? Use the example of RS(6,3) and 5 racks, the blocks will be written successfully. If rack_0 fails, read needs to reconstruct data_0 and data_5, but there is no data loss. This is just for my own education, to clarify why the write policy is different from verification policy for EC scenario. Again, this is not related this jira. rack_0 rack_1 rack_2 rack_3 rack_4 data_0 data_1 data_2 data_3 data_4 data_5 prty_0 prty_1 prty_2
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Brahma Reddy Battula for patch.
          Thanks Ming Ma for review,
          Those are good points.

          The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks,

          Yes, you are right. It should use getRealDataBlockNum as minracks. I found this is discussed in HDFS-7613

          The current patch doesn't apply to branch-2. If you agree with the above changes, could you try out if it applies to branch-2? If it doesn't apply, you will need to provide a separate patch for branch-2 later.

          Yes, of-course it will not apply. Instead of providing another branch-2 patch, which might create more conflicts when merging EC to branch-2, how about waiting for commit to branch-2. After that simple cherry-pick might work.

          A general question about striped EC. It uses "# of racks >= # of data blocks" to check if a given block has enough racks. But what if "# of racks for the whole cluster < # of data blocks"? Say we use RS(6,3) and the cluster has 5 racks. The write operation will spread the 9 blocks to 5 racks and succeed. But it will fail the "enough racks" check later in BM? But that has nothing to with the refactoring work here. I just want to bring it up in case others can chime in.

          You are right. Check will fail, that means block will not be removed from neededReplications map.
          "# of racks >= # of data blocks" requirement is to ensure rackwise failure doesn't create any dataloss in case of EC'ed file.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Brahma Reddy Battula for patch. Thanks Ming Ma for review, Those are good points. The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks, Yes, you are right. It should use getRealDataBlockNum as minracks. I found this is discussed in HDFS-7613 The current patch doesn't apply to branch-2. If you agree with the above changes, could you try out if it applies to branch-2? If it doesn't apply, you will need to provide a separate patch for branch-2 later. Yes, of-course it will not apply. Instead of providing another branch-2 patch, which might create more conflicts when merging EC to branch-2, how about waiting for commit to branch-2. After that simple cherry-pick might work. A general question about striped EC. It uses "# of racks >= # of data blocks" to check if a given block has enough racks. But what if "# of racks for the whole cluster < # of data blocks"? Say we use RS(6,3) and the cluster has 5 racks. The write operation will spread the 9 blocks to 5 racks and succeed. But it will fail the "enough racks" check later in BM? But that has nothing to with the refactoring work here. I just want to bring it up in case others can chime in. You are right. Check will fail, that means block will not be removed from neededReplications map. "# of racks >= # of data blocks" requirement is to ensure rackwise failure doesn't create any dataloss in case of EC'ed file.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula!

          Most of the change looks good. It seems this will also fix HDFS-9083. cc: Rushabh S Shah.

          There are some questions about the striped erasure coding block placement abstraction.

          • The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks,
          • It might be easier to not to pass isStriped to verifyBlockPlacement. Instead, have BlockPlacementPolicyRackFaultTolerant implement verifyBlockPlacement which will use numberOfReplicas as the minRacks. This also makes the patch more applicable to branch-2 which doesn't have striped EC.
          • The current patch doesn't apply to branch-2. If you agree with the above changes, could you try out if it applies to branch-2? If it doesn't apply, you will need to provide a separate patch for branch-2 later.
          • A general question about striped EC. It uses "# of racks >= # of data blocks" to check if a given block has enough racks. But what if "# of racks for the whole cluster < # of data blocks"? Say we use RS(6,3) and the cluster has 5 racks. The write operation will spread the 9 blocks to 5 racks and succeed. But it will fail the "enough racks" check later in BM? But that has nothing to with the refactoring work here. I just want to bring it up in case others can chime in.
          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula ! Most of the change looks good. It seems this will also fix HDFS-9083 . cc: Rushabh S Shah . There are some questions about the striped erasure coding block placement abstraction. The existing blockHasEnoughRacksStriped compares getRealDataBlockNum (# of data blocks) with the number racks. But after the refactoring, it compares getRealTotalBlockNum (# of total blocks) with the number racks, It might be easier to not to pass isStriped to verifyBlockPlacement . Instead, have BlockPlacementPolicyRackFaultTolerant implement verifyBlockPlacement which will use numberOfReplicas as the minRacks . This also makes the patch more applicable to branch-2 which doesn't have striped EC. The current patch doesn't apply to branch-2. If you agree with the above changes, could you try out if it applies to branch-2? If it doesn't apply, you will need to provide a separate patch for branch-2 later. A general question about striped EC. It uses "# of racks >= # of data blocks" to check if a given block has enough racks. But what if "# of racks for the whole cluster < # of data blocks"? Say we use RS(6,3) and the cluster has 5 racks. The write operation will spread the 9 blocks to 5 racks and succeed. But it will fail the "enough racks" check later in BM? But that has nothing to with the refactoring work here. I just want to bring it up in case others can chime in.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma rebased patch kindly review..Thanks..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma rebased patch kindly review..Thanks..
          Hide
          mingma Ming Ma added a comment -

          Brahma Reddy Battula, regarding the verifyBlockPlacement part, maybe it is ok to replace LocatedBlock with DatanodeInfo[] as in your earlier patch. If we need to add additional information for new scenarios, we can update the API again. In addition, such change might allow balancer to ask block placement policy if any move might violate the policy, as in HDFS-9007.

          In addition, many changes around block management and block placement policy have gone. So I understand it requires some effort to rebase your patch.

          Thanks for your work. It is close.

          Show
          mingma Ming Ma added a comment - Brahma Reddy Battula , regarding the verifyBlockPlacement part, maybe it is ok to replace LocatedBlock with DatanodeInfo[] as in your earlier patch. If we need to add additional information for new scenarios, we can update the API again. In addition, such change might allow balancer to ask block placement policy if any move might violate the policy, as in HDFS-9007 . In addition, many changes around block management and block placement policy have gone. So I understand it requires some effort to rebase your patch. Thanks for your work. It is close.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula! Here are couple comments.

          • NetworkTopology knows when the # of racks changes from 1 to 2. So instead of having DatanodeManager call setHasClusterEverBeenMultiRack on NetworkTopology, perhaps we can have a new method boolean NetworkTopology.add(Node node) where it return true only when the # of racks changes from 1 to 2. Then DatanodeManager can act only if the returned value is true.
          • verifyBlockPlacement's parameter changes from LocatedBlock to DatanodeInfo[]. LocatedBlock has other info such as storage type that DatanodeInfo doesn't have. If later we decide to use storage type to verify block placement, we will need to add it back. Any way BlockManager can construct LocatedBlock from the block id? For example, maybe it can get the DatanodeStorageInfo from blocksMap.getStorages(block) and then construct the block via newLocatedBlock method.
          • In verifyBlockPlacement's if (!clusterMap.hasClusterEverBeenMultiRack() && numRacks <= 1), is numRacks <= 1 still needed?
          • chooseReplicaToDelete becomes an internal method. So there is no need to make it public.
          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula ! Here are couple comments. NetworkTopology knows when the # of racks changes from 1 to 2. So instead of having DatanodeManager call setHasClusterEverBeenMultiRack on NetworkTopology, perhaps we can have a new method boolean NetworkTopology.add(Node node) where it return true only when the # of racks changes from 1 to 2. Then DatanodeManager can act only if the returned value is true. verifyBlockPlacement 's parameter changes from LocatedBlock to DatanodeInfo[] . LocatedBlock has other info such as storage type that DatanodeInfo doesn't have. If later we decide to use storage type to verify block placement, we will need to add it back. Any way BlockManager can construct LocatedBlock from the block id? For example, maybe it can get the DatanodeStorageInfo from blocksMap.getStorages(block) and then construct the block via newLocatedBlock method. In verifyBlockPlacement 's if (!clusterMap.hasClusterEverBeenMultiRack() && numRacks <= 1) , is numRacks <= 1 still needed? chooseReplicaToDelete becomes an internal method. So there is no need to make it public.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma kindly review the latest patch..dn't know why jenkins are not triggered on latest patch.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma kindly review the latest patch..dn't know why jenkins are not triggered on latest patch.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Maybe we can move hasClusterEverBeenMultiRack from DatanodeManager to NetworkTopology? Then BlockPlacementPolicyDefault's verifyBlockPlacement can ask clusterMap if the cluster has ever been multi rack. In that way, we completely remove the multi rack reference from BlockManager.

          Agree with you.

          Uploaded the patch to address all the comments..Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Maybe we can move hasClusterEverBeenMultiRack from DatanodeManager to NetworkTopology? Then BlockPlacementPolicyDefault's verifyBlockPlacement can ask clusterMap if the cluster has ever been multi rack. In that way, we completely remove the multi rack reference from BlockManager. Agree with you. Uploaded the patch to address all the comments..Kindly review..
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula. Maybe we can move hasClusterEverBeenMultiRack from DatanodeManager to NetworkTopology? Then BlockPlacementPolicyDefault's verifyBlockPlacement can ask clusterMap if the cluster has ever been multi rack. In that way, we completely remove the multi rack reference from BlockManager.

          Regardless the approach, there is a behavior change for BlockPlacementPolicyDefault's verifyBlockPlacement, which is used by fsck. When # of racks is reduced to 1, fsck used to return ok; but with the change, it will indicate it violates the rack policy. That should be ok.

          Nits: could you please clean up the whitespace? Also the descriptions you added chooseReplicaToDelete don't match the parameter names.

          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula . Maybe we can move hasClusterEverBeenMultiRack from DatanodeManager to NetworkTopology? Then BlockPlacementPolicyDefault 's verifyBlockPlacement can ask clusterMap if the cluster has ever been multi rack. In that way, we completely remove the multi rack reference from BlockManager. Regardless the approach, there is a behavior change for BlockPlacementPolicyDefault 's verifyBlockPlacement , which is used by fsck. When # of racks is reduced to 1, fsck used to return ok; but with the change, it will indicate it violates the rack policy. That should be ok. Nits: could you please clean up the whitespace? Also the descriptions you added chooseReplicaToDelete don't match the parameter names.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 332, now 324).
          -1 whitespace 0m 2s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 18s Pre-build of native portion
          -1 hdfs tests 168m 28s Tests failed in hadoop-hdfs.
              214m 29s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.fs.viewfs.TestViewFileSystemHdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762402/HDFS-8647-004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e65c5
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12680/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12680/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 26s The applied patch generated 2 new checkstyle issues (total was 332, now 324). -1 whitespace 0m 2s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 18s Pre-build of native portion -1 hdfs tests 168m 28s Tests failed in hadoop-hdfs.     214m 29s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.fs.viewfs.TestViewFileSystemHdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762402/HDFS-8647-004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12680/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12680/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12680/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma Uploaded the patch..Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma Uploaded the patch..Kindly review..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ideally HDFS should support auto replication for the case where # of rack changes from 1 to 2 after NN's restart. But that requires more work and it is an existing issue. You can open a new jira for that.

          Ming Ma Raised seperate jira to handle this..(HDFS-9127)..will update patch by keeping current behavior as it is.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ideally HDFS should support auto replication for the case where # of rack changes from 1 to 2 after NN's restart. But that requires more work and it is an existing issue. You can open a new jira for that. Ming Ma Raised seperate jira to handle this..( HDFS-9127 )..will update patch by keeping current behavior as it is.
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula for the useful analysis! My preference is to keep the current behavior so that auto replication continue to work as long as there is no NN restart. That provides stronger data durability.

          Ideally HDFS should support auto replication for the case where # of rack changes from 1 to 2 after NN's restart. But that requires more work and it is an existing issue. You can open a new jira for that.

          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula for the useful analysis! My preference is to keep the current behavior so that auto replication continue to work as long as there is no NN restart. That provides stronger data durability. Ideally HDFS should support auto replication for the case where # of rack changes from 1 to 2 after NN's restart. But that requires more work and it is an existing issue. You can open a new jira for that.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma any thoughts on this issue..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma any thoughts on this issue..?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          TestBlocksWithNotEnoughRacks are failing due to following cases.

          Before change

          if cluster was multi rack first and then later became single rack, then those blocks were getting added in NeededReplications, and the same value was expected in tests.
          a. But this was only till the namenode was alive. If NN was restarted after making into single rack, this NeededReplications will not have the block.
          b. And before NN restart, another rack was added, then auto replication happens to new rack. But Once NN restarted, and new rack added, then auto replication to new rack (if single rack have already enough replicas == RF) to new rack happens only if RF changes on those blocks.

          After change

          it will not add to NeededReplications immediately cluster became the single rack.
          a. After change, auto replication (if single rack have already enough replicas == RF) will not happen when the cluster is added with one more rack. Only this will be triggered only if RF changes on those blocks.

          If new change is Okay, then test case can be updated, else patch can be updated.

          Show
          brahmareddy Brahma Reddy Battula added a comment - TestBlocksWithNotEnoughRacks are failing due to following cases. Before change if cluster was multi rack first and then later became single rack, then those blocks were getting added in NeededReplications , and the same value was expected in tests. a. But this was only till the namenode was alive. If NN was restarted after making into single rack, this NeededReplications will not have the block. b. And before NN restart, another rack was added, then auto replication happens to new rack. But Once NN restarted, and new rack added, then auto replication to new rack (if single rack have already enough replicas == RF) to new rack happens only if RF changes on those blocks. After change it will not add to NeededReplications immediately cluster became the single rack. a. After change, auto replication (if single rack have already enough replicas == RF) will not happen when the cluster is added with one more rack. Only this will be triggered only if RF changes on those blocks. If new change is Okay, then test case can be updated, else patch can be updated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Upgrading from 2.7 to 2.8 (assume we put this change in 2.8) will break that unless they recompile their custom class. That is because BlockManager will start calling chooseReplicasToDelete method which isn't implemented in their class.

          AFAIK most customized policy extends BlockPlacementPolicyDefault. It's rare to implement a policy from scratch. Not only method signature, as long as we change default policy one byte, the customized policy need recompile anyway. Maven can solve the dependency and recompile if needed.
          The old chooseReplicasToDelete is limited. I suppose little user override it. After the change, it's more flexible to let user customize it. If user didn't override it before, then the customized policy class can cleanly recompile without complain.

          Note that this seems similar to BlockPlacementPolicy's verifyBlockPlacement. So if they can be combined, that is nice.

          That's a good idea.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Upgrading from 2.7 to 2.8 (assume we put this change in 2.8) will break that unless they recompile their custom class. That is because BlockManager will start calling chooseReplicasToDelete method which isn't implemented in their class. AFAIK most customized policy extends BlockPlacementPolicyDefault. It's rare to implement a policy from scratch. Not only method signature, as long as we change default policy one byte, the customized policy need recompile anyway. Maven can solve the dependency and recompile if needed. The old chooseReplicasToDelete is limited. I suppose little user override it. After the change, it's more flexible to let user customize it. If user didn't override it before, then the customized policy class can cleanly recompile without complain. Note that this seems similar to BlockPlacementPolicy's verifyBlockPlacement. So if they can be combined, that is nice. That's a good idea.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          TestBlocksWithNotEnoughRacks and TestBalancer are related this change..Will fix this and checkstyle comments and upload the patch.

          Show
          brahmareddy Brahma Reddy Battula added a comment - TestBlocksWithNotEnoughRacks and TestBalancer are related this change..Will fix this and checkstyle comments and upload the patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 10m 5s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 23s The applied patch generated 4 new checkstyle issues (total was 332, now 328).
          -1 whitespace 0m 2s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 164m 39s Tests failed in hadoop-hdfs.
              209m 52s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756004/HDFS-8647-003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 73e3a49
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12452/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12452/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 10m 5s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 23s The applied patch generated 4 new checkstyle issues (total was 332, now 328). -1 whitespace 0m 2s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 164m 39s Tests failed in hadoop-hdfs.     209m 52s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756004/HDFS-8647-003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 73e3a49 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12452/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12452/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12452/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ming Ma updated the patch, based on your comments.. Kindly Review the same..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ming Ma updated the patch, based on your comments.. Kindly Review the same..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks a lot Ming Ma for your review comments.. I will work on these comments and post the patch...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks a lot Ming Ma for your review comments.. I will work on these comments and post the patch...
          Hide
          mingma Ming Ma added a comment -

          Thanks Brahma Reddy Battula! The patch covers the basic idea of the rack policy refactoring.

          • Backward compatibility for BlockPlacementPolicy. There is a hypothetic scenario where people have developed some custom BlockPlacementPolicy class packaged in a separate jar from the hdfs jar. Upgrading from 2.7 to 2.8 (assume we put this change in 2.8) will break that unless they recompile their custom class. That is because BlockManager will start calling chooseReplicasToDelete method which isn't implemented in their class. Perhaps that hasn't been the supported scenario. For example, HDFS-6584 modifies chooseReplicaToDelete directly. If that is the case, you might as well remove the old method chooseReplicaToDelete from BlockPlacementPolicy.
          • BlockManager still has its own blockHasEnoughRacks function that implies rack based policy. Given we plan to add new block placement policy that isn't rack related, isNeededReplication is essentially checking if the existing replicas violate the policy. Perhaps we can rename that function to be something like verifyReplicasPlacement. Same for the new BlockPlacementPolicy's blockHasEnoughRacks. Note that this seems similar to BlockPlacementPolicy's verifyBlockPlacement. So if they can be combined, that is nice.
          • With the code refactored into BlockPlacementPolicy, it also allows us to write unit tests that might be tricky before. So it will be good if you can add unit tests for new methods added to BlockPlacementPolicy.
          Show
          mingma Ming Ma added a comment - Thanks Brahma Reddy Battula ! The patch covers the basic idea of the rack policy refactoring. Backward compatibility for BlockPlacementPolicy . There is a hypothetic scenario where people have developed some custom BlockPlacementPolicy class packaged in a separate jar from the hdfs jar. Upgrading from 2.7 to 2.8 (assume we put this change in 2.8) will break that unless they recompile their custom class. That is because BlockManager will start calling chooseReplicasToDelete method which isn't implemented in their class. Perhaps that hasn't been the supported scenario. For example, HDFS-6584 modifies chooseReplicaToDelete directly. If that is the case, you might as well remove the old method chooseReplicaToDelete from BlockPlacementPolicy . BlockManager still has its own blockHasEnoughRacks function that implies rack based policy. Given we plan to add new block placement policy that isn't rack related, isNeededReplication is essentially checking if the existing replicas violate the policy. Perhaps we can rename that function to be something like verifyReplicasPlacement . Same for the new BlockPlacementPolicy 's blockHasEnoughRacks . Note that this seems similar to BlockPlacementPolicy 's verifyBlockPlacement . So if they can be combined, that is nice. With the code refactored into BlockPlacementPolicy , it also allows us to write unit tests that might be tricky before. So it will be good if you can add unit tests for new methods added to BlockPlacementPolicy .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 36s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 9m 49s There were no new javac warning messages.
          +1 javadoc 12m 31s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 37s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 2m 9s mvn install still works.
          +1 eclipse:eclipse 0m 55s The patch built with eclipse:eclipse.
          -1 findbugs 3m 6s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 19s Pre-build of native portion
          +1 hdfs tests 186m 54s Tests passed in hadoop-hdfs.
              238m 25s  



          Reason Tests
          FindBugs module:hadoop-hdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754526/HDFS-8647-002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1dbd8e3
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12332/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12332/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 36s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 9m 49s There were no new javac warning messages. +1 javadoc 12m 31s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 37s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 2m 9s mvn install still works. +1 eclipse:eclipse 0m 55s The patch built with eclipse:eclipse. -1 findbugs 3m 6s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 19s Pre-build of native portion +1 hdfs tests 186m 54s Tests passed in hadoop-hdfs.     238m 25s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754526/HDFS-8647-002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1dbd8e3 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12332/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12332/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12332/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch to address test-case failures and check-style issues.Kindly Review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to address test-case failures and check-style issues.Kindly Review..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          I am looking into test-failures..

          Show
          brahmareddy Brahma Reddy Battula added a comment - I am looking into test-failures..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 8 new checkstyle issues (total was 216, now 217).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 2m 30s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 186m 34s Tests failed in hadoop-hdfs.
              231m 7s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754426/HDFS-8647-001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9b68577
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12324/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12324/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 8 new checkstyle issues (total was 216, now 217). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 2m 30s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 186m 34s Tests failed in hadoop-hdfs.     231m 7s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754426/HDFS-8647-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9b68577 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12324/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12324/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12324/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Attached the initial patch for review.. Kindly Review..

          Did the following

          1) Moved useDelHint
          2) replaced shouldCheckForEnoughRacks with hasClusterEverBeenMultiRack
          3) Moved excessReplicas logic(from racks)

          Show
          brahmareddy Brahma Reddy Battula added a comment - Attached the initial patch for review.. Kindly Review.. Did the following 1) Moved useDelHint 2) replaced shouldCheckForEnoughRacks with hasClusterEverBeenMultiRack 3) Moved excessReplicas logic(from racks)
          Hide
          walter.k.su Walter Su added a comment -

          Of course! Assigned to you. Thanks Brahma Reddy Battula for taking care of this.

          Show
          walter.k.su Walter Su added a comment - Of course! Assigned to you. Thanks Brahma Reddy Battula for taking care of this.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hi, Walter Su!
          I'm interested in this. I saw you don't update for a long while. Can you assign to me if you are busy?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hi, Walter Su ! I'm interested in this. I saw you don't update for a long while. Can you assign to me if you are busy?
          Hide
          walter.k.su Walter Su added a comment -

          thanks for filing this Ming Ma.

          I'm afraid the whole patch is huge. So I break the task to pieces. Since BlockManager is critical code and we should be careful. I prefer do it step by step. It also make review easier. Hope you're ok with that.

          Show
          walter.k.su Walter Su added a comment - thanks for filing this Ming Ma . I'm afraid the whole patch is huge. So I break the task to pieces. Since BlockManager is critical code and we should be careful. I prefer do it step by step. It also make review easier. Hope you're ok with that.
          Hide
          andrew.wang Andrew Wang added a comment -

          Great idea, thanks for filing this Ming Ma.

          Related to this, besides incorporating rack-knowledge from the BPP, we could also consider attributes like storage type when determining if a block is misreplicated. This would allow the NN to take over the Mover's responsibilities and do it online rather than offline, and also edge triggered rather than via periodic namespace scans.

          This is not something we need to pursue in this JIRA, but just an idea for the future.

          Show
          andrew.wang Andrew Wang added a comment - Great idea, thanks for filing this Ming Ma . Related to this, besides incorporating rack-knowledge from the BPP, we could also consider attributes like storage type when determining if a block is misreplicated. This would allow the NN to take over the Mover's responsibilities and do it online rather than offline, and also edge triggered rather than via periodic namespace scans. This is not something we need to pursue in this JIRA, but just an idea for the future.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development