Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1480

All replicas of a block can end up on the same rack when some datanodes are decommissioning.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It appears that all replicas of a block can end up in the same rack. The likelihood of such replicas seems to be directly related to decommissioning of nodes.

      Post rolling OS upgrade (decommission 3-10% of nodes, re-install etc, add them back) of a running cluster, all replicas of about 0.16% of blocks ended up in the same rack.

      Hadoop Namenode UI etc doesn't seem to know about such incorrectly replicated blocks. "hadoop fsck .." does report that the blocks must be replicated on additional racks.

      Looking at ReplicationTargetChooser.java, following seem suspect:

      snippet-01:

          int maxNodesPerRack =
            (totalNumOfReplicas-1)/clusterMap.getNumOfRacks()+2;
      

      snippet-02:

            case 2:
              if (clusterMap.isOnSameRack(results.get(0), results.get(1))) {
                chooseRemoteRack(1, results.get(0), excludedNodes,
                                 blocksize, maxNodesPerRack, results);
              } else if (newBlock){
                chooseLocalRack(results.get(1), excludedNodes, blocksize,
                                maxNodesPerRack, results);
              } else {
                chooseLocalRack(writer, excludedNodes, blocksize,
                                maxNodesPerRack, results);
              }
              if (--numOfReplicas == 0) {
                break;
              }
      

      snippet-03:

          do {
            DatanodeDescriptor[] selectedNodes =
              chooseRandom(1, nodes, excludedNodes);
            if (selectedNodes.length == 0) {
              throw new NotEnoughReplicasException(
                                                   "Not able to place enough replicas");
            }
            result = (DatanodeDescriptor)(selectedNodes[0]);
          } while(!isGoodTarget(result, blocksize, maxNodesPerRack, results));
      
      1. hdfs-1480.txt
        22 kB
        Todd Lipcon
      2. hdfs-1480.txt
        23 kB
        Todd Lipcon
      3. hdfs-1480.txt
        22 kB
        Todd Lipcon
      4. hdfs-1480-test.txt
        3 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/764/)
          HDFS-1480. All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/764/ ) HDFS-1480 . All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/783/)
          HDFS-1480. All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #783 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/783/ ) HDFS-1480 . All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #770 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/770/)
          HDFS-1480. All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #770 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/770/ ) HDFS-1480 . All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #773 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/773/)
          HDFS-1480. All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #773 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/773/ ) HDFS-1480 . All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #850 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/850/)
          HDFS-1480. All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #850 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/850/ ) HDFS-1480 . All replicas of a block can end up on the same rack when some datanodes are decommissioning. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160897 Files : /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java /hadoop/common/trunk/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          Todd Lipcon added a comment -

          The failing tests are also failing in trunk (see HDFS-2278). Will commit this to trunk momentarily, thanks for reviewing.

          Show
          Todd Lipcon added a comment - The failing tests are also failing in trunk (see HDFS-2278 ). Will commit this to trunk momentarily, thanks for reviewing.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491018/hdfs-1480.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//artifact/trunk/target/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491018/hdfs-1480.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//artifact/trunk/target/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1148//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491018/hdfs-1480.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//artifact/trunk/target/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491018/hdfs-1480.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//artifact/trunk/target/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1133//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Ah, I forgot that Hudson hadn't run this test yet. Uploading patch for Hudson. I'll commit if Hudson comes back clean.

          Show
          Todd Lipcon added a comment - Ah, I forgot that Hudson hadn't run this test yet. Uploading patch for Hudson. I'll commit if Hudson comes back clean.
          Hide
          Todd Lipcon added a comment -

          I'm going to move TestBlockManager back into src/test/java, since the unit tests aren't getting run in mavenized HDFS (see HDFS-2276). Will commit momentarily.

          Show
          Todd Lipcon added a comment - I'm going to move TestBlockManager back into src/test/java, since the unit tests aren't getting run in mavenized HDFS (see HDFS-2276 ). Will commit momentarily.
          Hide
          Eli Collins added a comment -

          +1 new mods look good

          Show
          Eli Collins added a comment - +1 new mods look good
          Hide
          Todd Lipcon added a comment -

          Moved the test to src/test/unit, and also rearranged the test methods a little bit, added some javadoc.

          Show
          Todd Lipcon added a comment - Moved the test to src/test/unit, and also rearranged the test methods a little bit, added some javadoc.
          Hide
          Eli Collins added a comment -

          +1 looks great.

          Move TestBlockManager to src/test/unit?

          Show
          Eli Collins added a comment - +1 looks great. Move TestBlockManager to src/test/unit?
          Hide
          Todd Lipcon added a comment -

          Here's a patch with a true unit test that shows this issue, as well as a fix. If this fix looks good, I will also backport it to the 0.20 branch.

          Show
          Todd Lipcon added a comment - Here's a patch with a true unit test that shows this issue, as well as a fix. If this fix looks good, I will also backport it to the 0.20 branch.
          Hide
          Todd Lipcon added a comment -

          Sorry, I think the above test actually fails because it will sometimes decommission all of the nodes on one of the test racks.

          But, if you bump it up to have 3 nodes in each rack, you'll see the new code path from HDFS-15 get triggered. – you can see it first re-replicate the block to be all one one host, and then after it gets the addStoredBlock calls, it notices it's not on enough racks, re-replicates elsewhere, and eventually the random choice gets it on the right one.

          Show
          Todd Lipcon added a comment - Sorry, I think the above test actually fails because it will sometimes decommission all of the nodes on one of the test racks. But, if you bump it up to have 3 nodes in each rack, you'll see the new code path from HDFS-15 get triggered. – you can see it first re-replicate the block to be all one one host, and then after it gets the addStoredBlock calls, it notices it's not on enough racks, re-replicates elsewhere, and eventually the random choice gets it on the right one.
          Hide
          Todd Lipcon added a comment -

          Here's a test which fails after you loop it a few times. I added some debug log messages and could see that maxNodesPerRack is getting set to 4.

          Show
          Todd Lipcon added a comment - Here's a test which fails after you loop it a few times. I added some debug log messages and could see that maxNodesPerRack is getting set to 4.
          Hide
          Todd Lipcon added a comment -

          There's definitely a bug still in trunk, here. Here's my analysis (in this case with repl=3):

          • Cluster has 2 racks, A and B, each with three nodes.
          • Block initially written on A1, B1, B2
          • Admin decommissions two of these nodes (let's say A1 and B1 but it doesn't matter)
          • in computeReplicationWorkForBlock, it calls chooseSourceDatanode which populates containingNodes with all three nodes regardless of decom status
          • this is then passed through as chosenNodes in the call to chooseTarget
          • maxNodesPerRack is then assigned (5-1)/2 + 2 = 4
          • chooseTarget initializes result with these same three nodes
          • {numOfResults}} is thus 3, and it calls chooseRandom rather than basing its decision on local/remote racks
          • chooseRandom is free to pick any two nodes since maxNodesPerRack is 4

          Will attach a failing unit test momentarily

          Show
          Todd Lipcon added a comment - There's definitely a bug still in trunk, here. Here's my analysis (in this case with repl=3): Cluster has 2 racks, A and B, each with three nodes. Block initially written on A1, B1, B2 Admin decommissions two of these nodes (let's say A1 and B1 but it doesn't matter) in computeReplicationWorkForBlock , it calls chooseSourceDatanode which populates containingNodes with all three nodes regardless of decom status this is then passed through as chosenNodes in the call to chooseTarget maxNodesPerRack is then assigned (5-1)/2 + 2 = 4 chooseTarget initializes result with these same three nodes {numOfResults}} is thus 3, and it calls chooseRandom rather than basing its decision on local/remote racks chooseRandom is free to pick any two nodes since maxNodesPerRack is 4 Will attach a failing unit test momentarily
          Hide
          Tsz Wo Nicholas Sze added a comment -

          If numOfResults==2 and two existing replicas are in the same rack, we are going to choose a remote node as shown below.

          //BlockPlacementPolicyDefault.chooseTarget(..)
          //line 194
                if (numOfResults <= 2) {
                  if (clusterMap.isOnSameRack(results.get(0), results.get(1))) {
                    chooseRemoteRack(1, results.get(0), excludedNodes,
                                     blocksize, maxNodesPerRack, results);
                  } else ...
          
          Show
          Tsz Wo Nicholas Sze added a comment - If numOfResults==2 and two existing replicas are in the same rack, we are going to choose a remote node as shown below. //BlockPlacementPolicyDefault.chooseTarget(..) //line 194 if (numOfResults <= 2) { if (clusterMap.isOnSameRack(results.get(0), results.get(1))) { chooseRemoteRack(1, results.get(0), excludedNodes, blocksize, maxNodesPerRack, results); } else ...
          Hide
          T Meyarivan added a comment -

          Sequence seems to be:

          chooseTarget() -> chooseLocalRack() -> chooseRandom() -> isGoodTarget()
          (IIUC numOfResults=2 if both replicas for a block with repl=2 are available)

          Multiple paths:

          [1] In chooseTarget(), writer may not be the decommissioning node
          [2] Whole rack (including the writer) is decommissioning => chooseLocalRack() picks a node from the same rack as other replica

          Show
          T Meyarivan added a comment - Sequence seems to be: chooseTarget() -> chooseLocalRack() -> chooseRandom() -> isGoodTarget() (IIUC numOfResults=2 if both replicas for a block with repl=2 are available) Multiple paths: [1] In chooseTarget(), writer may not be the decommissioning node [2] Whole rack (including the writer) is decommissioning => chooseLocalRack() picks a node from the same rack as other replica –
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Formatted the description.

          Show
          Tsz Wo Nicholas Sze added a comment - Formatted the description.
          Hide
          T Meyarivan added a comment -

          Does balancer check whether the movement of blocks satisfies replication policy ? (balancer was running most of the time during this upgrade etc)

          Show
          T Meyarivan added a comment - Does balancer check whether the movement of blocks satisfies replication policy ? (balancer was running most of the time during this upgrade etc) –
          Hide
          T Meyarivan added a comment -

          When repl=2:

          If the copies are on different racks and if one of them is being decommissioned, it is possible that both replicas end up on same rack (maxNodesPerRack =2, isGoodTarget returns True because maxNodesPerRack =2)

          Show
          T Meyarivan added a comment - When repl=2: If the copies are on different racks and if one of them is being decommissioned, it is possible that both replicas end up on same rack (maxNodesPerRack =2, isGoodTarget returns True because maxNodesPerRack =2) –
          Hide
          T Meyarivan added a comment -

          Probably the case.

          Is it likely to be backported to 20.x releases ?

          Show
          T Meyarivan added a comment - Probably the case. Is it likely to be backported to 20.x releases ? –
          Hide
          Todd Lipcon added a comment -

          Is this a duplicate of HDFS-15?

          Show
          Todd Lipcon added a comment - Is this a duplicate of HDFS-15 ?
          Hide
          T Meyarivan added a comment -

          [1] verified via "dfsadmin -report" that rack id etc is being computed correctly for all datanodes
          [2] number of such replicas doesn't seem to be reducing
          [3] balancer is/was running

          -

          [A] isGoodTarget() is broken for repl=2 ?
          [B] 'hadoop fsck ...' uses slightly different logic to determine whether a block is correctly replicated ?

          Show
          T Meyarivan added a comment - [1] verified via "dfsadmin -report" that rack id etc is being computed correctly for all datanodes [2] number of such replicas doesn't seem to be reducing [3] balancer is/was running - [A] isGoodTarget() is broken for repl=2 ? [B] 'hadoop fsck ...' uses slightly different logic to determine whether a block is correctly replicated ? –

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              T Meyarivan
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development