Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8204

Mover/Balancer should not schedule two replicas to the same DN

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Balancer moves blocks between Datanode(Ver. <2.6 ).
      Balancer moves blocks between StorageGroups ( introduced by HDFS-6584) , in the new version(Ver. >=2.6) .
      function

      class DBlock extends Locations<StorageGroup>
      DBlock.isLocatedOn(StorageGroup loc)
      

      is flawed, may causes 2 replicas ends in same node after running balance.

      For example:
      We have 2 nodes. Each node has two storages.
      We have (DN0, SSD), (DN0, DISK), (DN1, SSD), (DN1, DISK).
      We have a block with ONE_SSD storage policy.
      The block has 2 replicas. They are in (DN0,SSD) and (DN1,DISK).
      Replica in (DN0,SSD) should not be moved to (DN1,SSD) after running Balancer.
      Otherwise DN1 has 2 replicas.
      --------------
      UPDATE(Thanks Tsz Wo Nicholas Sze for pointing it out):

      This bug will NOT causes 2 replicas end in same node after running balance, thanks to Datanode rejecting it.

      We see a lot of ERROR when running test.

      2015-04-27 10:08:15,809 ERROR datanode.DataNode (DataXceiver.java:run(277)) - host1.foo.com:59537:DataXceiver error processing REPLACE_BLOCK operation  src: /127.0.0.1:52532 dst: /127.0.0.1:59537
      org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-264794661-9.96.1.34-1430100451121:blk_1073741825_1001 already exists in state FINALIZED and thus cannot be created.
          at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:1447)
          at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:186)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(DataXceiver.java:1158)
          at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReplaceBlock(Receiver.java:229)
          at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:77)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
          at java.lang.Thread.run(Thread.java:722)
      

      The Balancer runs 5~20 times iterations in the test, before it exits.
      It's ineffecient.
      Balancer should not schedule it in the first place, even though it'll failed anyway. In the test, it should exit after 5 times iteration.

      1. HDFS-8204.001.patch
        3 kB
        Walter Su
      2. HDFS-8204.002.patch
        5 kB
        Walter Su
      3. HDFS-8204.003.patch
        5 kB
        Walter Su

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          This seems a duplicate of HDFS-8147.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - This seems a duplicate of HDFS-8147 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The problem is actually that two replicas will be scheduled to be moved to the same node. The first replica will be moved to the node successfully but the second replica move will fail since there is already a replica in the node. Therefore the bug will NOT decrease the number of replicas. It only makes balancer scheduling inefficient.

          Walter Su, do you agree?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The problem is actually that two replicas will be scheduled to be moved to the same node. The first replica will be moved to the node successfully but the second replica move will fail since there is already a replica in the node. Therefore the bug will NOT decrease the number of replicas. It only makes balancer scheduling inefficient. Walter Su , do you agree?
          Hide
          walter.k.su Walter Su added a comment -

          Update: This issue is about "DN". HDFS-8147 is about "Storage". It's different.

          To Tsz Wo Nicholas Sze,
          >... It only makes balancer scheduling inefficient.
          Yes, it is. A little, not much.
          For example. A file with "cold" policy, 3 replicas.. Because one DN can only have one "cold" StorageGroup. There is no chance one DN has 2 replicas. The trunk code will reject it. So the 001 patch doesn't make balancer inefficient. "hot", "allssd" are the same.

          For example. A file with "onessd" policy, 5 replicas. So there is 1 ssd replica, 4 disk replicas. Balancer is more likely choose 2 "disk" StorageGroups to make a (source, target) pair. The trunk code will reject it. So the 001 patch doesn't make balancer inefficient.

          Only when choosing ("ssd" StorageGroup, "disk" StorageGroup) to make a (source, target) pair, trunk will accept it while the 001 patch reject it.

          For example. A file with "onessd" policy, 3 replicas. So there is 1 ssd replica, 2 disk replicas. What if the cluster is midium/huge? So many "disk" StorageGroups, balancer can choose (source, target) whatever it want. So it wil not be inefficient.

          What's more important, 2 replicas located on 1 DN doesn't match placement policy.

          Show
          walter.k.su Walter Su added a comment - Update: This issue is about "DN". HDFS-8147 is about "Storage". It's different. To Tsz Wo Nicholas Sze , >... It only makes balancer scheduling inefficient. Yes, it is. A little, not much. For example. A file with "cold" policy, 3 replicas.. Because one DN can only have one "cold" StorageGroup. There is no chance one DN has 2 replicas. The trunk code will reject it. So the 001 patch doesn't make balancer inefficient. "hot", "allssd" are the same. For example. A file with "onessd" policy, 5 replicas. So there is 1 ssd replica, 4 disk replicas. Balancer is more likely choose 2 "disk" StorageGroups to make a (source, target) pair. The trunk code will reject it. So the 001 patch doesn't make balancer inefficient. Only when choosing ("ssd" StorageGroup, "disk" StorageGroup) to make a (source, target) pair, trunk will accept it while the 001 patch reject it. For example. A file with "onessd" policy, 3 replicas. So there is 1 ssd replica, 2 disk replicas. What if the cluster is midium/huge? So many "disk" StorageGroups, balancer can choose (source, target) whatever it want. So it wil not be inefficient. What's more important, 2 replicas located on 1 DN doesn't match placement policy.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          My previous comment was for the bug, not the patch.

          > What's more important, 2 replicas located on 1 DN doesn't match placement policy.

          I am confused. Do you actually see 2 replicas stored in 1 DN? Or you only see two replicas scheduled to be moved to the same DN?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - My previous comment was for the bug, not the patch. > What's more important, 2 replicas located on 1 DN doesn't match placement policy. I am confused. Do you actually see 2 replicas stored in 1 DN? Or you only see two replicas scheduled to be moved to the same DN?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Just found that you updated the summary. So the bug is only in scheduling but it does not decease the number of replicas. Then, it is the same as HDFS-8147.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Just found that you updated the summary. So the bug is only in scheduling but it does not decease the number of replicas. Then, it is the same as HDFS-8147 .
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > Update: This issue is about "DN". HDFS-8147 is about "Storage". It's different.

          I see. Then, please checking the same storage group is not good enough. It should check also the node.

          Some comments for the patch:

          • The logic could be simplified as below.
                final DatanodeInfo targetDatanode = target.getDatanodeInfo();
                for (StorageGroup blockLocation : block.getLocations()) {
                  if (blockLocation.getDatanodeInfo().equals(targetDatanode)) {
                    return false;
                  }
                }
            
          • The following is not needed since isGoodBlockCandidate is already called via source.addPendingMove(db, target).
            +            if (!dispatcher.isGoodBlockCandidate(source, target, t, db)) {
            +              continue;
            +            }
            
          • Could you add/modify a test?
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > Update: This issue is about "DN". HDFS-8147 is about "Storage". It's different. I see. Then, please checking the same storage group is not good enough. It should check also the node. Some comments for the patch: The logic could be simplified as below. final DatanodeInfo targetDatanode = target.getDatanodeInfo(); for (StorageGroup blockLocation : block.getLocations()) { if (blockLocation.getDatanodeInfo().equals(targetDatanode)) { return false ; } } The following is not needed since isGoodBlockCandidate is already called via source.addPendingMove(db, target). + if (!dispatcher.isGoodBlockCandidate(source, target, t, db)) { + continue ; + } Could you add/modify a test?
          Hide
          walter.k.su Walter Su added a comment -

          upload 002 patch, please review.
          And sorry the description is not clear. I add an example to it.

          Show
          walter.k.su Walter Su added a comment - upload 002 patch, please review. And sorry the description is not clear. I add an example to it.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Pending Jenkins.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Pending Jenkins.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... may causes 2 replicas ends in same node after running balance.

          Indeed, it won't happen. The target datanode will reject to receive the replica since it already has it. So this is a bug only making the scheduling inefficient but won't cause 2 replicas moved to the same node.

          Try your new test without the Dispatcher change. It still passes and you will find ReplicaAlreadyExistsException in the log.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... may causes 2 replicas ends in same node after running balance. Indeed, it won't happen. The target datanode will reject to receive the replica since it already has it. So this is a bug only making the scheduling inefficient but won't cause 2 replicas moved to the same node. Try your new test without the Dispatcher change. It still passes and you will find ReplicaAlreadyExistsException in the log.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 20s The applied patch generated 1 additional checkstyle issues.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 13s Pre-build of native portion
          -1 hdfs tests 162m 27s Tests failed in hadoop-hdfs.
              208m 16s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.mover.TestMover



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728130/HDFS-8204.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a00e001
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10394/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10394/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10394/testReport/
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10394/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 20s The applied patch generated 1 additional checkstyle issues. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 162m 27s Tests failed in hadoop-hdfs.     208m 16s   Reason Tests Failed unit tests hadoop.hdfs.server.mover.TestMover Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728130/HDFS-8204.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a00e001 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10394/artifact/patchprocess/checkstyle-result-diff.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10394/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10394/testReport/ Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10394/console This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          > ... Indeed, it won't happen. ...It still passes and you will find ReplicaAlreadyExistsException in the log.
          Yes. I didn't realized that. Thanks Tsz Wo Nicholas Sze.

          Update description. And 003 patch fixes Failed Tests.

          Show
          walter.k.su Walter Su added a comment - > ... Indeed, it won't happen. ...It still passes and you will find ReplicaAlreadyExistsException in the log. Yes. I didn't realized that. Thanks Tsz Wo Nicholas Sze . Update description. And 003 patch fixes Failed Tests.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 27s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 3m 59s The applied patch generated 1 additional checkstyle issues.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 170m 9s Tests passed in hadoop-hdfs.
              214m 45s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728301/HDFS-8204.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1a2459b
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10404/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10404/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10404/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10404/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 27s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 59s The applied patch generated 1 additional checkstyle issues. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 170m 9s Tests passed in hadoop-hdfs.     214m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728301/HDFS-8204.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1a2459b checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10404/artifact/patchprocess/checkstyle-result-diff.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10404/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10404/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10404/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 5m 24s There were no new checkstyle issues.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 167m 11s Tests passed in hadoop-hdfs.
              213m 8s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728324/HDFS-8204.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 618ba70
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10407/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10407/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10407/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 24s There were no new checkstyle issues. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 167m 11s Tests passed in hadoop-hdfs.     213m 8s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728324/HDFS-8204.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 618ba70 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10407/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10407/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10407/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -
              if (source.getDatanodeInfo().equals(targetDatanode)) {
                // the block is moved inside same DN
                return true;
              }
          

          It seems that the DataTransferProtocol currently does not support moving within the same node since it will get ReplicaAlreadyExistsException when creating the new block. Do you agree?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - if (source.getDatanodeInfo().equals(targetDatanode)) { // the block is moved inside same DN return true ; } It seems that the DataTransferProtocol currently does not support moving within the same node since it will get ReplicaAlreadyExistsException when creating the new block. Do you agree?
          Hide
          walter.k.su Walter Su added a comment -

          Actually, It does support. You can run TestMover.testScheduleBlockWithinSameNode(). The testcase moves block within same DN.

          DataXceiver.java
          1103     try {
          1104       // Move the block to different storage in the same datanode
          1105       if (proxySource.equals(datanode.getDatanodeId())) {
          1106         ReplicaInfo oldReplica = datanode.data.moveBlockAcrossStorage(block,
          1107             storageType);
          1108         if (oldReplica != null) {
          1109           LOG.info("Moved " + block + " from StorageType "
          1110               + oldReplica.getVolume().getStorageType() + " to " + storageType);
          1111         }
          1112       } else {
          
          Show
          walter.k.su Walter Su added a comment - Actually, It does support. You can run TestMover.testScheduleBlockWithinSameNode(). The testcase moves block within same DN. DataXceiver.java 1103 try { 1104 // Move the block to different storage in the same datanode 1105 if (proxySource.equals(datanode.getDatanodeId())) { 1106 ReplicaInfo oldReplica = datanode.data.moveBlockAcrossStorage(block, 1107 storageType); 1108 if (oldReplica != null ) { 1109 LOG.info( "Moved " + block + " from StorageType " 1110 + oldReplica.getVolume().getStorageType() + " to " + storageType); 1111 } 1112 } else {
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > Actually, It does support. ...

          You are right. Thanks for point it out.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > Actually, It does support. ... You are right. Thanks for point it out.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Walter!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Walter!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7694 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7694/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7694 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7694/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2110 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2110/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2110 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2110/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #169 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/169/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #169 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/169/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/178/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/178/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #912 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/912/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #912 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/912/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/179/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/179/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2128/)
          HDFS-8204. Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2128/ ) HDFS-8204 . Mover/Balancer should not schedule two replicas to the same datanode. Contributed by Walter Su (szetszwo: rev 5639bf02da716b3ecda785979b3d08cdca15972d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              walter.k.su Walter Su
              Reporter:
              walter.k.su Walter Su
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development