Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8863

The remaining space check in BlockPlacementPolicyDefault is flawed

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The block placement policy calls DatanodeDescriptor#getRemaining(StorageType}} to check whether the block is going to fit. Since the method is adding up all remaining spaces, namenode can allocate a new block on a full node. This causes pipeline construction failure and abandonBlock. If the cluster is nearly full, the client might hit this multiple times and the write can fail permanently.

      1. HDFS-8863.patch
        7 kB
        Kihwal Lee
      2. HDFS-8863.v2.patch
        7 kB
        Kihwal Lee
      3. HDFS-8863.v3.patch
        9 kB
        Kihwal Lee
      4. HDFS-8863-branch-2.6.1.txt
        10 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          We have a patch that has been tested. I will post it soon.

          Show
          kihwal Kihwal Lee added a comment - We have a patch that has been tested. I will post it soon.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 31s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 14s There were no new javac warning messages.
          +1 javadoc 10m 13s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 24s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 19s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 9s Pre-build of native portion
          -1 hdfs tests 162m 40s Tests failed in hadoop-hdfs.
              210m 1s  



          Reason Tests
          Timed out tests org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749647/HDFS-8863.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11954/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11954/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11954/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 31s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 14s There were no new javac warning messages. +1 javadoc 10m 13s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 162m 40s Tests failed in hadoop-hdfs.     210m 1s   Reason Tests Timed out tests org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   org.apache.hadoop.hdfs.server.namenode.TestFSImageWithAcl Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749647/HDFS-8863.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11954/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11954/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11954/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          The patch looks good, thanks Kihwal.

          Agree to check the maximum remaining of storage instead of summing all remaining spaces, since we can't split a block to multiple storage.
          One question is: should we still keep getRemaining(StorageType t) there? No place to call it, so can remove it?

          Nits in the test:

          DatanodeStorageInfo si = new DatanodeStorageInfo(
                  storages[5].getDatanodeDescriptor(), extraStorage);
          

          It's unused and can be removed.

          // No available space in the extra storage of dn0
          updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L);
          

          typo in comment: dn0 should be dn5

          It's better to set no available space in the extra storage at the end of testChooseNodeWithMultipleStorages too:

          updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L);
          
          Show
          hitliuyi Yi Liu added a comment - - edited The patch looks good, thanks Kihwal. Agree to check the maximum remaining of storage instead of summing all remaining spaces, since we can't split a block to multiple storage. One question is: should we still keep getRemaining(StorageType t) there? No place to call it, so can remove it? Nits in the test: DatanodeStorageInfo si = new DatanodeStorageInfo( storages[5].getDatanodeDescriptor(), extraStorage); It's unused and can be removed. // No available space in the extra storage of dn0 updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L); typo in comment: dn0 should be dn5 It's better to set no available space in the extra storage at the end of testChooseNodeWithMultipleStorages too: updateHeartbeatForExtraStorage(0L, 0L, 0L, 0L);
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The change looks good, thanks for fixing this. Agree that we can eliminate getRemaining(StorageType t).

          Show
          arpitagarwal Arpit Agarwal added a comment - The change looks good, thanks for fixing this. Agree that we can eliminate getRemaining(StorageType t) .
          Hide
          hitliuyi Yi Liu added a comment - - edited

          I found an issue here, I think getRemaining should take parameter DatanodeStorageInfo instead of StorageType, and it should just return current storage remaining space instead of get the maximum remaining space of all storages, since when choosing datanode storage to place block, BlockPlacementPolicyDefault will iterator all storages and try to find a suitable one. Both sum and maximum of remaining is not correct.

          I give an example here, suppose the datanode has two storages whose storage type is DISK, and we are going to find the DISK type too, but the remaining space of the first storage is not enough to place the block, and the second one has enough space. According to current patch, the first datanode storage will return, actually we want the second one.

          Show
          hitliuyi Yi Liu added a comment - - edited I found an issue here, I think getRemaining should take parameter DatanodeStorageInfo instead of StorageType , and it should just return current storage remaining space instead of get the maximum remaining space of all storages, since when choosing datanode storage to place block, BlockPlacementPolicyDefault will iterator all storages and try to find a suitable one. Both sum and maximum of remaining is not correct. I give an example here, suppose the datanode has two storages whose storage type is DISK, and we are going to find the DISK type too, but the remaining space of the first storage is not enough to place the block, and the second one has enough space. According to current patch, the first datanode storage will return, actually we want the second one.
          Hide
          hitliuyi Yi Liu added a comment -
          final long requiredSize = blockSize * HdfsServerConstants.MIN_BLOCKS_FOR_WRITE;
          final long scheduledSize = blockSize * node.getBlocksScheduled(storage.getStorageType());
          final long remaining = node.getRemaining(storage.getStorageType());
          if (requiredSize > remaining - scheduledSize) {
          

          Another issue, getBlocksScheduled is for storage type, not for per storage. So about the getRemaining:

          1. sum of remaining space for storages of the storage type. (current trunk)
          2. the maximum remaining space for storages of the storage type. (current patch)
          3. the remaining space of current storage. (As I said in previous comment)

          All of them are not correct, I think we should make the blocksScheduled to track per storage instead of storage type. Then we can choose #3 as my previous comment.

          Show
          hitliuyi Yi Liu added a comment - final long requiredSize = blockSize * HdfsServerConstants.MIN_BLOCKS_FOR_WRITE; final long scheduledSize = blockSize * node.getBlocksScheduled(storage.getStorageType()); final long remaining = node.getRemaining(storage.getStorageType()); if (requiredSize > remaining - scheduledSize) { Another issue, getBlocksScheduled is for storage type, not for per storage. So about the getRemaining : sum of remaining space for storages of the storage type. (current trunk) the maximum remaining space for storages of the storage type. (current patch) the remaining space of current storage. (As I said in previous comment) All of them are not correct, I think we should make the blocksScheduled to track per storage instead of storage type. Then we can choose #3 as my previous comment.
          Hide
          kihwal Kihwal Lee added a comment -

          it should just return current storage remaining space instead of get the maximum remaining space of all storages

          Datanodes only care about the storage type, so checking a particular storagewon't do any good. It will just cause block placement to re-pick target more.

          Another issue, getBlocksScheduled is for storage type, not for per storage.

          Tracking scheduled writes per storage is not going to solve the problem since datanodes are free to choose any storage as long as the type matches. Trying to achieve precise accounting will have diminishing return as there are uncertainties around actual storage being used, blocks being abandoned, control loop delays (heartbeats), etc.

          What if we let it check against storage type level sum and also make sure there is at least one storage with enough space? I actually had a version of patch that does just that. I will remove unused method and post the patch.

          Show
          kihwal Kihwal Lee added a comment - it should just return current storage remaining space instead of get the maximum remaining space of all storages Datanodes only care about the storage type, so checking a particular storagewon't do any good. It will just cause block placement to re-pick target more. Another issue, getBlocksScheduled is for storage type, not for per storage. Tracking scheduled writes per storage is not going to solve the problem since datanodes are free to choose any storage as long as the type matches. Trying to achieve precise accounting will have diminishing return as there are uncertainties around actual storage being used, blocks being abandoned, control loop delays (heartbeats), etc. What if we let it check against storage type level sum and also make sure there is at least one storage with enough space? I actually had a version of patch that does just that. I will remove unused method and post the patch.
          Hide
          kihwal Kihwal Lee added a comment -

          Attaching new patch.

          Show
          kihwal Kihwal Lee added a comment - Attaching new patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 0s There were no new javac warning messages.
          +1 javadoc 10m 35s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 30s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 25s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 17s Pre-build of native portion
          -1 hdfs tests 175m 14s Tests failed in hadoop-hdfs.
              221m 54s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDatanodeDeath
            hadoop.hdfs.TestReplaceDatanodeOnFailure
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749845/HDFS-8863.v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fa1d84a
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11963/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11963/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11963/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 35s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 30s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 17s Pre-build of native portion -1 hdfs tests 175m 14s Tests failed in hadoop-hdfs.     221m 54s   Reason Tests Failed unit tests hadoop.hdfs.TestDatanodeDeath   hadoop.hdfs.TestReplaceDatanodeOnFailure Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749845/HDFS-8863.v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fa1d84a hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11963/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11963/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11963/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          What if we let it check against storage type level sum and also make sure there is at least one storage with enough space?

          Still have potential issue. For example, we have datanode dn0, and three storages(s1, s2, s3) of required storage type. Both s1 and s3 has 2/3 block size remaining space, and s2 has 1+2/3 block size remaining space. We just scheduled one block on dn0, it's certainly on s2, now a new block is adding and block placement checks dn0, for current patch, it will see the maximum of remaining space is 1 + 2/3 block size (s2), and also the sum satisfy, so treat it as a good target, but actually it's not.

          I am thinking we can do as following: do storage type level sum, but for each storage, we only count the remaining space of multiple block size part, so for above example, remaining space of s1 and s3 is counted 0, s2 is 1, then the sum is 1, dn0 is not a good target. In this approach, we don't need to check the maximum too.

          Datanodes only care about the storage type, so checking a particular storagewon't do any good. It will just cause block placement to re-pick target more.

          You are right, I also had another meaning: when iterating storages, it's to check the remaining space of storage type, but actually some back storages may be State.FAILED or State.READ_ONLY_SHARED, it's remaining space is still be counted, right? So I think you can do these check in getRemaining. See my JIRA HDFS-8884, which has relation to this, I do fast-fail check for datanode, of course, I can do this part in my JIRA if you don't do it here.

          Show
          hitliuyi Yi Liu added a comment - - edited What if we let it check against storage type level sum and also make sure there is at least one storage with enough space? Still have potential issue. For example, we have datanode dn0, and three storages(s1, s2, s3) of required storage type. Both s1 and s3 has 2/3 block size remaining space, and s2 has 1+2/3 block size remaining space. We just scheduled one block on dn0, it's certainly on s2, now a new block is adding and block placement checks dn0, for current patch, it will see the maximum of remaining space is 1 + 2/3 block size (s2), and also the sum satisfy, so treat it as a good target, but actually it's not. I am thinking we can do as following: do storage type level sum, but for each storage, we only count the remaining space of multiple block size part, so for above example, remaining space of s1 and s3 is counted 0, s2 is 1, then the sum is 1, dn0 is not a good target. In this approach, we don't need to check the maximum too. Datanodes only care about the storage type, so checking a particular storagewon't do any good. It will just cause block placement to re-pick target more. You are right, I also had another meaning: when iterating storages, it's to check the remaining space of storage type, but actually some back storages may be State.FAILED or State.READ_ONLY_SHARED , it's remaining space is still be counted, right? So I think you can do these check in getRemaining . See my JIRA HDFS-8884 , which has relation to this, I do fast-fail check for datanode, of course, I can do this part in my JIRA if you don't do it here.
          Hide
          kihwal Kihwal Lee added a comment -

          I am thinking we can do as following: do storage type level sum, but for each storage, we only count the remaining space of multiple block size part...

          That sounds like a better idea.

          So I think you can do these check in getRemaining.

          I think that is reasonable thing to do in this patch.

          Show
          kihwal Kihwal Lee added a comment - I am thinking we can do as following: do storage type level sum, but for each storage, we only count the remaining space of multiple block size part... That sounds like a better idea. So I think you can do these check in getRemaining. I think that is reasonable thing to do in this patch.
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Kihwal Lee for updating, +1 pending Jenkins.

          Show
          hitliuyi Yi Liu added a comment - Thanks Kihwal Lee for updating, +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 0s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 9m 55s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 32s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 12s Pre-build of native portion
          -1 hdfs tests 204m 2s Tests failed in hadoop-hdfs.
              246m 51s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.TestClientReportBadBlock
            hadoop.hdfs.server.namenode.TestFileTruncate
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751137/HDFS-8863.v3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 30e342a
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12032/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12032/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12032/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 0s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 32s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 204m 2s Tests failed in hadoop-hdfs.     246m 51s   Reason Tests Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.TestClientReportBadBlock   hadoop.hdfs.server.namenode.TestFileTruncate Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751137/HDFS-8863.v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 30e342a hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12032/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12032/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12032/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          The failures are not related and they run successfully in my local env, will commit shortly.

          Show
          hitliuyi Yi Liu added a comment - The failures are not related and they run successfully in my local env, will commit shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8327 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8327/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8327 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8327/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          hitliuyi Yi Liu added a comment -

          Committed to trunk, branch-2, branch-2.7. Thanks Kihwal Lee for contribution, and Arpit Agarwal for review.

          Show
          hitliuyi Yi Liu added a comment - Committed to trunk, branch-2, branch-2.7. Thanks Kihwal Lee for contribution, and Arpit Agarwal for review.
          Hide
          kihwal Kihwal Lee added a comment -

          Thanks, Yi Liu.

          Show
          kihwal Kihwal Lee added a comment - Thanks, Yi Liu .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/291/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/291/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1024 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1024/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1024 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1024/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #294 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/294/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #294 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/294/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2240 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2240/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2240 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2240/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #283 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/283/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #283 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/283/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2221/)
          HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2221/ ) HDFS-8863 . The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) (yliu: rev 5e8fe8943718309b5e39a794360aebccae28b331) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1.

          Had to fix a couple of minor merge conflicts and make minor changes to TestReplicationPolicy to make it work on 2.6.1.

          Ran compilation and TestReplicationPolicy before the push.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. Had to fix a couple of minor merge conflicts and make minor changes to TestReplicationPolicy to make it work on 2.6.1. Ran compilation and TestReplicationPolicy before the push.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Attaching patch that I committed to 2.6.1.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Attaching patch that I committed to 2.6.1.

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development