Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8946

Improve choosing datanode storage for block placement

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA is to:
      Improve chooseing datanode storage for block placement:
      In BlockPlacementPolicyDefault (chooseLocalStorage, chooseRandom), we have following logic to choose datanode storage to place block.
      For given storage type, we iterate storages of the datanode. But for datanode, it only cares about the storage type. In the loop, we check according to Storage type and return the first storage if the storages of the type on the datanode fit in requirement. So we can remove the iteration of storages, and just need to do once to find a good storage of given type, it's efficient if the storages of the type on the datanode don't fit in requirement since we don't need to loop all storages and do the same check.
      Besides, no need to shuffle the storages, since we only need to check according to the storage type on the datanode once.
      This also improves the logic and make it more clear.

            if (excludedNodes.add(localMachine) // was not in the excluded list
                && isGoodDatanode(localDatanode, maxNodesPerRack, false,
                    results, avoidStaleNodes)) {
              for (Iterator<Map.Entry<StorageType, Integer>> iter = storageTypes
                  .entrySet().iterator(); iter.hasNext(); ) {
                Map.Entry<StorageType, Integer> entry = iter.next();
                for (DatanodeStorageInfo localStorage : DFSUtil.shuffle(
                    localDatanode.getStorageInfos())) {
                  StorageType type = entry.getKey();
                  if (addIfIsGoodTarget(localStorage, excludedNodes, blocksize,
                      results, type) >= 0) {
                    int num = entry.getValue();
                    ...
      

      (current logic above)

      1. HDFS-8946.001.patch
        14 kB
        Yi Liu
      2. HDFS-8946.002.patch
        14 kB
        Yi Liu
      3. HDFS-8946.003.patch
        14 kB
        Yi Liu

        Issue Links

          Activity

          Hide
          hitliuyi Yi Liu added a comment - - edited
          1. Add chooseStorage4Block to choose a good storage of given type from the datanode. It will checks the state of storage, and whether there is enough space to place the block on the datanode. These conditions are the same as original. Since the datanode only cares about the storage type when placing a block, this method will return the first storage of given type we see.
            Then isGoodTarget can be removed, since we check all the conditions in chooseStorage4Block.
          2. No need to shuffle the storages of datanode and no need to iterate all the storages of datanode in chooseLocalStorage and chooseRandom, since we can choose the storage of given type using chooseStorage4Block once.
          3. addToExcludedNodes is redundant after finding a good storage, since we already add the data node to excludedNodes at the begging of checking the node.
          4. numOfAvailableNodes -= newExcludedNodes; in chooseRandom is redundant, since we already do numOfAvailableNodes-- at the begging of checking the node.
            (we can't remove addToExcludedNodes, since it's override by subclass)
          5. In DatanodeDescriptor#chooseStorage4Block, t == null is unnecessary, since it never be null and it's added in HDFS-8863.
          Show
          hitliuyi Yi Liu added a comment - - edited Add chooseStorage4Block to choose a good storage of given type from the datanode. It will checks the state of storage, and whether there is enough space to place the block on the datanode. These conditions are the same as original. Since the datanode only cares about the storage type when placing a block, this method will return the first storage of given type we see. Then isGoodTarget can be removed, since we check all the conditions in chooseStorage4Block . No need to shuffle the storages of datanode and no need to iterate all the storages of datanode in chooseLocalStorage and chooseRandom , since we can choose the storage of given type using chooseStorage4Block once. addToExcludedNodes is redundant after finding a good storage, since we already add the data node to excludedNodes at the begging of checking the node. numOfAvailableNodes -= newExcludedNodes; in chooseRandom is redundant, since we already do numOfAvailableNodes-- at the begging of checking the node. (we can't remove addToExcludedNodes , since it's override by subclass) In DatanodeDescriptor#chooseStorage4Block , t == null is unnecessary, since it never be null and it's added in HDFS-8863 .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 17s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 85, now 83).
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 188m 13s Tests failed in hadoop-hdfs.
              233m 9s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
            hadoop.hdfs.server.namenode.TestFSNamesystem
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752472/HDFS-8946.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a4d9acc
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12132/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12132/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 17s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 85, now 83). -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 188m 13s Tests failed in hadoop-hdfs.     233m 9s   Reason Tests Failed unit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling   hadoop.hdfs.server.namenode.TestFSNamesystem   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752472/HDFS-8946.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12132/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12132/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12132/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Update the patch to fix test failure, the reason is: we can't remove addToExcludedNodes, since it's override by subclass.

          Show
          hitliuyi Yi Liu added a comment - Update the patch to fix test failure, the reason is: we can't remove addToExcludedNodes, since it's override by subclass.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 24s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 19s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          +1 hdfs tests 161m 10s Tests passed in hadoop-hdfs.
              205m 25s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752741/HDFS-8946.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0bf2854
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12162/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12162/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12162/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12162/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 24s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 19s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion +1 hdfs tests 161m 10s Tests passed in hadoop-hdfs.     205m 25s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752741/HDFS-8946.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0bf2854 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12162/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12162/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12162/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12162/console This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks for working on this, Yi Liu. I read the code of BlockPlacementPolicyDefault for HDFS-8945 recently and comment here while my memory is fresh

          Besides, no need to shuffle the storages, since we only need to check according to the storage type on the datanode once.

          Here is my understanding of this. Please correct me if I'm wrong:

          LocatedBlock returned by ClientProtocol#addBlock, ClientProtocol#getAdditionalDatanode and ClientProtocol#updateBlockForPipeline contains storageIDs given by BlockPlacementPolicy#chooseTarget but the user of these APIs (which is only DataStreamer) does not uses storageIDs. DataStreamer just send storage type to DataNode and the DataNode decides which volume to use on its own by using VolumeChoosingPolicy.

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks for working on this, Yi Liu . I read the code of BlockPlacementPolicyDefault for HDFS-8945 recently and comment here while my memory is fresh Besides, no need to shuffle the storages, since we only need to check according to the storage type on the datanode once. Here is my understanding of this. Please correct me if I'm wrong: LocatedBlock returned by ClientProtocol#addBlock , ClientProtocol#getAdditionalDatanode and ClientProtocol#updateBlockForPipeline contains storageIDs given by BlockPlacementPolicy#chooseTarget but the user of these APIs (which is only DataStreamer) does not uses storageIDs. DataStreamer just send storage type to DataNode and the DataNode decides which volume to use on its own by using VolumeChoosingPolicy .
          Hide
          iwasakims Masatake Iwasaki added a comment -

          DatanodeDescriptor.java

          679	    final long requiredSize =
          680	        blockSize * HdfsServerConstants.MIN_BLOCKS_FOR_WRITE;
          681	    final long scheduledSize = getBlocksScheduled(t);
          682	    long remaining = 0;
          683	    DatanodeStorageInfo storage = null;
          684	    for (DatanodeStorageInfo s : getStorageInfos()) {
          685	      if (s.getState() == State.NORMAL &&
          686	          s.getStorageType() == t) {
          687	        if (storage == null) {
          688	          storage = s;
          689	        }
          690	        long r = s.getRemaining();
          691	        if (r >= requiredSize) {
          692	          remaining += r;
          693	        }
          694	      }
          695	    }
          696	    if (requiredSize > remaining - scheduledSize) {
          697	      return null;
          

          scheduledSize is number of blocks but used as if it's bytes.

          Any reason to change the logic of remaining size checking?

          Show
          iwasakims Masatake Iwasaki added a comment - DatanodeDescriptor.java 679 final long requiredSize = 680 blockSize * HdfsServerConstants.MIN_BLOCKS_FOR_WRITE; 681 final long scheduledSize = getBlocksScheduled(t); 682 long remaining = 0; 683 DatanodeStorageInfo storage = null ; 684 for (DatanodeStorageInfo s : getStorageInfos()) { 685 if (s.getState() == State.NORMAL && 686 s.getStorageType() == t) { 687 if (storage == null ) { 688 storage = s; 689 } 690 long r = s.getRemaining(); 691 if (r >= requiredSize) { 692 remaining += r; 693 } 694 } 695 } 696 if (requiredSize > remaining - scheduledSize) { 697 return null ; scheduledSize is number of blocks but used as if it's bytes. Any reason to change the logic of remaining size checking?
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Masatake for the comments.

          LocatedBlock returned by ClientProtocol#addBlock, ClientProtocol#getAdditionalDatanode and ClientProtocol#updateBlockForPipeline contains storageIDs given by BlockPlacementPolicy#chooseTarget but the user of these APIs (which is only DataStreamer) does not uses storageIDs. DataStreamer just send storage type to DataNode and the DataNode decides which volume to use on its own by using VolumeChoosingPolicy.

          Yes, storageIDs is not used.

          Any reason to change the logic of remaining size checking?

          Nice find, It's my fault to forget to add blockSize * while coping this logic from original isGoodTarget

          Show
          hitliuyi Yi Liu added a comment - Thanks Masatake for the comments. LocatedBlock returned by ClientProtocol#addBlock, ClientProtocol#getAdditionalDatanode and ClientProtocol#updateBlockForPipeline contains storageIDs given by BlockPlacementPolicy#chooseTarget but the user of these APIs (which is only DataStreamer) does not uses storageIDs. DataStreamer just send storage type to DataNode and the DataNode decides which volume to use on its own by using VolumeChoosingPolicy. Yes, storageIDs is not used. Any reason to change the logic of remaining size checking? Nice find, It's my fault to forget to add blockSize * while coping this logic from original isGoodTarget
          Hide
          hitliuyi Yi Liu added a comment -

          Will update the patch later.

          Show
          hitliuyi Yi Liu added a comment - Will update the patch later.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 20s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 21s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 18s Pre-build of native portion
          -1 hdfs tests 185m 32s Tests failed in hadoop-hdfs.
              231m 51s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestRollingUpgrade



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752975/HDFS-8946.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e166c03
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12194/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12194/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12194/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 20s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 21s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 18s Pre-build of native portion -1 hdfs tests 185m 32s Tests failed in hadoop-hdfs.     231m 51s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752975/HDFS-8946.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e166c03 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12194/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12194/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12194/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          The test failure is not related, they run successfully in my local env.

          Show
          hitliuyi Yi Liu added a comment - The test failure is not related, they run successfully in my local env.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 the patch looks good.

          Show
          kihwal Kihwal Lee added a comment - +1 the patch looks good.
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks a lot for the view, Kihwal Lee!
          Will commit shortly.

          Show
          hitliuyi Yi Liu added a comment - Thanks a lot for the view, Kihwal Lee ! Will commit shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8375/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8375/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1058 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1058/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1058 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1058/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #325 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/325/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #325 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/325/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/332/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/332/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2274/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2274/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/316/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/316/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2255/)
          HDFS-8946. Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2255/ ) HDFS-8946 . Improve choosing datanode storage for block placement. (yliu) (yliu: rev 8fa41d9dd4b923bf4141f019414a1a8b079124c6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          One suggestion :

          In this jira some logs are removed.
          1.

          if (requiredSize > remaining - scheduledSize) {
                logNodeIsNotChosen(storage, "the node does not have enough "
                    + storage.getStorageType() + " space"
                    + " (required=" + requiredSize
                    + ", scheduled=" + scheduledSize
                    + ", remaining=" + remaining + ")");
                return false;
              }
          

          2.

          logNodeIsNotChosen(storage, "storage is read-only");
          

          3.

          logNodeIsNotChosen(storage, "storage has failed");
          

          I think this is very important for debugging issue. From debug log we will get to know "why DN is not selected for block replication".

          Can we add this back? At least first log should be there.

          In HDFS-9023 some log improvement I want to do, when NN is not able to identify DN's for replication.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - One suggestion : In this jira some logs are removed. 1. if (requiredSize > remaining - scheduledSize) { logNodeIsNotChosen(storage, "the node does not have enough " + storage.getStorageType() + " space" + " (required=" + requiredSize + ", scheduled=" + scheduledSize + ", remaining=" + remaining + ")" ); return false ; } 2. logNodeIsNotChosen(storage, "storage is read-only" ); 3. logNodeIsNotChosen(storage, "storage has failed" ); I think this is very important for debugging issue. From debug log we will get to know "why DN is not selected for block replication". Can we add this back? At least first log should be there. In HDFS-9023 some log improvement I want to do, when NN is not able to identify DN's for replication.

            People

            • Assignee:
              hitliuyi Yi Liu
              Reporter:
              hitliuyi Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development