Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5188

Clean up BlockPlacementPolicy and its implementations

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Below is a list of code cleanup tasks for BlockPlacementPolicy and its implementations:

      • Define constants for dfs.block.replicator.classname.
      • Combine adjustExcludedNodes and addToExcludedNodes: subclasses should always add all the related nodes in addToExcludedNodes(..).
      • Remove duplicated code.
      1. h5188_20130915.patch
        77 kB
        Tsz Wo Nicholas Sze
      2. h5188_20130914.patch
        78 kB
        Tsz Wo Nicholas Sze
      3. h5188_20130912b.patch
        76 kB
        Tsz Wo Nicholas Sze
      4. h5188_20130912.patch
        73 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          h5188_20130912.patch: 1st patch.

          Show
          Tsz Wo Nicholas Sze added a comment - h5188_20130912.patch: 1st patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          More details about h5188_20130912.patch:

          • removes a BlockPlacementPolicy.chooseTarget(..) used only in tests;
          • removes another BlockPlacementPolicy.chooseTarget(..) used only once;
          • replaces HashMap with Map in parameter declarations;
          • removes adjustExcludedNodes(..);
          • defines DFS_BLOCK_REPLICATOR_CLASSNAME_KEY and DFS_BLOCK_REPLICATOR_CLASSNAME_DEFAULT;
          • cleans up duplicated code by adding new methods.
          Show
          Tsz Wo Nicholas Sze added a comment - More details about h5188_20130912.patch: removes a BlockPlacementPolicy.chooseTarget(..) used only in tests; removes another BlockPlacementPolicy.chooseTarget(..) used only once; replaces HashMap with Map in parameter declarations; removes adjustExcludedNodes(..); defines DFS_BLOCK_REPLICATOR_CLASSNAME_KEY and DFS_BLOCK_REPLICATOR_CLASSNAME_DEFAULT; cleans up duplicated code by adding new methods.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12602740/h5188_20130912.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4959//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4959//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602740/h5188_20130912.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4959//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4959//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h5188_20130912b.patch: fixes test failure.

          Show
          Tsz Wo Nicholas Sze added a comment - h5188_20130912b.patch: fixes test failure.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12602818/h5188_20130912b.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4960//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4960//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602818/h5188_20130912b.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4960//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4960//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks for the patch. Nicholas! I haven't finished my review, some initial comments below, and more will come later.
          1.

          +  public static final String DFS_BLOCK_REPLICATOR_CLASSNAME_KEY = "dfs.block.replicator.classname";
          

          Shall we move it to DFSConfigKeys as other configurations?

          2.

          +    LOG.info("XXX numOfResults=" + numOfResults + ", writer=" + writer
          +        + ", excludedNodes=" + excludedNodes);
               // Keep a copy of original excludedNodes
          -    final HashMap<Node, Node> oldExcludedNodes = avoidStaleNodes ? 
          +    final Map<Node, Node> oldExcludedNodes = avoidStaleNodes ? 
                   new HashMap<Node, Node>(excludedNodes) : null;
               try {
                 if (numOfResults == 0) {
                   writer = chooseLocalNode(writer, excludedNodes, blocksize,
                       maxNodesPerRack, results, avoidStaleNodes);
          +        LOG.info("XXX2 numOfResults=" + numOfResults + ", writer=" + writer);
                   if (--numOfReplicas == 0) {
                     return writer;
                   }
          

          XXX should be replaced with something else?

          3.

            @Override
             public DatanodeDescriptor[] chooseTarget(String srcPath,
                                               int numOfReplicas,
                                               DatanodeDescriptor writer,
                                               List<DatanodeDescriptor> chosenNodes,
          -                                    long blocksize) {
          -    return chooseTarget(numOfReplicas, writer, chosenNodes, false,
          -        null, blocksize);
          -  }
          -
          -  @Override
          -  public DatanodeDescriptor[] chooseTarget(String srcPath,
          -                                    int numOfReplicas,
          -                                    DatanodeDescriptor writer,
          -                                    List<DatanodeDescriptor> chosenNodes,
                                               boolean returnChosenNodes,
          -                                    HashMap<Node, Node> excludedNodes,
          +                                    Map<Node, Node> excludedNodes,
                                               long blocksize) {
               return chooseTarget(numOfReplicas, writer, chosenNodes, returnChosenNodes,
                   excludedNodes, blocksize);
             }
          

          We may consider to remove chooseTarget(String srcPatch, ...) completely as srcPath is not used actually. Isn't it?

          4. I am thinking to refactor chooseTarget(..., DatanodeDescriptor writer, ...) to chooseTarget (..., node writer, ...) as the only important property of writer is to identify other nodes' location relationship so more generic one could be better. It may helps to cover the case that client node is not a Datanode also. Does it make sense?

          Btw, the JIRA's number sounds lucky!

          Show
          Junping Du added a comment - Thanks for the patch. Nicholas! I haven't finished my review, some initial comments below, and more will come later. 1. + public static final String DFS_BLOCK_REPLICATOR_CLASSNAME_KEY = "dfs.block.replicator.classname" ; Shall we move it to DFSConfigKeys as other configurations? 2. + LOG.info( "XXX numOfResults=" + numOfResults + ", writer=" + writer + + ", excludedNodes=" + excludedNodes); // Keep a copy of original excludedNodes - final HashMap<Node, Node> oldExcludedNodes = avoidStaleNodes ? + final Map<Node, Node> oldExcludedNodes = avoidStaleNodes ? new HashMap<Node, Node>(excludedNodes) : null ; try { if (numOfResults == 0) { writer = chooseLocalNode(writer, excludedNodes, blocksize, maxNodesPerRack, results, avoidStaleNodes); + LOG.info( "XXX2 numOfResults=" + numOfResults + ", writer=" + writer); if (--numOfReplicas == 0) { return writer; } XXX should be replaced with something else? 3. @Override public DatanodeDescriptor[] chooseTarget( String srcPath, int numOfReplicas, DatanodeDescriptor writer, List<DatanodeDescriptor> chosenNodes, - long blocksize) { - return chooseTarget(numOfReplicas, writer, chosenNodes, false , - null , blocksize); - } - - @Override - public DatanodeDescriptor[] chooseTarget( String srcPath, - int numOfReplicas, - DatanodeDescriptor writer, - List<DatanodeDescriptor> chosenNodes, boolean returnChosenNodes, - HashMap<Node, Node> excludedNodes, + Map<Node, Node> excludedNodes, long blocksize) { return chooseTarget(numOfReplicas, writer, chosenNodes, returnChosenNodes, excludedNodes, blocksize); } We may consider to remove chooseTarget(String srcPatch, ...) completely as srcPath is not used actually. Isn't it? 4. I am thinking to refactor chooseTarget(..., DatanodeDescriptor writer, ...) to chooseTarget (..., node writer, ...) as the only important property of writer is to identify other nodes' location relationship so more generic one could be better. It may helps to cover the case that client node is not a Datanode also. Does it make sense? Btw, the JIRA's number sounds lucky!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Junping, thanks for reviewing the patch.

          1. Sure, let's move it.
          2. Oops, forgot to remove the debug message.
          3. srcPath is useful so that BlockPlacementPolicy can possibly use different policies for different files. The BlockPlacementPolicyRaid was using it so that some files are in raid and some files are not.
          4. I tried to change it but it is quite involved. Let's change it separately as the patch is already not small.

          Here is a new patch: h5188_20130914.patch

          Show
          Tsz Wo Nicholas Sze added a comment - Junping, thanks for reviewing the patch. 1. Sure, let's move it. 2. Oops, forgot to remove the debug message. 3. srcPath is useful so that BlockPlacementPolicy can possibly use different policies for different files. The BlockPlacementPolicyRaid was using it so that some files are in raid and some files are not. 4. I tried to change it but it is quite involved. Let's change it separately as the patch is already not small. Here is a new patch: h5188_20130914.patch
          Hide
          Junping Du added a comment -

          Thanks for quickly response, Nicholas!
          3. Agree. It may be helpful for RAID cases. Just curious on where the code of BlockPlaceementPolicyRaid going? I remember we used to have that. Am I missing anything here?
          4. Sure. Just filed HDFS-5207 to address this.
          I will review your new patch soon. Thx!

          Show
          Junping Du added a comment - Thanks for quickly response, Nicholas! 3. Agree. It may be helpful for RAID cases. Just curious on where the code of BlockPlaceementPolicyRaid going? I remember we used to have that. Am I missing anything here? 4. Sure. Just filed HDFS-5207 to address this. I will review your new patch soon. Thx!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603176/h5188_20130914.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4973//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4973//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603176/h5188_20130914.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4973//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4973//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks for nice work, Nicholas! The patch looks good to me. Only a minor comment here: I see EMPTY_ARRAY defined twice here, once in DatanodeDescriptor and the other in DatanodeInfo, can we leave DatanodeDescriptor.EMPTY_ARRAY only?
          I have no further comments any more. Given this is a patch for trunk, may be you need a committer's +1?

          Show
          Junping Du added a comment - Thanks for nice work, Nicholas! The patch looks good to me. Only a minor comment here: I see EMPTY_ARRAY defined twice here, once in DatanodeDescriptor and the other in DatanodeInfo, can we leave DatanodeDescriptor.EMPTY_ARRAY only? I have no further comments any more. Given this is a patch for trunk, may be you need a committer's +1?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Junping for reviewing the patch.

          h5188_20130915.patch:

          • removes DatanodeInfo.EMPTY_ARRAY;
          • always creates debugLoggingBuilder like before for supporting changing log level at runtime.

          For simple JIRAs, non-committer can review and +1 on patches. Although the patch here is not small, I think this one is a simple JIRA since all changes are straightforward. If you are uncomfortable, I could find someone the review this.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Junping for reviewing the patch. h5188_20130915.patch: removes DatanodeInfo.EMPTY_ARRAY; always creates debugLoggingBuilder like before for supporting changing log level at runtime. For simple JIRAs, non-committer can review and +1 on patches. Although the patch here is not small, I think this one is a simple JIRA since all changes are straightforward. If you are uncomfortable, I could find someone the review this.
          Hide
          Junping Du added a comment -

          For simple JIRAs, non-committer can review and +1 on patches. Although the patch here is not small, I think this one is a simple JIRA since all changes are straightforward. If you are uncomfortable, I could find someone the review this.

          Actually, I am confident on code changes here but just not sure the process. Thanks for explanation, Nicholas! Agree that this is not a complex patch. +1. Please feel free to go ahead and commit it.

          Show
          Junping Du added a comment - For simple JIRAs, non-committer can review and +1 on patches. Although the patch here is not small, I think this one is a simple JIRA since all changes are straightforward. If you are uncomfortable, I could find someone the review this. Actually, I am confident on code changes here but just not sure the process. Thanks for explanation, Nicholas! Agree that this is not a complex patch. +1. Please feel free to go ahead and commit it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4416 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4416/)
          HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4416 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4416/ ) HDFS-5188 . In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603217/h5188_20130915.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4975//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4975//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603217/h5188_20130915.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4975//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4975//console This message is automatically generated.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #333 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/333/)
          HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #333 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/333/ ) HDFS-5188 . In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1523/)
          HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1523 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1523/ ) HDFS-5188 . In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1549 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1549/)
          HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1549 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1549/ ) HDFS-5188 . In BlockPlacementPolicy, reduce the number of chooseTarget(..) methods; replace HashMap with Map in parameter declarations and cleanup some related code. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523400 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Hide
          Junping Du added a comment -

          Hi Nicholas, one related JIRA of HDFS-5207 (to fix type of writer) is patch available, would you help to take a look at it? Thx!

          Show
          Junping Du added a comment - Hi Nicholas, one related JIRA of HDFS-5207 (to fix type of writer) is patch available, would you help to take a look at it? Thx!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Sure, will review your patch. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - Sure, will review your patch. Thanks.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development