Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockPlacementPolicy supports chosen nodes in some of the chooseTarget(..) methods. The chosen nodes will be removed from the output array. For adding new datanodes to an existing pipeline, it is useful to keep the chosen nodes in the output array.

      1. h1629_20110217.patch
        11 kB
        Tsz Wo Nicholas Sze
      2. h1629_20110216.patch
        11 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1629_20110216.patch: added a new method.

          Show
          Tsz Wo Nicholas Sze added a comment - h1629_20110216.patch: added a new method.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471163/h1629_20110216.patch
          against trunk revision 1071023.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.namenode.TestReplicationPolicy
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471163/h1629_20110216.patch against trunk revision 1071023. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.namenode.TestReplicationPolicy org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          +1, looks fine to me.

          Show
          dhruba borthakur added a comment - +1, looks fine to me.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1629_20110217.patch: fixed the new unit test.

          Show
          Tsz Wo Nicholas Sze added a comment - h1629_20110217.patch: fixed the new unit test.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Dhruba for review the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Dhruba for review the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471247/h1629_20110217.patch
          against trunk revision 1071023.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471247/h1629_20110217.patch against trunk revision 1071023. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #544 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/544/)
          HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen nodes in the output array.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #544 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/544/ ) HDFS-1629 . Add a method to BlockPlacementPolicy for keeping the chosen nodes in the output array.
          Hide
          Todd Lipcon added a comment -

          This patch seems to have broken the MR contrib build:

          [javac] /data/1/scratch/patchqueue/patch-worker-32764/patch_62/svnrepo/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyRaid.java:50: org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid is not abstract and does not override abstract method chooseTarget(java.lang.String,int,org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor,java.util.List<org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor>,boolean,java.util.HashMap<org.apache.hadoop.net.Node,org.apache.hadoop.net.Node>,long) in org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy
          [javac] public class BlockPlacementPolicyRaid extends BlockPlacementPolicy {
          [javac] ^

          as well as a couple other errors there. Should we revert this until we have a fix ready for MR?

          Show
          Todd Lipcon added a comment - This patch seems to have broken the MR contrib build: [javac] /data/1/scratch/patchqueue/patch-worker-32764/patch_62/svnrepo/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyRaid.java:50: org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid is not abstract and does not override abstract method chooseTarget(java.lang.String,int,org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor,java.util.List<org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor>,boolean,java.util.HashMap<org.apache.hadoop.net.Node,org.apache.hadoop.net.Node>,long) in org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy [javac] public class BlockPlacementPolicyRaid extends BlockPlacementPolicy { [javac] ^ as well as a couple other errors there. Should we revert this until we have a fix ready for MR?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Let me take a look. The fix seems obvious.

          Show
          Tsz Wo Nicholas Sze added a comment - Let me take a look. The fix seems obvious.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Created MAPREDUCE-2334 to update BlockPlacementPolicyRaid.

          Show
          Tsz Wo Nicholas Sze added a comment - Created MAPREDUCE-2334 to update BlockPlacementPolicyRaid .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          MAPREDUCE-2334 is committed. Feel free to reopen this if you find any problem.

          Show
          Tsz Wo Nicholas Sze added a comment - MAPREDUCE-2334 is committed. Feel free to reopen this if you find any problem.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development