Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      BlockPlacementPolicy supports chosen nodes in some of the chooseTarget(..) methods. The chosen nodes will be removed from the output array. For adding new datanodes to an existing pipeline, it is useful to keep the chosen nodes in the output array.

      1. h1629_20110217.patch
        11 kB
        Tsz Wo Nicholas Sze
      2. h1629_20110216.patch
        11 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Tsz Wo Nicholas Sze created issue -
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Component/s data-node [ 12312927 ]
          Component/s hdfs client [ 12312928 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1629_20110216.patch: added a new method.

          Show
          Tsz Wo Nicholas Sze added a comment - h1629_20110216.patch: added a new method.
          Tsz Wo Nicholas Sze made changes -
          Attachment h1629_20110216.patch [ 12471163 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471163/h1629_20110216.patch
          against trunk revision 1071023.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.namenode.TestReplicationPolicy
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471163/h1629_20110216.patch against trunk revision 1071023. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.namenode.TestReplicationPolicy org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/166//console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          +1, looks fine to me.

          Show
          dhruba borthakur added a comment - +1, looks fine to me.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h1629_20110217.patch: fixed the new unit test.

          Show
          Tsz Wo Nicholas Sze added a comment - h1629_20110217.patch: fixed the new unit test.
          Tsz Wo Nicholas Sze made changes -
          Attachment h1629_20110217.patch [ 12471247 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Dhruba for review the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Dhruba for review the patch.
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s name-node [ 12312926 ]
          Component/s data-node [ 12312927 ]
          Component/s hdfs client [ 12312928 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471247/h1629_20110217.patch
          against trunk revision 1071023.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471247/h1629_20110217.patch against trunk revision 1071023. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/169//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315571 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #544 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/544/)
          HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen nodes in the output array.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #544 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/544/ ) HDFS-1629 . Add a method to BlockPlacementPolicy for keeping the chosen nodes in the output array.
          Hide
          Todd Lipcon added a comment -

          This patch seems to have broken the MR contrib build:

          [javac] /data/1/scratch/patchqueue/patch-worker-32764/patch_62/svnrepo/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyRaid.java:50: org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid is not abstract and does not override abstract method chooseTarget(java.lang.String,int,org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor,java.util.List<org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor>,boolean,java.util.HashMap<org.apache.hadoop.net.Node,org.apache.hadoop.net.Node>,long) in org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy
          [javac] public class BlockPlacementPolicyRaid extends BlockPlacementPolicy {
          [javac] ^

          as well as a couple other errors there. Should we revert this until we have a fix ready for MR?

          Show
          Todd Lipcon added a comment - This patch seems to have broken the MR contrib build: [javac] /data/1/scratch/patchqueue/patch-worker-32764/patch_62/svnrepo/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyRaid.java:50: org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyRaid is not abstract and does not override abstract method chooseTarget(java.lang.String,int,org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor,java.util.List<org.apache.hadoop.hdfs.server.namenode.DatanodeDescriptor>,boolean,java.util.HashMap<org.apache.hadoop.net.Node,org.apache.hadoop.net.Node>,long) in org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy [javac] public class BlockPlacementPolicyRaid extends BlockPlacementPolicy { [javac] ^ as well as a couple other errors there. Should we revert this until we have a fix ready for MR?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Let me take a look. The fix seems obvious.

          Show
          Tsz Wo Nicholas Sze added a comment - Let me take a look. The fix seems obvious.
          Todd Lipcon made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to MAPREDUCE-2334 [ MAPREDUCE-2334 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Created MAPREDUCE-2334 to update BlockPlacementPolicyRaid.

          Show
          Tsz Wo Nicholas Sze added a comment - Created MAPREDUCE-2334 to update BlockPlacementPolicyRaid .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          MAPREDUCE-2334 is committed. Feel free to reopen this if you find any problem.

          Show
          Tsz Wo Nicholas Sze added a comment - MAPREDUCE-2334 is committed. Feel free to reopen this if you find any problem.
          Tsz Wo Nicholas Sze made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          18h 13m 1 Tsz Wo Nicholas Sze 17/Feb/11 02:44
          Open Open Patch Available Patch Available
          54m 52s 2 Tsz Wo Nicholas Sze 17/Feb/11 02:45
          Patch Available Patch Available Resolved Resolved
          4h 21m 1 Tsz Wo Nicholas Sze 17/Feb/11 07:06
          Resolved Resolved Reopened Reopened
          53m 5s 1 Todd Lipcon 17/Feb/11 07:59
          Reopened Reopened Resolved Resolved
          1d 8h 6m 1 Tsz Wo Nicholas Sze 18/Feb/11 16:06

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development