Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ReplicationTargetChooser chooses targets by iteratively selecting random nodes first and then filtering good targets until the required number of targets are chosen. This code may require selecting random nodes multiple times, thus introducing multiple traversals of the given portion of the cluster map. This code can be improved by traversing the portion of the cluster map only once.

      1. blockPlace.patch
        4 kB
        Hairong Kuang
      2. blockPlace1.patch
        4 kB
        Hairong Kuang

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          2d 22h 24m 1 Hairong Kuang 06/Apr/09 21:51
          Open Open Patch Available Patch Available
          3d 1h 11m 2 Hairong Kuang 06/Apr/09 21:52
          Patch Available Patch Available Resolved Resolved
          20h 48m 1 Hairong Kuang 07/Apr/09 18:40
          Resolved Resolved Closed Closed
          504d 2h 55m 1 Tom White 24/Aug/10 21:36
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Hairong Kuang made changes -
          Link This issue relates to HADOOP-5638 [ HADOOP-5638 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hairong Kuang added a comment -

          -1 Eclipse classpath is caused by HADOOP-5518 not by this patch. The change is covered by existing test cases. There is no need of new tests.

          I've committed this!

          Show
          Hairong Kuang added a comment - -1 Eclipse classpath is caused by HADOOP-5518 not by this patch. The change is covered by existing test cases. There is no need of new tests. I've committed this!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12404772/blockPlace1.patch
          against trunk revision 762509.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12404772/blockPlace1.patch against trunk revision 762509. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/156/console This message is automatically generated.
          Hairong Kuang made changes -
          Attachment blockPlace1.patch [ 12404767 ]
          Hairong Kuang made changes -
          Attachment blockPlace1.patch [ 12404772 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Attachment blockPlace1.patch [ 12404767 ]
          Hide
          Hairong Kuang added a comment -

          This patch fixed a bug that caused some unit tests to fail.

          Show
          Hairong Kuang added a comment - This patch fixed a bug that caused some unit tests to fail.
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12404270/blockPlace.patch
          against trunk revision 761632.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12404270/blockPlace.patch against trunk revision 761632. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/113/console This message is automatically generated.
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hairong Kuang added a comment -

          I did an experiment in a dfs cluster with 3150 node. The cluster is full with no space to place any block. The trunk takes around 6.5s to declare failure in an effort to place a block to 2 nodes. With the patch, it takes around 2.1s to declare failure.

          Show
          Hairong Kuang added a comment - I did an experiment in a dfs cluster with 3150 node. The cluster is full with no space to place any block. The trunk takes around 6.5s to declare failure in an effort to place a block to 2 nodes. With the patch, it takes around 2.1s to declare failure.
          Hairong Kuang made changes -
          Field Original Value New Value
          Attachment blockPlace.patch [ 12404270 ]
          Hide
          Hairong Kuang added a comment -

          Here is the patch that made the suggested change.

          with the patch, both ReplicationTargetChooser#chooseRandom(int, String, List<Node>, long, int, List<DatanodeDescriptor>) and ReplicationTarget#chooseRandom(String, List<Node>, long, int, List<DatanodeDescriptor>) traverse every node in the given portion of the cluster map at most once in the worst case.

          Show
          Hairong Kuang added a comment - Here is the patch that made the suggested change. with the patch, both ReplicationTargetChooser#chooseRandom(int, String, List<Node>, long, int, List<DatanodeDescriptor>) and ReplicationTarget#chooseRandom(String, List<Node>, long, int, List<DatanodeDescriptor>) traverse every node in the given portion of the cluster map at most once in the worst case.
          Hairong Kuang created issue -

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development