Hadoop Common
  1. Hadoop Common
  2. HADOOP-5734

HDFS architecture documentation describes outdated placement policy

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The "Replica Placement: The First Baby Steps" section of HDFS architecture document states:

      "...
      For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack. This policy cuts the inter-rack write traffic which generally improves write performance.
      ..."

      However, according to the ReplicationTargetChooser.chooseTarger()'s code the actual logic is to put the second replica on a different rack as well as the third replica. So you have two replicas located on a different nodes of remote rack and one (initial replica) on the local rack's node. Thus, the sentence should say something like this:

      "For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same remote rack. This policy cuts the inter-rack write traffic which generally improves write performance."

      1. HADOOP-5734.patch
        4 kB
        Konstantin Boudnik

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        3d 23h 55m 1 Konstantin Boudnik 28/Apr/09 01:48
        Patch Available Patch Available Resolved Resolved
        1d 57m 1 Konstantin Shvachko 29/Apr/09 02:45
        Resolved Resolved Closed Closed
        482d 18h 52m 1 Tom White 24/Aug/10 21:37
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/)
        . Correct block placement policy description in HDFS Design document. Contributed by Konstantin Boudnik.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/ ) . Correct block placement policy description in HDFS Design document. Contributed by Konstantin Boudnik.
        Konstantin Shvachko made changes -
        Component/s dfs [ 12310710 ]
        Konstantin Shvachko made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Congratulations, Konstantin!

        Show
        Konstantin Shvachko added a comment - I just committed this. Congratulations, Konstantin!
        Konstantin Boudnik made changes -
        Assignee Konstantin Boudnik [ cos ]
        Konstantin Boudnik made changes -
        Remaining Estimate 0h [ 0 ]
        Time Spent 2h [ 7200 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12406316/HADOOP-5734.patch
        against trunk revision 769174.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12406316/HADOOP-5734.patch against trunk revision 769174. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/console This message is automatically generated.
        Konstantin Boudnik made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hairong Kuang made changes -
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12313563 ]
        Hide
        Hairong Kuang added a comment -

        +1. The change looks good to me.

        Show
        Hairong Kuang added a comment - +1. The change looks good to me.
        Konstantin Boudnik made changes -
        Field Original Value New Value
        Attachment HADOOP-5734.patch [ 12406316 ]
        Hide
        Konstantin Boudnik added a comment - - edited

        The fix is clearly a minor modification of the placement logic's description.
        It doesn't affect any code but the documentation template and the JavaDoc of the ReplicationTargetChooser class, which is invisible anywhere but in the source code of the class.

        However, as I'm a very new to the project I'd appreciate if someone can make a quick review of at the patch to the internal javadoc to make sure it is "accurate, sufficient for future maintainability".

        Show
        Konstantin Boudnik added a comment - - edited The fix is clearly a minor modification of the placement logic's description. It doesn't affect any code but the documentation template and the JavaDoc of the ReplicationTargetChooser class, which is invisible anywhere but in the source code of the class. However, as I'm a very new to the project I'd appreciate if someone can make a quick review of at the patch to the internal javadoc to make sure it is "accurate, sufficient for future maintainability".
        Konstantin Boudnik created issue -
        Konstantin Boudnik logged work - 23/Apr/09 16:42
        • Time Spent:
          2h
           
          <No comment>

          People

          • Assignee:
            Konstantin Boudnik
            Reporter:
            Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 2h
              2h

                Development