Hadoop Common
  1. Hadoop Common
  2. HADOOP-5734

HDFS architecture documentation describes outdated placement policy

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The "Replica Placement: The First Baby Steps" section of HDFS architecture document states:

      "...
      For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack. This policy cuts the inter-rack write traffic which generally improves write performance.
      ..."

      However, according to the ReplicationTargetChooser.chooseTarger()'s code the actual logic is to put the second replica on a different rack as well as the third replica. So you have two replicas located on a different nodes of remote rack and one (initial replica) on the local rack's node. Thus, the sentence should say something like this:

      "For the common case, when the replication factor is three, HDFS's placement policy is to put one replica on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the same remote rack. This policy cuts the inter-rack write traffic which generally improves write performance."

      1. HADOOP-5734.patch
        4 kB
        Konstantin Boudnik

        Activity

        Konstantin Boudnik logged work - 23/Apr/09 16:42
        • Time Spent:
          2h
           
          <No comment>
        Konstantin Boudnik created issue -
        Hide
        Konstantin Boudnik added a comment - - edited

        The fix is clearly a minor modification of the placement logic's description.
        It doesn't affect any code but the documentation template and the JavaDoc of the ReplicationTargetChooser class, which is invisible anywhere but in the source code of the class.

        However, as I'm a very new to the project I'd appreciate if someone can make a quick review of at the patch to the internal javadoc to make sure it is "accurate, sufficient for future maintainability".

        Show
        Konstantin Boudnik added a comment - - edited The fix is clearly a minor modification of the placement logic's description. It doesn't affect any code but the documentation template and the JavaDoc of the ReplicationTargetChooser class, which is invisible anywhere but in the source code of the class. However, as I'm a very new to the project I'd appreciate if someone can make a quick review of at the patch to the internal javadoc to make sure it is "accurate, sufficient for future maintainability".
        Konstantin Boudnik made changes -
        Field Original Value New Value
        Attachment HADOOP-5734.patch [ 12406316 ]
        Hide
        Hairong Kuang added a comment -

        +1. The change looks good to me.

        Show
        Hairong Kuang added a comment - +1. The change looks good to me.
        Hairong Kuang made changes -
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12313563 ]
        Konstantin Boudnik made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12406316/HADOOP-5734.patch
        against trunk revision 769174.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12406316/HADOOP-5734.patch against trunk revision 769174. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/253/console This message is automatically generated.
        Konstantin Boudnik made changes -
        Remaining Estimate 0h [ 0 ]
        Time Spent 2h [ 7200 ]
        Konstantin Boudnik made changes -
        Assignee Konstantin Boudnik [ cos ]
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Congratulations, Konstantin!

        Show
        Konstantin Shvachko added a comment - I just committed this. Congratulations, Konstantin!
        Konstantin Shvachko made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Konstantin Shvachko made changes -
        Component/s dfs [ 12310710 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/)
        . Correct block placement policy description in HDFS Design document. Contributed by Konstantin Boudnik.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/ ) . Correct block placement policy description in HDFS Design document. Contributed by Konstantin Boudnik.
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Konstantin Boudnik
            Reporter:
            Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 2h
              2h

                Development