Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11833

HDFS architecture documentation describes outdated placement policy

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Hello

      I've noticed inconsistencies in the default block placement policy between what one can read on several websites and what is stated in the official documentation.

      After digging, I've found out that the proper default placement policy is :

       * The replica placement strategy is that if the writer is on a datanode,
       * the 1st replica is placed on the local machine, 
       * otherwise a random datanode. The 2nd replica is placed on a datanode
       * that is on a different rack. The 3rd replica is placed on a datanode
       * which is on a different node of the rack as the second replica.
       

      source

      HADOOP-5734 has been opened regarding this mistake several years ago and has eventually been merged but unfortunately this fix has been overwritten by Alejandro Abdelnur in SVN commit.

      dud

      1. HADOOP-14411.002.patch
        3 kB
        Chen Liang
      2. HADOOP-14411.001.patch
        3 kB
        Chen Liang

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11736 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11736/)
        HDFS-11833. HDFS architecture documentation descibes outdated placement (aajisaka: rev 1d1c52b42feae5a4271ef4b771d0d8de43e83c15)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11736 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11736/ ) HDFS-11833 . HDFS architecture documentation descibes outdated placement (aajisaka: rev 1d1c52b42feae5a4271ef4b771d0d8de43e83c15) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed to trunk, branch-2, and branch-2.8. Thanks Chen Liang for the contribution.

        Show
        ajisakaa Akira Ajisaka added a comment - Committed to trunk, branch-2, and branch-2.8. Thanks Chen Liang for the contribution.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        LGTM, +1

        Show
        ajisakaa Akira Ajisaka added a comment - LGTM, +1
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 16m 27s trunk passed
        +1 mvnsite 1m 6s trunk passed
        +1 mvnsite 1m 2s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        19m 45s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14411
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867681/HADOOP-14411.002.patch
        Optional Tests asflicense mvnsite
        uname Linux d7fcdc9c5629 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0d5c8ed
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12298/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 16m 27s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 19m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14411 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867681/HADOOP-14411.002.patch Optional Tests asflicense mvnsite uname Linux d7fcdc9c5629 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0d5c8ed modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12298/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vagarychen Chen Liang added a comment -

        Thanks Akira Ajisaka for the comments! updated in v002 patch.

        Show
        vagarychen Chen Liang added a comment - Thanks Akira Ajisaka for the comments! updated in v002 patch.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks dud for the report and thanks Chen Liang for providing the patch. Would you update the sentence for the 1st replica as well?

        Show
        ajisakaa Akira Ajisaka added a comment - Thanks dud for the report and thanks Chen Liang for providing the patch. Would you update the sentence for the 1st replica as well?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 14m 8s trunk passed
        +1 mvnsite 0m 54s trunk passed
        +1 mvnsite 0m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        16m 54s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14411
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867634/HADOOP-14411.001.patch
        Optional Tests asflicense mvnsite
        uname Linux 719c556c234e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 09f28da
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12295/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 14m 8s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvnsite 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 16m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14411 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867634/HADOOP-14411.001.patch Optional Tests asflicense mvnsite uname Linux 719c556c234e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 09f28da modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12295/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vagarychen Chen Liang added a comment -

        Thanks dud for the catch and related links! Upload v001 patch to fix it, basically following HADOOP-5734.

        Show
        vagarychen Chen Liang added a comment - Thanks dud for the catch and related links! Upload v001 patch to fix it, basically following HADOOP-5734 .

          People

          • Assignee:
            vagarychen Chen Liang
            Reporter:
            dud dud
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development