Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10910

HDFS Erasure Coding doc should state its currently supported erasure coding policies

    Details

      Description

      While HDFS Erasure Coding doc states a variety of possible combinations of algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be documented.

      1. HDFS-10910.001.patch
        2 kB
        Yiqun Lin
      2. HDFS-10910.002.patch
        2 kB
        Yiqun Lin
      3. HDFS-10910.003.patch
        2 kB
        Yiqun Lin

        Activity

        Hide
        linyiqun Yiqun Lin added a comment - - edited

        Attach a initial patch for this. I found that some sequence number not showed right in hadoop documentaion page, I also make a fix in my patch.

        Show
        linyiqun Yiqun Lin added a comment - - edited Attach a initial patch for this. I found that some sequence number not showed right in hadoop documentaion page, I also make a fix in my patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 54s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvnsite 0m 49s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        9m 20s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10910
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830621/HDFS-10910.001.patch
        Optional Tests asflicense mvnsite
        uname Linux 17aaf6507875 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6437ba1
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16896/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 54s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvnsite 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 9m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10910 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830621/HDFS-10910.001.patch Optional Tests asflicense mvnsite uname Linux 17aaf6507875 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6437ba1 modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16896/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        It's great to add currently supported erasure coding policies in the doc. I would suggest,
        1. Since default cell size is mentioned, it's better to add the default cell size number in the doc.
        2. Default schema is "RS_6_3_SCHEMA".
        3. A policy has a schema and the stripping cell size. It seems schema name is used as the policy name when we talk about currently supported policies.

        Show
        Sammi SammiChen added a comment - It's great to add currently supported erasure coding policies in the doc. I would suggest, 1. Since default cell size is mentioned, it's better to add the default cell size number in the doc. 2. Default schema is "RS_6_3_SCHEMA". 3. A policy has a schema and the stripping cell size. It seems schema name is used as the policy name when we talk about currently supported policies.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks SammiChen for the comments. Upload a new patch to address your comments.

        Show
        linyiqun Yiqun Lin added a comment - Thanks SammiChen for the comments. Upload a new patch to address your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 8m 37s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvnsite 1m 1s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        11m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10910
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830655/HDFS-10910.002.patch
        Optional Tests asflicense mvnsite
        uname Linux 41d7d55f0b5d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 03f519a
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16901/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 8m 37s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvnsite 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 11m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10910 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830655/HDFS-10910.002.patch Optional Tests asflicense mvnsite uname Linux 41d7d55f0b5d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 03f519a modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16901/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        Thanks Yiqun for update the patch. The v2 patch seems good to me.

        Show
        Sammi SammiChen added a comment - Thanks Yiqun for update the patch. The v2 patch seems good to me.
        Hide
        jojochuang Wei-Chiu Chuang added a comment - - edited

        Thanks a lot Yiqun Lin for submitting the patch and thanks SammiChen for the first review.
        I think the patch is mostly good, just one nit for clarity:

        The current codec algorithms support three policies:

        To be more specific, you may say "There are three policies currently being supported". Because a policy constitutes a schema (=coding algorithm + data unit and parity unit) and a cell size.

        Show
        jojochuang Wei-Chiu Chuang added a comment - - edited Thanks a lot Yiqun Lin for submitting the patch and thanks SammiChen for the first review. I think the patch is mostly good, just one nit for clarity: The current codec algorithms support three policies: To be more specific, you may say "There are three policies currently being supported". Because a policy constitutes a schema (=coding algorithm + data unit and parity unit) and a cell size.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Actually, to be accurate, we should not just spell out the schema name. Instead, the policy names are RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This is how you specify ec policy using hdfs erasurecode -setPolicy -p

        Show
        jojochuang Wei-Chiu Chuang added a comment - Actually, to be accurate, we should not just spell out the schema name. Instead, the policy names are RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This is how you specify ec policy using hdfs erasurecode -setPolicy -p
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Wei-Chiu Chuang for the reiview and comments. Attach a new patch to address your comments.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Wei-Chiu Chuang for the reiview and comments. Attach a new patch to address your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 51s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvnsite 0m 57s the patch passed
        +1 whitespace 0m 1s The patch has no whitespace issues.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        9m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10910
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831045/HDFS-10910.003.patch
        Optional Tests asflicense mvnsite
        uname Linux 77dd29973452 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 10be459
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16940/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 51s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvnsite 0m 57s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 9m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10910 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831045/HDFS-10910.003.patch Optional Tests asflicense mvnsite uname Linux 77dd29973452 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 10be459 modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16940/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        +1

        Show
        jojochuang Wei-Chiu Chuang added a comment - +1
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the patch to trunk. Thanks Yiqun Lin for contributing the patch and SammiChen for the review!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the patch to trunk. Thanks Yiqun Lin for contributing the patch and SammiChen for the review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10521/)
        HDFS-10910. HDFS Erasure Coding doc should state its currently supported (weichiu: rev ee33a02234511ac69c1e491fd38490a141ec907e)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10521 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10521/ ) HDFS-10910 . HDFS Erasure Coding doc should state its currently supported (weichiu: rev ee33a02234511ac69c1e491fd38490a141ec907e) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Wei-Chiu Chuang for the commit!

        Show
        linyiqun Yiqun Lin added a comment - Thanks Wei-Chiu Chuang for the commit!

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development