Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11033

Add documents for native raw erasure coder in XOR codes

    Details

      Description

      Add document for native raw erasure coder in XOR codes.

      1. HDFS-11033-v2.patch
        4 kB
        SammiChen
      2. HDFS-11033-v1.patch
        4 kB
        SammiChen

        Activity

        Hide
        Sammi SammiChen added a comment -

        Initial patch

        Show
        Sammi SammiChen added a comment - Initial patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 1s trunk passed
        +1 compile 7m 14s trunk passed
        +1 mvnsite 1m 55s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 javadoc 1m 24s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 7m 7s the patch passed
        +1 javac 7m 7s the patch passed
        +1 mvnsite 1m 44s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 1m 21s the patch passed
        +1 unit 7m 28s hadoop-common in the patch passed.
        -1 unit 57m 34s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        96m 45s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11033
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834125/HDFS-11033-v1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 2b7cfa9f5eeb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5573e6
        Default Java 1.8.0_101
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17217/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17217/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 1s trunk passed +1 compile 7m 14s trunk passed +1 mvnsite 1m 55s trunk passed +1 mvneclipse 0m 25s trunk passed +1 javadoc 1m 24s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 7m 7s the patch passed +1 javac 7m 7s the patch passed +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 1m 21s the patch passed +1 unit 7m 28s hadoop-common in the patch passed. -1 unit 57m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 96m 45s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11033 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834125/HDFS-11033-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 2b7cfa9f5eeb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5573e6 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17217/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17217/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Looks mostly good to me.

        For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the encoding and decoding calculation.

        You may want to be more specific by saying "to improve the performance of codec."

        Show
        jojochuang Wei-Chiu Chuang added a comment - Looks mostly good to me. For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the encoding and decoding calculation. You may want to be more specific by saying "to improve the performance of codec."
        Hide
        Sammi SammiChen added a comment -

        Thanks so much Wei-Chiu for reviewing the patch! I have uploaded a new patch address the issue.

        Show
        Sammi SammiChen added a comment - Thanks so much Wei-Chiu for reviewing the patch! I have uploaded a new patch address the issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 2m 26s Maven dependency ordering for branch
        +1 mvninstall 8m 8s trunk passed
        +1 compile 8m 6s trunk passed
        +1 mvnsite 2m 1s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 javadoc 1m 26s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 41s the patch passed
        +1 compile 8m 0s the patch passed
        +1 javac 8m 0s the patch passed
        +1 mvnsite 1m 59s the patch passed
        +1 mvneclipse 0m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 1m 23s the patch passed
        +1 unit 8m 10s hadoop-common in the patch passed.
        -1 unit 68m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        114m 28s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11033
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834328/HDFS-11033-v2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 9b7fe0e2d764 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 73504b1
        Default Java 1.8.0_101
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17235/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17235/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17235/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 2m 26s Maven dependency ordering for branch +1 mvninstall 8m 8s trunk passed +1 compile 8m 6s trunk passed +1 mvnsite 2m 1s trunk passed +1 mvneclipse 0m 26s trunk passed +1 javadoc 1m 26s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 8m 0s the patch passed +1 javac 8m 0s the patch passed +1 mvnsite 1m 59s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 1m 23s the patch passed +1 unit 8m 10s hadoop-common in the patch passed. -1 unit 68m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 114m 28s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11033 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834328/HDFS-11033-v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 9b7fe0e2d764 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 73504b1 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17235/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17235/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17235/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the patch.
        The last patch looks good to me. Would any one else like to comment?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the patch. The last patch looks good to me. Would any one else like to comment?
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM too thanks Sammi and Wei-Chiu!

        Show
        andrew.wang Andrew Wang added a comment - LGTM too thanks Sammi and Wei-Chiu!
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committing v2 patch

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committing v2 patch
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks to SammiChen for contributing the patch and to Andrew Wang for +1!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks to SammiChen for contributing the patch and to Andrew Wang for +1!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10660 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10660/)
        HDFS-11033. Add documents for native raw erasure coder in XOR codes. (weichiu: rev 25438526edd6b3fef23daddd29facfca8f840913)

        • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10660 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10660/ ) HDFS-11033 . Add documents for native raw erasure coder in XOR codes. (weichiu: rev 25438526edd6b3fef23daddd29facfca8f840913) (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        Hide
        Sammi SammiChen added a comment -

        Thanks to Wei-Chiu Chuang and Andrew Wang for reviewing the patch and to Wei-Chiu Chuang also commiting the patch!

        Show
        Sammi SammiChen added a comment - Thanks to Wei-Chiu Chuang and Andrew Wang for reviewing the patch and to Wei-Chiu Chuang also commiting the patch!

          People

          • Assignee:
            Sammi SammiChen
            Reporter:
            Sammi SammiChen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development