Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      This provides a native implementation of XOR codec by leveraging Intel ISA-L library function to achieve a better performance.

      Description

      Raw XOR coder is utilized in Reed-Solomon erasure coder in an optimization to recover only one erased block which is in most often case. It can also be used in HitchHiker coder. Therefore a native implementation of it would be deserved for performance gain.

      1. HADOOP-11798-v1.patch
        23 kB
        Kai Zheng
      2. HADOOP-11798-v2.patch
        18 kB
        SammiChen
      3. HADOOP-11798-v3.patch
        28 kB
        SammiChen
      4. HADOOP-11798-v4.patch
        29 kB
        SammiChen
      5. HADOOP-11798-v5.patch
        28 kB
        SammiChen

        Activity

        Hide
        drankye Kai Zheng added a comment -

        To make it easy for the native part, I would start with this simpler one. It will help prepare for some basic setups for complex ones, like based on ISA-L.

        Show
        drankye Kai Zheng added a comment - To make it easy for the native part, I would start with this simpler one. It will help prepare for some basic setups for complex ones, like based on ISA-L.
        Hide
        drankye Kai Zheng added a comment -

        Uploaded the patch. Pending on HADOOP-11887 and HADOOP-11996.

        Show
        drankye Kai Zheng added a comment - Uploaded the patch. Pending on HADOOP-11887 and HADOOP-11996 .
        Hide
        andrew.wang Andrew Wang added a comment -

        I think this one is pretty important for XOR performance.

        Kai Zheng, related question, do we have an EC policy for XOR? This would be nice for small clusters, since right now the smallest stripe width is (3,2) which requires at least 5 racks. An XOR (2,1) would only require 3.

        Show
        andrew.wang Andrew Wang added a comment - I think this one is pretty important for XOR performance. Kai Zheng , related question, do we have an EC policy for XOR? This would be nice for small clusters, since right now the smallest stripe width is (3,2) which requires at least 5 racks. An XOR (2,1) would only require 3.
        Hide
        drankye Kai Zheng added a comment -

        Hi Andrew Wang,

        Good catch on this. It's good to make XOR codec work and we'll check to see if any gap exists. Will also resume the work here and update the patch. Thanks!

        Show
        drankye Kai Zheng added a comment - Hi Andrew Wang , Good catch on this. It's good to make XOR codec work and we'll check to see if any gap exists. Will also resume the work here and update the patch. Thanks!
        Hide
        Sammi SammiChen added a comment -

        Hi Andrew and Kai, I created JIRA HDFS-10994 to export XOR EC policy in "hdfs erasurecode" command. And will working on it. Thanks!

        Show
        Sammi SammiChen added a comment - Hi Andrew and Kai, I created JIRA HDFS-10994 to export XOR EC policy in "hdfs erasurecode" command. And will working on it. Thanks!
        Hide
        Sammi SammiChen added a comment -

        1. Patch rebase
        2. Add test cases for native XOR codec

        Show
        Sammi SammiChen added a comment - 1. Patch rebase 2. Add test cases for native XOR codec
        Hide
        drankye Kai Zheng added a comment -

        The Jenkins building doesn't trigger, probably because of the archived fix version. I can't get rid of it. Andrew Wang do you have any idea? Thanks.

        Show
        drankye Kai Zheng added a comment - The Jenkins building doesn't trigger, probably because of the archived fix version. I can't get rid of it. Andrew Wang do you have any idea? Thanks.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Kai, I manually triggered the build just now, you should have access too as a committer: https://builds.apache.org/job/PreCommit-HADOOP-Build/

        I think in this case though, the issue is that the JIRA is in "In Progress" state rather than "Patch Available". SammiChen could you "stop progress" and then "submit patch"? Only the JIRA assignee can "stop progress".

        Show
        andrew.wang Andrew Wang added a comment - Hi Kai, I manually triggered the build just now, you should have access too as a committer: https://builds.apache.org/job/PreCommit-HADOOP-Build/ I think in this case though, the issue is that the JIRA is in "In Progress" state rather than "Patch Available". SammiChen could you "stop progress" and then "submit patch"? Only the JIRA assignee can "stop progress".
        Hide
        drankye Kai Zheng added a comment -

        Thank you Andrew! I will bookmark the HADOOP Build link.

        Show
        drankye Kai Zheng added a comment - Thank you Andrew! I will bookmark the HADOOP Build link.
        Hide
        drankye Kai Zheng added a comment -

        Thank you SammiChen for the checking and working on that.

        Show
        drankye Kai Zheng added a comment - Thank you SammiChen for the checking and working on that.
        Hide
        Sammi SammiChen added a comment -

        I have followed "stop progress" and "submit patch". Hope we can have the Jenkins building soon.

        Show
        Sammi SammiChen added a comment - I have followed "stop progress" and "submit patch". Hope we can have the Jenkins building soon.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 30s trunk passed
        +1 compile 9m 6s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 17s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 38s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 9m 29s the patch passed
        +1 cc 9m 29s the patch passed
        +1 javac 9m 29s the patch passed
        -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        +1 mvnsite 1m 13s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 8m 25s hadoop-common in the patch passed.
        -1 asflicense 0m 21s The patch generated 2 ASF License warnings.
        48m 20s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-11798
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832652/HADOOP-11798-v2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle
        uname Linux 460e2a959ffc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b84c489
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 30s trunk passed +1 compile 9m 6s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 17s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 9m 29s the patch passed +1 cc 9m 29s the patch passed +1 javac 9m 29s the patch passed -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 8m 25s hadoop-common in the patch passed. -1 asflicense 0m 21s The patch generated 2 ASF License warnings. 48m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11798 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832652/HADOOP-11798-v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle uname Linux 460e2a959ffc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b84c489 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10738/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Hi SammiChen,

        Thanks for working on this.
        1. Looks like we need to add changes for Windows platform.
        2. Could we follow the pattern for the tests for RS raw coder to do the tests here? It can share the test cases between Java impl and native impl for the same codec.

        Thanks.

        Show
        drankye Kai Zheng added a comment - Hi SammiChen , Thanks for working on this. 1. Looks like we need to add changes for Windows platform. 2. Could we follow the pattern for the tests for RS raw coder to do the tests here? It can share the test cases between Java impl and native impl for the same codec. Thanks.
        Hide
        drankye Kai Zheng added a comment -

        Another idea is, better to add inter-operable tests for this native xor coder against the existing Java version.

        Show
        drankye Kai Zheng added a comment - Another idea is, better to add inter-operable tests for this native xor coder against the existing Java version.
        Hide
        Sammi SammiChen added a comment -

        Hi Kai, thanks for the suggestion. Sure, Will add the interoperatability test case.

        Show
        Sammi SammiChen added a comment - Hi Kai, thanks for the suggestion. Sure, Will add the interoperatability test case.
        Hide
        Sammi SammiChen added a comment - - edited

        1. Include native files in Windows build script
        2. add native XOR and java XOR inter-operable test case
        3. refactor native XOR and java XOR test case
        4. fix ASF issues

        Show
        Sammi SammiChen added a comment - - edited 1. Include native files in Windows build script 2. add native XOR and java XOR inter-operable test case 3. refactor native XOR and java XOR test case 4. fix ASF issues
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        +1 mvninstall 6m 37s trunk passed
        +1 compile 6m 47s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 18s trunk passed
        +1 javadoc 0m 43s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 6m 48s the patch passed
        +1 cc 6m 48s the patch passed
        +1 javac 6m 48s the patch passed
        -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 25s the patch passed
        +1 javadoc 0m 41s the patch passed
        +1 unit 8m 5s hadoop-common in the patch passed.
        -1 asflicense 0m 21s The patch generated 1 ASF License warnings.
        38m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-11798
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833328/HADOOP-11798-v3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle
        uname Linux d905536c6e03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dbe663d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 6m 48s the patch passed +1 cc 6m 48s the patch passed +1 javac 6m 48s the patch passed -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 25s the patch passed +1 javadoc 0m 41s the patch passed +1 unit 8m 5s hadoop-common in the patch passed. -1 asflicense 0m 21s The patch generated 1 ASF License warnings. 38m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11798 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833328/HADOOP-11798-v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle uname Linux d905536c6e03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbe663d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10785/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Sammi SammiChen added a comment -

        Fix ASF and style check issue

        Show
        Sammi SammiChen added a comment - Fix ASF and style check issue
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        +1 mvninstall 7m 28s trunk passed
        +1 compile 7m 8s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 18s trunk passed
        +1 javadoc 0m 42s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 7m 9s the patch passed
        +1 cc 7m 9s the patch passed
        +1 javac 7m 9s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 29s the patch passed
        +1 javadoc 0m 43s the patch passed
        +1 unit 8m 59s hadoop-common in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        40m 43s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-11798
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833344/HADOOP-11798-v4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle
        uname Linux b9da617a3995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dbe663d
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10787/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10787/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 7m 28s trunk passed +1 compile 7m 8s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 7m 9s the patch passed +1 cc 7m 9s the patch passed +1 javac 7m 9s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 29s the patch passed +1 javadoc 0m 43s the patch passed +1 unit 8m 59s hadoop-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 40m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11798 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833344/HADOOP-11798-v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle uname Linux b9da617a3995 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbe663d Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10787/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10787/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I think the latest patch generally looks to me. One nit is lack of documentation to switch between native v.s. Java-based codec, but that can go into a new jira.

        Show
        jojochuang Wei-Chiu Chuang added a comment - I think the latest patch generally looks to me. One nit is lack of documentation to switch between native v.s. Java-based codec, but that can go into a new jira.
        Hide
        Sammi SammiChen added a comment -

        Hi Wei-Chiu, thank you so much for reviewing the code! Talking about documentation, one thing comes to my mind is we can add notes in HDFSErasureCoding.md, use different values of key "io.erasurecode.codec.xor.rawcoder" to switch between native v.s. Java-based codec.
        If only HDFSErasureCoding.md need to be updated, I will upload a new patch including this document update later.

        Show
        Sammi SammiChen added a comment - Hi Wei-Chiu, thank you so much for reviewing the code! Talking about documentation, one thing comes to my mind is we can add notes in HDFSErasureCoding.md, use different values of key "io.erasurecode.codec.xor.rawcoder" to switch between native v.s. Java-based codec. If only HDFSErasureCoding.md need to be updated, I will upload a new patch including this document update later.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        In addition to HDFSErasureCoding.md you should also add the configuration key/value/description to core-default.xml.

        You may also consider adding the same information for io.erasurecode.codec.rs-default.rawcoder and io.erasurecode.codec.rs-legacy.rawcoder

        Show
        jojochuang Wei-Chiu Chuang added a comment - In addition to HDFSErasureCoding.md you should also add the configuration key/value/description to core-default.xml. You may also consider adding the same information for io.erasurecode.codec.rs-default.rawcoder and io.erasurecode.codec.rs-legacy.rawcoder
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        In addition to HDFSErasureCoding.md you should also add the configuration key/value/description to core-default.xml.

        You may also consider adding the same information for io.erasurecode.codec.rs-default.rawcoder and io.erasurecode.codec.rs-legacy.rawcoder

        Show
        jojochuang Wei-Chiu Chuang added a comment - In addition to HDFSErasureCoding.md you should also add the configuration key/value/description to core-default.xml. You may also consider adding the same information for io.erasurecode.codec.rs-default.rawcoder and io.erasurecode.codec.rs-legacy.rawcoder
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        BTW I only looked at the Java code. Someone (Kai Zheng?) should take a look at the native implementation. Thanks.

        Show
        jojochuang Wei-Chiu Chuang added a comment - BTW I only looked at the Java code. Someone ( Kai Zheng ?) should take a look at the native implementation. Thanks.
        Hide
        drankye Kai Zheng added a comment -

        The native code looks good overall. One minor is, please clean up the unnecessary #include statements in the *.c files.

        Show
        drankye Kai Zheng added a comment - The native code looks good overall. One minor is, please clean up the unnecessary #include statements in the *.c files.
        Hide
        Sammi SammiChen added a comment -

        Hi, Wei-Chiu and Kai, thanks so much for review the patch! I realize there may be more comments for documents than for code. So I created a new JIRA HDFS-11033 to track the documentation. Let's focus on code in this JIRA.

        Show
        Sammi SammiChen added a comment - Hi, Wei-Chiu and Kai, thanks so much for review the patch! I realize there may be more comments for documents than for code. So I created a new JIRA HDFS-11033 to track the documentation. Let's focus on code in this JIRA.
        Hide
        Sammi SammiChen added a comment -

        Remove unnecessary #include in .c files.

        Show
        Sammi SammiChen added a comment - Remove unnecessary #include in .c files.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 6m 49s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 19s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 6m 46s the patch passed
        +1 cc 6m 46s the patch passed
        +1 javac 6m 46s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 28s the patch passed
        +1 javadoc 0m 42s the patch passed
        +1 unit 8m 12s hadoop-common in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        38m 36s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-11798
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834115/HADOOP-11798-v5.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle
        uname Linux 12d97e29d2d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5573e6
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10827/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10827/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 6m 52s trunk passed +1 compile 6m 49s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 19s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 6m 46s the patch passed +1 cc 6m 46s the patch passed +1 javac 6m 46s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 42s the patch passed +1 unit 8m 12s hadoop-common in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 38m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11798 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834115/HADOOP-11798-v5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml cc findbugs checkstyle uname Linux 12d97e29d2d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5573e6 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10827/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10827/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment - - edited

        +1. I'll postpone until end of tomorrow to allow any watchers to comment.

        Show
        jojochuang Wei-Chiu Chuang added a comment - - edited +1. I'll postpone until end of tomorrow to allow any watchers to comment.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed patch v5 to trunk. Thanks to SammiChen for contributing the patch and Kai Zheng for driving the review!

        SammiChen would you mind to add a short release note to this jira?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed patch v5 to trunk. Thanks to SammiChen for contributing the patch and Kai Zheng for driving the review! SammiChen would you mind to add a short release note to this jira?
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10679 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10679/)
        HADOOP-11798. Native raw erasure coder in XOR codes. Contributed by (weichiu: rev d88dca844a8437397959f28dadf1294e6db31df9)

        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawEncoder.java
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderBase.java
        • (edit) hadoop-common-project/hadoop-common/src/CMakeLists.txt
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawErasureCoderFactory.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawDecoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
        • (add) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_xor_decoder.c
        • (add) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_xor_encoder.c
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoder.java
        • (edit) hadoop-common-project/hadoop-common/pom.xml
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderInteroperable1.java
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderInteroperable2.java
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestNativeXORRawCoder.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10679 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10679/ ) HADOOP-11798 . Native raw erasure coder in XOR codes. Contributed by (weichiu: rev d88dca844a8437397959f28dadf1294e6db31df9) (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawEncoder.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderBase.java (edit) hadoop-common-project/hadoop-common/src/CMakeLists.txt (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawErasureCoderFactory.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/NativeXORRawDecoder.java (edit) hadoop-common-project/hadoop-common/src/main/native/native.vcxproj (add) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_xor_decoder.c (add) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/erasurecode/jni_xor_encoder.c (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoder.java (edit) hadoop-common-project/hadoop-common/pom.xml (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderInteroperable1.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXORRawCoderInteroperable2.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestNativeXORRawCoder.java
        Hide
        Sammi SammiChen added a comment -

        Thanks Wei-Chiu Chuang so much for reviewing and commit the patch! Sure, I will add the release note.

        Show
        Sammi SammiChen added a comment - Thanks Wei-Chiu Chuang so much for reviewing and commit the patch! Sure, I will add the release note.

          People

          • Assignee:
            Sammi SammiChen
            Reporter:
            drankye Kai Zheng
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development