Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: io
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Use configuration properties io.erasurecode.codec.{rs-legacy,rs,xor}.rawcoders to control erasure coding codec. These properties support codec fallback in case the previous codec is not loaded.

      Description

      The current EC codec supports a single coder only (by default pure Java implementation). If the native coder is specified but is unavailable, it should fallback to pure Java implementation.

      One possible solution is to follow the convention of existing Hadoop native codec, such as transport encryption (see CryptoCodec.java). It supports fallback by specifying two or multiple coders as the value of property, and loads coders in order.

      1. HADOOP-13665.01.patch
        10 kB
        Kai Sasaki
      2. HADOOP-13665.02.patch
        9 kB
        Kai Sasaki
      3. HADOOP-13665.03.patch
        10 kB
        Kai Sasaki
      4. HADOOP-13665.04.patch
        10 kB
        Kai Sasaki
      5. HADOOP-13665.05.patch
        6 kB
        Kai Sasaki
      6. HADOOP-13665.06.patch
        11 kB
        Kai Sasaki
      7. HADOOP-13665.07.patch
        11 kB
        Kai Sasaki
      8. HADOOP-13665.08.patch
        12 kB
        Kai Sasaki
      9. HADOOP-13665.09.patch
        12 kB
        Kai Sasaki
      10. HADOOP-13665.10.patch
        15 kB
        Kai Sasaki
      11. HADOOP-13665.11.patch
        23 kB
        Kai Sasaki
      12. HADOOP-13665.12.patch
        23 kB
        Kai Sasaki

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          HADOOP-13061 is a related jira and the patch is pending.

          Show
          jojochuang Wei-Chiu Chuang added a comment - HADOOP-13061 is a related jira and the patch is pending.
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Sorry for late update. It seems good to fallback multiple coder like CryptoCodec.
          After merging HADOOP-13061, I'll work on this.

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Sorry for late update. It seems good to fallback multiple coder like CryptoCodec . After merging HADOOP-13061 , I'll work on this.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Kai Sasaki appreciate if you can work on this when you get a chance. Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Kai Sasaki appreciate if you can work on this when you get a chance. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 9m 28s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 23s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 9m 8s the patch passed
          +1 javac 9m 8s the patch passed
          -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 10 new + 2 unchanged - 0 fixed = 12 total (was 2)
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 7m 51s hadoop-common in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          44m 18s



          Reason Tests
          Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842712/HADOOP-13665.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a13e10168292 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c38f11
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 42s trunk passed +1 compile 9m 28s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 9m 8s the patch passed +1 javac 9m 8s the patch passed -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 10 new + 2 unchanged - 0 fixed = 12 total (was 2) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 51s hadoop-common in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 44m 18s Reason Tests Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842712/HADOOP-13665.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a13e10168292 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11243/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 55s trunk passed
          +1 compile 9m 39s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 46s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 12s the patch passed
          +1 javac 9m 12s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 47s the patch passed
          +1 unit 8m 25s hadoop-common in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          51m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843756/HADOOP-13665.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 102cc9d25804 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fcbe152
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11298/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11298/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 55s trunk passed +1 compile 9m 39s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 12s the patch passed +1 javac 9m 12s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 47s the patch passed +1 unit 8m 25s hadoop-common in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 51m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843756/HADOOP-13665.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 102cc9d25804 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fcbe152 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11298/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11298/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Could you review this when you get a chance?

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Could you review this when you get a chance?
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          Kai Sasaki thanks for the patch!
          The patch looks good overall. I would like to ask you to add a few log messages for better supportability.

          • Please log a debug message for the codec that is eventually loaded.
          • Inside {{catch (RuntimeException e) }} block, please add either a WARN log that prints the cause of the exception.

          These messages can greatly help diagnosing performance issues.

          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited Kai Sasaki thanks for the patch! The patch looks good overall. I would like to ask you to add a few log messages for better supportability. Please log a debug message for the codec that is eventually loaded. Inside {{catch (RuntimeException e) }} block, please add either a WARN log that prints the cause of the exception. These messages can greatly help diagnosing performance issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 23s trunk passed
          +1 compile 9m 31s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 9m 11s the patch passed
          +1 javac 9m 11s the patch passed
          -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 8m 11s hadoop-common in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          50m 25s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844632/HADOOP-13665.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1ea40631199b 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c721f78
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 23s trunk passed +1 compile 9m 31s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 9m 11s the patch passed +1 javac 9m 11s the patch passed -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 8m 11s hadoop-common in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 50m 25s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844632/HADOOP-13665.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1ea40631199b 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c721f78 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11322/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 27s trunk passed
          +1 compile 9m 57s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 9m 58s the patch passed
          +1 javac 9m 58s the patch passed
          +1 checkstyle 0m 33s the patch passed
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 8m 0s hadoop-common in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          52m 23s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844636/HADOOP-13665.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a722af64e431 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c721f78
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 27s trunk passed +1 compile 9m 57s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 9m 58s the patch passed +1 javac 9m 58s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 8m 0s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 52m 23s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844636/HADOOP-13665.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a722af64e431 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c721f78 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11324/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          Hi Kai Sasaki,
          thank you for your updated patch, but I am sorry that I think I missed a few critical issues in the patch in my last review.

          First of all, ec codecs are specified without implementation – For example, DFSStripedoutputStream called CodecUtil.createRawEncoder but currently all defined schemas specifies either rs-default, rs-legacy or xor. It should be intuitive that codecs are not implementation-specific, that is, a native rs-default codec implementation should be interoperable with a pure Java rs-default codec implementation. So adding a new codec like NATIVE_XOR_CODEC_NAME doesn't make sense.

          Second, it is potentially risky to assume native codec is always preferred over Java codec implementation, for instance if native codec has a bug and we are forced to use Java codec until the native one is fixed.

          Lastly, can we follow the existing conventions used in in-transit encryption? For example, the default value of configuration key hadoop.security.crypto.codec.classes.aes.ctr.nopadding is org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec which means OpensslAesCtrCryptoCodec is preferred over JceAesCtrCryptoCodec, but it is configurable.

          Also one nit: please use LOG.debug with curl bracket style is preferred for performance reasons: for example, instead of

          LOG.debug("Creating raw encoder by using " + rawCoderFactoryKey);
          

          use

          LOG.debug("Creating raw encoder by using {}", rawCoderFactoryKey);
          
          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited Hi Kai Sasaki , thank you for your updated patch, but I am sorry that I think I missed a few critical issues in the patch in my last review. First of all, ec codecs are specified without implementation – For example, DFSStripedoutputStream called CodecUtil.createRawEncoder but currently all defined schemas specifies either rs-default, rs-legacy or xor. It should be intuitive that codecs are not implementation-specific, that is, a native rs-default codec implementation should be interoperable with a pure Java rs-default codec implementation. So adding a new codec like NATIVE_XOR_CODEC_NAME doesn't make sense. Second, it is potentially risky to assume native codec is always preferred over Java codec implementation, for instance if native codec has a bug and we are forced to use Java codec until the native one is fixed. Lastly, can we follow the existing conventions used in in-transit encryption? For example, the default value of configuration key hadoop.security.crypto.codec.classes.aes.ctr.nopadding is org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec which means OpensslAesCtrCryptoCodec is preferred over JceAesCtrCryptoCodec, but it is configurable. Also one nit: please use LOG.debug with curl bracket style is preferred for performance reasons: for example, instead of LOG.debug( "Creating raw encoder by using " + rawCoderFactoryKey); use LOG.debug( "Creating raw encoder by using {}" , rawCoderFactoryKey);
          Hide
          andrew.wang Andrew Wang added a comment -

          Marking ec must do's as blockers for alpha3

          Show
          andrew.wang Andrew Wang added a comment - Marking ec must do's as blockers for alpha3
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Ping. Kai Sasaki thanks again for the patch. This is a pretty important part to make this feature more supportable. Would you be able to revive this work?

          Show
          jojochuang Wei-Chiu Chuang added a comment - Ping. Kai Sasaki thanks again for the patch. This is a pretty important part to make this feature more supportable. Would you be able to revive this work?
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Very sorry for late response and thanks for pinging.

          So adding a new codec like NATIVE_XOR_CODEC_NAME doesn't make sense.

          I'll remove native codec configuration to make it transparently. But how can we specify native coder? If Java implementation is always preferred over native coder, there is no way to use native coder, I think. Does it mean it is unnecessary fallback any coder?

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Very sorry for late response and thanks for pinging. So adding a new codec like NATIVE_XOR_CODEC_NAME doesn't make sense. I'll remove native codec configuration to make it transparently. But how can we specify native coder? If Java implementation is always preferred over native coder, there is no way to use native coder, I think. Does it mean it is unnecessary fallback any coder?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I think the key here is to make the order of fallback codecs configurable, like in the case of security crypto codec. There are two benefits: (1) we might be able to support more codecs in the future (2) if one of the preferred codec is buggy, we could replace it without writing new code. This comes handy when troubleshotting a cluster, especially since this feature has not been verified in production (at least for us).

          Show
          jojochuang Wei-Chiu Chuang added a comment - I think the key here is to make the order of fallback codecs configurable, like in the case of security crypto codec. There are two benefits: (1) we might be able to support more codecs in the future (2) if one of the preferred codec is buggy, we could replace it without writing new code. This comes handy when troubleshotting a cluster, especially since this feature has not been verified in production (at least for us).
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Thanks for clear explanation

          to make the order of fallback codecs configurable

          I understand. As you said, I think we can adopt same convention like crypto codec. I'll update the patch accordingly. Thanks!

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Thanks for clear explanation to make the order of fallback codecs configurable I understand. As you said, I think we can adopt same convention like crypto codec. I'll update the patch accordingly. Thanks!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          FYI Kai Sasaki,
          see the comment at https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=15890254&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15890254
          There are folks who are implementing a new EC codec. Let's make sure the patch is pluggable so that adding a new EC codec is easy.

          Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - FYI Kai Sasaki , see the comment at https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=15890254&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15890254 There are folks who are implementing a new EC codec. Let's make sure the patch is pluggable so that adding a new EC codec is easy. Thanks!
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Thanks for letting me know. Will check and update accordingly.

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Thanks for letting me know. Will check and update accordingly.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Kai Sasaki, any update on this JIRA?

          Show
          andrew.wang Andrew Wang added a comment - Hi Kai Sasaki , any update on this JIRA?
          Hide
          lewuathe Kai Sasaki added a comment - - edited

          Andrew Wang Sorry for late response. I updated the patch.

          • Codec order can be specified by io.erasurecode.codec.coders in comma separated format.
          • As default (not specify io.erasure.codec.coders) the bahaviour is not changed. codec passed to createRawEncoder and createRawDecoder is used as before.
          Show
          lewuathe Kai Sasaki added a comment - - edited Andrew Wang Sorry for late response. I updated the patch. Codec order can be specified by io.erasurecode.codec.coders in comma separated format. As default (not specify io.erasure.codec.coders ) the bahaviour is not changed. codec passed to createRawEncoder and createRawDecoder is used as before.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 19s trunk passed
          +1 compile 10m 27s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 16s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 11m 39s the patch passed
          +1 javac 11m 39s the patch passed
          +1 checkstyle 0m 40s the patch passed
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 57s the patch passed
          -1 unit 8m 26s hadoop-common in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          57m 20s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859383/HADOOP-13665.05.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b5d30d65af67 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7f8e928
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 19s trunk passed +1 compile 10m 27s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 11m 39s the patch passed +1 javac 11m 39s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 8m 26s hadoop-common in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 57m 20s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859383/HADOOP-13665.05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b5d30d65af67 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7f8e928 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11847/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Thanks all for the discussion and I took a quick look.

          +    conf.set(CodecUtil.IO_ERASURECODE_CODEC_CODERS,
          +            ErasureCodeConstants.RS_CODEC_NAME + "," +
          +            ErasureCodeConstants.XOR_CODEC_NAME);
          

          It doesn't make sense to mix and configure something for RS codec and XOR codec together. Instead, it should be to allow configuring more coders for the same codec in order, IIUC.

          For example, for the RS codec, we have the pure Java coder and also the ISA-L based native coder. As Chuang proposed, what we want to do is to allow configuring the two coders using the property like io.erasurecode.codec.rs.rawcoders in addition to what we have already as below

            /** Raw coder factory for the RS codec. */
            public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY =
                "io.erasurecode.codec.rs.rawcoder";
            public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT =
                RSRawErasureCoderFactory.class.getCanonicalName();
          
            /** Raw coder factory for the RS legacy codec. */
            public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY =
                "io.erasurecode.codec.rs-legacy.rawcoder";
            public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_DEFAULT =
                RSRawErasureCoderFactoryLegacy.class.getCanonicalName();
          
            /** Raw coder factory for the XOR codec. */
            public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY =
                "io.erasurecode.codec.xor.rawcoder";
            public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_DEFAULT =
                XORRawErasureCoderFactory.class.getCanonicalName();
          
          Show
          drankye Kai Zheng added a comment - Thanks all for the discussion and I took a quick look. + conf.set(CodecUtil.IO_ERASURECODE_CODEC_CODERS, + ErasureCodeConstants.RS_CODEC_NAME + "," + + ErasureCodeConstants.XOR_CODEC_NAME); It doesn't make sense to mix and configure something for RS codec and XOR codec together. Instead, it should be to allow configuring more coders for the same codec in order, IIUC. For example, for the RS codec, we have the pure Java coder and also the ISA-L based native coder. As Chuang proposed, what we want to do is to allow configuring the two coders using the property like io.erasurecode.codec.rs.rawcoders in addition to what we have already as below /** Raw coder factory for the RS codec. */ public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY = "io.erasurecode.codec.rs.rawcoder" ; public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT = RSRawErasureCoderFactory.class.getCanonicalName(); /** Raw coder factory for the RS legacy codec. */ public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY = "io.erasurecode.codec.rs-legacy.rawcoder" ; public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_DEFAULT = RSRawErasureCoderFactoryLegacy.class.getCanonicalName(); /** Raw coder factory for the XOR codec. */ public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY = "io.erasurecode.codec.xor.rawcoder" ; public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_DEFAULT = XORRawErasureCoderFactory.class.getCanonicalName();
          Hide
          lewuathe Kai Sasaki added a comment - - edited

          Kai Zheng Thanks for comment!

          It doesn't make sense to mix and configure something for RS codec and XOR codec together.

          So I think we can create io.erasurecode.codec.rs.rawcoders, io.erasurecode.codec.xor.rawcoders etc. And if we have io.erasurecode.codec.rs.rawcoders} and {{io.erasurecode.codec.rs.rawcoder, we can prioritize io.erasurecode.codec.rs.rawcoder, right?

          Show
          lewuathe Kai Sasaki added a comment - - edited Kai Zheng Thanks for comment! It doesn't make sense to mix and configure something for RS codec and XOR codec together. So I think we can create io.erasurecode.codec.rs.rawcoders , io.erasurecode.codec.xor.rawcoders etc. And if we have io.erasurecode.codec.rs.rawcoders} and {{io.erasurecode.codec.rs.rawcoder , we can prioritize io.erasurecode.codec.rs.rawcoder , right?
          Hide
          drankye Kai Zheng added a comment -

          Thanks Kai Sasaki!

          I guess we're much aligned. We don't need io.erasurecode.codec.rs.rawcoder any more if we have io.erasurecode.codec.rs.rawcoders. I mean, if the value of ...rawcoders contains more than one, we use the first one; if just one, it's OK and we just use it. Sounds good?

          Show
          drankye Kai Zheng added a comment - Thanks Kai Sasaki ! I guess we're much aligned. We don't need io.erasurecode.codec.rs.rawcoder any more if we have io.erasurecode.codec.rs.rawcoders . I mean, if the value of ...rawcoders contains more than one, we use the first one; if just one, it's OK and we just use it. Sounds good?
          Hide
          lewuathe Kai Sasaki added a comment -

          Kai Zheng Thanks for explanation! It's very clear to me. I'll update accordingly.

          Show
          lewuathe Kai Sasaki added a comment - Kai Zheng Thanks for explanation! It's very clear to me. I'll update accordingly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 59s trunk passed
          +1 compile 17m 5s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 13m 49s the patch passed
          +1 javac 13m 49s the patch passed
          -0 checkstyle 0m 36s hadoop-common-project/hadoop-common: The patch generated 8 new + 2 unchanged - 0 fixed = 10 total (was 2)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 35s the patch passed
          +1 javadoc 0m 50s the patch passed
          -1 unit 8m 26s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          64m 9s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860372/HADOOP-13665.06.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 860d884b355a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ab759e9
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 59s trunk passed +1 compile 17m 5s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 13m 49s the patch passed +1 javac 13m 49s the patch passed -0 checkstyle 0m 36s hadoop-common-project/hadoop-common: The patch generated 8 new + 2 unchanged - 0 fixed = 10 total (was 2) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 35s the patch passed +1 javadoc 0m 50s the patch passed -1 unit 8m 26s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 64m 9s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860372/HADOOP-13665.06.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 860d884b355a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ab759e9 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11918/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 45s trunk passed
          +1 compile 15m 29s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 25s trunk passed
          +1 javadoc 0m 49s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 13m 43s the patch passed
          +1 javac 13m 43s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 49s the patch passed
          -1 unit 7m 58s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          61m 55s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860526/HADOOP-13665.07.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c08fd341aa98 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 84ddedc
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 45s trunk passed +1 compile 15m 29s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 25s trunk passed +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 13m 43s the patch passed +1 javac 13m 43s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 7m 58s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 61m 55s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860526/HADOOP-13665.07.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c08fd341aa98 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 84ddedc Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11933/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Kai for the update!

          +  private static String getRawCoders(Configuration conf, String codec) {
          +    switch (codec) {
          +    case ErasureCodeConstants.RS_CODEC_NAME:
          +    case ErasureCodeConstants.RS_LEGACY_CODEC_NAME:
          +      return conf.get(
          +        IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY,
          +        IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT);
          

          1. RS_CODEC and RS_LEGACY_CODEC should be different codecs. They both need to be configured which raw coders to use.
          2. The switch could be avoided here. To be simple, I think we can reduce the property key name according to the codec name. For example, if the codec name is xcodec, then the property key to be used to get the raw coders can be: "io.erasurecode.codec." + xcodec + ".rawcoders";

          +  private static String getRawCoderFactNameWithFallback(Configuration conf,
          +                                                        String codecs) {
          +    for (String codec : Splitter.on(",").split(codecs)) {
          +      String factName = getRawCoderFactNameFromCodec(conf, codec);
          +      if (factName != null) {
          +        return factName;
          +      }
          +      // In case of not found raw coder, fallback to next
          +    }
          +    throw new IllegalArgumentException("Raw coder factory not configured " +
          +            "for custom codecs: " + codecs);
          +  }
          

          Just calling getRawCoderFactNameFromCodec may be not enough, the failure can happen during coder instance creating, for example, if native library loading failed.

          An elegant way could be to come up a composite RawErasureCoderFactory that wraps and favors the configured raw coders list one by one during createEncoder/Decoder call.

          Show
          drankye Kai Zheng added a comment - Thanks Kai for the update! + private static String getRawCoders(Configuration conf, String codec) { + switch (codec) { + case ErasureCodeConstants.RS_CODEC_NAME: + case ErasureCodeConstants.RS_LEGACY_CODEC_NAME: + return conf.get( + IO_ERASURECODE_CODEC_RS_RAWCODERS_KEY, + IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT); 1. RS_CODEC and RS_LEGACY_CODEC should be different codecs. They both need to be configured which raw coders to use. 2. The switch could be avoided here. To be simple, I think we can reduce the property key name according to the codec name. For example, if the codec name is xcodec , then the property key to be used to get the raw coders can be: "io.erasurecode.codec." + xcodec + ".rawcoders" ; + private static String getRawCoderFactNameWithFallback(Configuration conf, + String codecs) { + for ( String codec : Splitter.on( "," ).split(codecs)) { + String factName = getRawCoderFactNameFromCodec(conf, codec); + if (factName != null ) { + return factName; + } + // In case of not found raw coder, fallback to next + } + throw new IllegalArgumentException( "Raw coder factory not configured " + + " for custom codecs: " + codecs); + } Just calling getRawCoderFactNameFromCodec may be not enough, the failure can happen during coder instance creating, for example, if native library loading failed. An elegant way could be to come up a composite RawErasureCoderFactory that wraps and favors the configured raw coders list one by one during createEncoder/Decoder call.
          Hide
          lewuathe Kai Sasaki added a comment -

          Thanks Kai Zheng for comments.

          For example, if the codec name is xcodec, then the property key to be used to get the raw coders can be: "io.erasurecode.codec." + xcodec + ".rawcoders";

          The default value of each config cannot be decided by given codec directly. So I set the default value into DEFAULT_CODERS_MAP.

          Show
          lewuathe Kai Sasaki added a comment - Thanks Kai Zheng for comments. For example, if the codec name is xcodec, then the property key to be used to get the raw coders can be: "io.erasurecode.codec." + xcodec + ".rawcoders"; The default value of each config cannot be decided by given codec directly. So I set the default value into DEFAULT_CODERS_MAP .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 15s trunk passed
          +1 compile 18m 6s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 39s the patch passed
          +1 compile 14m 57s the patch passed
          +1 javac 14m 57s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 8m 21s hadoop-common in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          66m 43s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861626/HADOOP-13665.08.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b969d0186857 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a4b5aa8
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 15s trunk passed +1 compile 18m 6s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 14m 57s the patch passed +1 javac 14m 57s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 8m 21s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 66m 43s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861626/HADOOP-13665.08.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b969d0186857 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a4b5aa8 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12006/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 26s trunk passed
          +1 compile 16m 24s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 27s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 15m 23s the patch passed
          +1 javac 15m 23s the patch passed
          -0 checkstyle 0m 38s hadoop-common-project/hadoop-common: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 49s the patch passed
          -1 unit 7m 52s hadoop-common in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          66m 20s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861629/HADOOP-13665.08.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bf1245e4ac3d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a4b5aa8
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 26s trunk passed +1 compile 16m 24s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 27s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 15m 23s the patch passed +1 javac 15m 23s the patch passed -0 checkstyle 0m 38s hadoop-common-project/hadoop-common: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 7m 52s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 66m 20s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861629/HADOOP-13665.08.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf1245e4ac3d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a4b5aa8 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12007/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 58s trunk passed
          +1 compile 15m 7s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 13m 54s the patch passed
          +1 javac 13m 54s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 49s the patch passed
          -1 unit 7m 48s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          61m 40s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861701/HADOOP-13665.09.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6ef1ee383c10 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 845529b
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 58s trunk passed +1 compile 15m 7s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 13m 54s the patch passed +1 javac 13m 54s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 7m 48s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 61m 40s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861701/HADOOP-13665.09.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6ef1ee383c10 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 845529b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12013/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          Hello Kai Sasaki and Kai Zheng.
          Thank you both for contributing the patch and posting comments. I wanted to review the patch and post comments too. Sorry that I am late to coming to it late.

          Given that native codec is also preferred whenever possible, would it make sense to update IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT to be org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory,org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory? Also the same for other codecs.

          In CodecUtil#createRawEncoderWithFallback, please print a warning message if fallback happens

                catch (Exception e) {
                  // Fallback to next coder if possible
                }
          

          The warning message should be like "Failed to create raw erasure encoder <codec>, fallback to next codec if possible"

          createRawDecoderWithFallback should also have the same warning log

          Could you also use IO_ERASURECODE_CODEC instead of "io.erasurecode.codec."?

          Please also update core-default.xml to include the configuration key io.erasurecode.codec.coders and its default value.

          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited Hello Kai Sasaki and Kai Zheng . Thank you both for contributing the patch and posting comments. I wanted to review the patch and post comments too. Sorry that I am late to coming to it late. Given that native codec is also preferred whenever possible, would it make sense to update IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT to be org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory,org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory ? Also the same for other codecs. In CodecUtil#createRawEncoderWithFallback , please print a warning message if fallback happens catch (Exception e) { // Fallback to next coder if possible } The warning message should be like "Failed to create raw erasure encoder <codec>, fallback to next codec if possible" createRawDecoderWithFallback should also have the same warning log Could you also use IO_ERASURECODE_CODEC instead of "io.erasurecode.codec."? Please also update core-default.xml to include the configuration key io.erasurecode.codec.coders and its default value.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Kai Sasaki for the update and Wei-Chiu Chuang for the comments! It looks close now. I like the idea of createRawEncoderWithFallback a lot.

          would it make sense to update IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT to be org....NativeRSRawErasureCoderFactory,org.....RSRawErasureCoderFactory?

          Yes, we should configure like this, instead of the way as below:

          +  public static final String IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT =
          +      "rs,rs-legacy";
          

          Kai Sasaki, for your understanding, rs and rs-legacy are different codecs so we'd better not mix them.

          Show
          drankye Kai Zheng added a comment - Thanks Kai Sasaki for the update and Wei-Chiu Chuang for the comments! It looks close now. I like the idea of createRawEncoderWithFallback a lot. would it make sense to update IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT to be org....NativeRSRawErasureCoderFactory,org.....RSRawErasureCoderFactory? Yes, we should configure like this, instead of the way as below: + public static final String IO_ERASURECODE_CODEC_RS_RAWCODERS_DEFAULT = + "rs,rs-legacy" ; Kai Sasaki , for your understanding, rs and rs-legacy are different codecs so we'd better not mix them.
          Hide
          lewuathe Kai Sasaki added a comment -

          Wei-Chiu Chuang Kai Zheng Thanks for comment. They makes sense to me. I'll update accordingly.

          Show
          lewuathe Kai Sasaki added a comment - Wei-Chiu Chuang Kai Zheng Thanks for comment. They makes sense to me. I'll update accordingly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 52s trunk passed
          +1 compile 15m 37s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 13m 19s the patch passed
          +1 javac 13m 19s the patch passed
          -0 checkstyle 0m 35s hadoop-common-project/hadoop-common: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 47s the patch passed
          -1 unit 7m 49s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          61m 30s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862290/HADOOP-13665.10.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a6aea5d4e815 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1a9439e
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 52s trunk passed +1 compile 15m 37s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 13m 19s the patch passed +1 javac 13m 19s the patch passed -0 checkstyle 0m 35s hadoop-common-project/hadoop-common: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 47s the patch passed -1 unit 7m 49s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 61m 30s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862290/HADOOP-13665.10.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a6aea5d4e815 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1a9439e Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12041/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Kai Sasaki thanks again for your quick turn-around!

          The v10 patch looks very close.

          A few nits

          • IO_ERASURECODE_CODEC_XOR_RAWCODERS_DEFAULT should also support native fallback, and its value should be org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory,org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory. io.erasurecode.codec.xor.rawcoders in core-default should be updated as well.
          • In getRawCoders(), io.erasurecode.codec. can be replaced by IO_ERASURECODE_CODEC.
          • The private method getRawCoderFactNameFromCodec() is no longer used, and its use is replaced by createRawEncoderWithFallback() and createRawDecoderWithFallback. It should be removed if I understand it correctly. The associated variables such as IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY should also be removed as well because they are not used.
          • In createRawEncoderWithFallback() and createRawDecoderWithFallback(), I think it's better to print the exception so that we know what goes wrong. So instead of
            LOG.warn("Failed to create raw erasure encoder " + factName +
                        ", fallback to next codec if possible");
            

            , do

            LOG.warn("Failed to create raw erasure encoder " + factName +
                        ", fallback to next codec if possible", e);
            
          • We should also update the documentation HDFSErasureCoding.md, but I prefer to do that in a separate jira.
          Show
          jojochuang Wei-Chiu Chuang added a comment - Kai Sasaki thanks again for your quick turn-around! The v10 patch looks very close. A few nits IO_ERASURECODE_CODEC_XOR_RAWCODERS_DEFAULT should also support native fallback, and its value should be org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory,org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory. io.erasurecode.codec.xor.rawcoders in core-default should be updated as well. In getRawCoders(), io.erasurecode.codec. can be replaced by IO_ERASURECODE_CODEC. The private method getRawCoderFactNameFromCodec() is no longer used, and its use is replaced by createRawEncoderWithFallback() and createRawDecoderWithFallback. It should be removed if I understand it correctly. The associated variables such as IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY should also be removed as well because they are not used. In createRawEncoderWithFallback() and createRawDecoderWithFallback(), I think it's better to print the exception so that we know what goes wrong. So instead of LOG.warn( "Failed to create raw erasure encoder " + factName + ", fallback to next codec if possible" ); , do LOG.warn( "Failed to create raw erasure encoder " + factName + ", fallback to next codec if possible" , e); We should also update the documentation HDFSErasureCoding.md, but I prefer to do that in a separate jira.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Mark as incompatible change, because we no longer use the config keys io.erasurecode.codec.{rs-legacy.rawcoder,rs.rawcoder,xor.rawcoder}
          instead,
          io.erasurecode.codec.{rs-legacy.rawcoders,rs.rawcoders,xor.rawcoders} are used.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Mark as incompatible change, because we no longer use the config keys io.erasurecode.codec.{rs-legacy.rawcoder,rs.rawcoder,xor.rawcoder} instead, io.erasurecode.codec.{rs-legacy.rawcoders,rs.rawcoders,xor.rawcoders} are used.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 40s Maven dependency ordering for branch
          +1 mvninstall 13m 20s trunk passed
          +1 compile 15m 49s trunk passed
          +1 checkstyle 2m 4s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 1m 39s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 45s the patch passed
          +1 compile 14m 18s the patch passed
          +1 javac 14m 18s the patch passed
          -0 checkstyle 1m 57s root: The patch generated 6 new + 33 unchanged - 0 fixed = 39 total (was 33)
          +1 mvnsite 2m 8s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 42s the patch passed
          +1 javadoc 1m 46s the patch passed
          -1 unit 7m 59s hadoop-common in the patch failed.
          -1 unit 64m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          164m 45s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862602/HADOOP-13665.11.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 0e53e3a5b3b7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5d38504
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 40s Maven dependency ordering for branch +1 mvninstall 13m 20s trunk passed +1 compile 15m 49s trunk passed +1 checkstyle 2m 4s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 39s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 45s the patch passed +1 compile 14m 18s the patch passed +1 javac 14m 18s the patch passed -0 checkstyle 1m 57s root: The patch generated 6 new + 33 unchanged - 0 fixed = 39 total (was 33) +1 mvnsite 2m 8s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 42s the patch passed +1 javadoc 1m 46s the patch passed -1 unit 7m 59s hadoop-common in the patch failed. -1 unit 64m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 164m 45s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862602/HADOOP-13665.11.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 0e53e3a5b3b7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5d38504 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12068/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 5s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 1m 37s Maven dependency ordering for branch
          +1 mvninstall 14m 23s trunk passed
          +1 compile 15m 59s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 2m 3s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 3m 25s trunk passed
          +1 javadoc 1m 42s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 13m 56s the patch passed
          +1 javac 13m 56s the patch passed
          +1 checkstyle 1m 55s the patch passed
          +1 mvnsite 2m 7s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 44s the patch passed
          +1 javadoc 1m 45s the patch passed
          +1 unit 8m 0s hadoop-common in the patch passed.
          -1 unit 66m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          181m 10s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestStartupOptionUpgrade



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-13665
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862689/HADOOP-13665.12.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 979da39d1436 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 443aa51
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 5s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 14m 23s trunk passed +1 compile 15m 59s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 2m 3s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 3m 25s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 13m 56s the patch passed +1 javac 13m 56s the patch passed +1 checkstyle 1m 55s the patch passed +1 mvnsite 2m 7s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 44s the patch passed +1 javadoc 1m 45s the patch passed +1 unit 8m 0s hadoop-common in the patch passed. -1 unit 66m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 181m 10s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestStartupOptionUpgrade Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-13665 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862689/HADOOP-13665.12.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 979da39d1436 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 443aa51 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12074/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Kai Sasaki for the update! The latest patch LGTM. Guess Wei-Chiu Chuang will take another look and commit it. Thanks.

          Show
          drankye Kai Zheng added a comment - Thanks Kai Sasaki for the update! The latest patch LGTM. Guess Wei-Chiu Chuang will take another look and commit it. Thanks.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1 test failure is unrelated. Will commit soon.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1 test failure is unrelated. Will commit soon.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed to trunk. Thanks Kai Zheng for driving the review and Kai Sasaki for contributing this patch!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed to trunk. Thanks Kai Zheng for driving the review and Kai Sasaki for contributing this patch!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11569 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11569/)
          HADOOP-13665. Erasure Coding codec should support fallback coder. (weichiu: rev f050afb5785dc38875cf644fd4f80a219d4345e7)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11569 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11569/ ) HADOOP-13665 . Erasure Coding codec should support fallback coder. (weichiu: rev f050afb5785dc38875cf644fd4f80a219d4345e7) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
          Hide
          lewuathe Kai Sasaki added a comment -

          Kai Zheng Wei-Chiu Chuang Thanks for review!

          Show
          lewuathe Kai Sasaki added a comment - Kai Zheng Wei-Chiu Chuang Thanks for review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
          HADOOP-13665. Erasure Coding codec should support fallback coder. (weichiu: rev f050afb5785dc38875cf644fd4f80a219d4345e7)

          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) HADOOP-13665 . Erasure Coding codec should support fallback coder. (weichiu: rev f050afb5785dc38875cf644fd4f80a219d4345e7) (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java

            People

            • Assignee:
              lewuathe Kai Sasaki
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development