Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Hadoop Flags:
      Reviewed
    1. HADOOP-13061.01.patch
      50 kB
      Kai Sasaki
    2. HADOOP-13061.02.patch
      53 kB
      Kai Sasaki
    3. HADOOP-13061.03.patch
      105 kB
      Kai Sasaki
    4. HADOOP-13061.04.patch
      105 kB
      Kai Sasaki
    5. HADOOP-13061.05.patch
      105 kB
      Kai Sasaki
    6. HADOOP-13061.06.patch
      84 kB
      Kai Sasaki
    7. HADOOP-13061.07.patch
      87 kB
      Kai Sasaki
    8. HADOOP-13061.08.patch
      79 kB
      Kai Sasaki
    9. HADOOP-13061.09.patch
      78 kB
      Kai Sasaki
    10. HADOOP-13061.10.patch
      78 kB
      Kai Sasaki
    11. HADOOP-13061.11.patch
      81 kB
      Kai Sasaki
    12. HADOOP-13061.12.patch
      81 kB
      Kai Sasaki
    13. HADOOP-13061.13.patch
      81 kB
      Kai Sasaki
    14. HADOOP-13061.14.patch
      83 kB
      Kai Sasaki
    15. HADOOP-13061.15.patch
      101 kB
      Kai Zheng
    16. HADOOP-13061.16.patch
      101 kB
      Kai Zheng
    17. HADOOP-13061.17.patch
      84 kB
      Kai Sasaki
    18. HADOOP-13061.18.patch
      100 kB
      Kai Sasaki
    19. HADOOP-13061.19.patch
      100 kB
      Kai Zheng

      Issue Links

        Activity

        Hide
        lirui Rui Li added a comment -

        Hi Kai Zheng, per our discussion, Wei Zhou would help with this task. But I can't re-assign the JIRA. Please feel free to assign it to Wei if you can edit.

        Show
        lirui Rui Li added a comment - Hi Kai Zheng , per our discussion, Wei Zhou would help with this task. But I can't re-assign the JIRA. Please feel free to assign it to Wei if you can edit.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Rui. Could you ask Wei to create a JIRA account?

        Show
        zhz Zhe Zhang added a comment - Thanks Rui. Could you ask Wei to create a JIRA account?
        Hide
        drankye Kai Zheng added a comment -

        Thank you both, Rui and Zhe. I'm working on this, so will assign it to me.

        Show
        drankye Kai Zheng added a comment - Thank you both, Rui and Zhe. I'm working on this, so will assign it to me.
        Hide
        drankye Kai Zheng added a comment -

        Re-assigned this to Kai per some discussion.

        Hi Kai Sasaki, to have the idea how to do the refactoring, please understand how HADOOP-13010 did.

        Show
        drankye Kai Zheng added a comment - Re-assigned this to Kai per some discussion. Hi Kai Sasaki , to have the idea how to do the refactoring, please understand how HADOOP-13010 did.
        Hide
        lewuathe Kai Sasaki added a comment -

        As did in HADOOP-13010, I'll do refactoring basically

        • Create erasure coder from CodecUtil
        • Options is passed by ErasureCoderOptions.
        • Eliminate AbstractErasureCoder and other AbstractErasure* classes in order to avoid rely on class inheritance. Code is shared by using CodecUtil.
        • Add DummyEncoder and DummyDecoder which wraps DummyRawEncoder and DummyRawDecoder because HDFS will use ErasureCoder interface internally.

        Kai Zheng If there is something to be added or to be removed, please give me any feedback. Thanks.

        Show
        lewuathe Kai Sasaki added a comment - As did in HADOOP-13010 , I'll do refactoring basically Create erasure coder from CodecUtil Options is passed by ErasureCoderOptions . Eliminate AbstractErasureCoder and other AbstractErasure* classes in order to avoid rely on class inheritance. Code is shared by using CodecUtil . Add DummyEncoder and DummyDecoder which wraps DummyRawEncoder and DummyRawDecoder because HDFS will use ErasureCoder interface internally. Kai Zheng If there is something to be added or to be removed, please give me any feedback. Thanks.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Kai for the notes. It looks great and I can't wait to see the effect!

        Show
        drankye Kai Zheng added a comment - Thanks Kai for the notes. It looks great and I can't wait to see the effect!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 3s trunk passed
        +1 compile 8m 4s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 23s trunk passed
        +1 javadoc 0m 45s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 7m 2s the patch passed
        +1 javac 7m 2s the patch passed
        -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 17 new + 27 unchanged - 13 fixed = 44 total (was 40)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 28s the patch passed
        +1 javadoc 0m 46s the patch passed
        -1 unit 7m 36s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        40m 57s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl
          hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821215/HADOOP-13061.01.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0c77c9b924f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3d191cc
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 3s trunk passed +1 compile 8m 4s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 7m 2s the patch passed +1 javac 7m 2s the patch passed -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 17 new + 27 unchanged - 13 fixed = 44 total (was 40) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 36s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 40m 57s Reason Tests Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821215/HADOOP-13061.01.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c77c9b924f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3d191cc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10132/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 10s trunk passed
        +1 compile 7m 11s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 21s trunk passed
        +1 javadoc 0m 46s trunk passed
        +1 mvninstall 0m 41s the patch passed
        +1 compile 7m 14s the patch passed
        +1 javac 7m 14s the patch passed
        +1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 0 new + 27 unchanged - 13 fixed = 27 total (was 40)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 33s the patch passed
        +1 javadoc 0m 46s the patch passed
        -1 unit 8m 44s hadoop-common in the patch failed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        40m 36s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821323/HADOOP-13061.02.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7b822d10dbb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 95694b7
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 10s trunk passed +1 compile 7m 11s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 21s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 7m 14s the patch passed +1 javac 7m 14s the patch passed +1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 0 new + 27 unchanged - 13 fixed = 27 total (was 40) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 8m 44s hadoop-common in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 40m 36s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821323/HADOOP-13061.02.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7b822d10dbb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 95694b7 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10140/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Fixed check style. The test failure seems not related to the patch.
        Could you review this when you have a time? Thanks you.

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Fixed check style. The test failure seems not related to the patch. Could you review this when you have a time? Thanks you.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Kai for the patch. It looks great. Some comments:

        1. Ref. below:

        +  public static ErasureEncoder createEncoder(ErasureCodec codec) {
        +    return codec.createEncoder();
        +  }
        

        Could we have createEncoder(ErasureCodecOptions) instead? ErasureCodecOptions could include all the necessary information from configuration object that can be used to initialize a codec and encoder/decoder thereafter.

        2. So given above, the following could be changed to AbstractErasureCodec(ErasureCodecOptions options)

        +  public AbstractErasureCodec(ECSchema schema, ErasureCoderOptions options)
        
        Show
        drankye Kai Zheng added a comment - Thanks Kai for the patch. It looks great. Some comments: 1. Ref. below: + public static ErasureEncoder createEncoder(ErasureCodec codec) { + return codec.createEncoder(); + } Could we have createEncoder(ErasureCodecOptions) instead? ErasureCodecOptions could include all the necessary information from configuration object that can be used to initialize a codec and encoder/decoder thereafter. 2. So given above, the following could be changed to AbstractErasureCodec(ErasureCodecOptions options) + public AbstractErasureCodec(ECSchema schema, ErasureCoderOptions options)
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Thanks for reviewing. I have one thing to confirm.

        ErasureCodecOptions could include all the necessary information from configuration object that can be used to initialize a codec and encoder/decoder thereafter.

        Since codec information is already included in ECSchema, do you mean making ECSchema deprecated? Or do you intend to ECSchema is included in ErasureCodecOptions?
        I think the latter is better because ECSchema is already used by HDFS Erasure Coding.

        My assumption is here.

        1. ErasureCoderOptions can include ECSchema.

        public final class ErasureCoderOptions {
          private final ECSchema schema;
          private final int numDataUnits;
          private final int numParityUnits;
          private final int numAllUnits;
          private final boolean allowChangeInputs;
          private final boolean allowVerboseDump;
        }
        

        2. CodecUtil creates corresponding ErasureCoder class with given codec which is included ECSchema in ErasureCoderOptions.

        If ECSchema is not used anymore, ErasureCodecOptions needs to only include codec name.

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Thanks for reviewing. I have one thing to confirm. ErasureCodecOptions could include all the necessary information from configuration object that can be used to initialize a codec and encoder/decoder thereafter. Since codec information is already included in ECSchema , do you mean making ECSchema deprecated? Or do you intend to ECSchema is included in ErasureCodecOptions ? I think the latter is better because ECSchema is already used by HDFS Erasure Coding. My assumption is here. 1. ErasureCoderOptions can include ECSchema . public final class ErasureCoderOptions { private final ECSchema schema; private final int numDataUnits; private final int numParityUnits; private final int numAllUnits; private final boolean allowChangeInputs; private final boolean allowVerboseDump; } 2. CodecUtil creates corresponding ErasureCoder class with given codec which is included ECSchema in ErasureCoderOptions . If ECSchema is not used anymore, ErasureCodecOptions needs to only include codec name.
        Hide
        drankye Kai Zheng added a comment -

        I think the latter is better because ECSchema is already used by HDFS Erasure Coding.

        Right. Please keep ECSchema. When you introduce ErasureCodecOption (not ErasureCoderOption), for now it may just contain a schema object, and later more can be added to support additional codec level options. Note some coder level only options like allowChangeInputs won't be needed here. You may ask where we can get such information since it won't be passed down from codec level, it could be retrieved from configuration.

        Show
        drankye Kai Zheng added a comment - I think the latter is better because ECSchema is already used by HDFS Erasure Coding. Right. Please keep ECSchema. When you introduce ErasureCodecOption (not ErasureCoderOption), for now it may just contain a schema object, and later more can be added to support additional codec level options. Note some coder level only options like allowChangeInputs won't be needed here. You may ask where we can get such information since it won't be passed down from codec level, it could be retrieved from configuration.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
        0 mvndep 1m 36s Maven dependency ordering for branch
        +1 mvninstall 8m 30s trunk passed
        +1 compile 8m 19s trunk passed
        +1 checkstyle 1m 37s trunk passed
        +1 mvnsite 2m 46s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 5m 6s trunk passed
        +1 javadoc 2m 26s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 2m 21s the patch passed
        +1 compile 8m 22s the patch passed
        +1 javac 8m 22s the patch passed
        -0 checkstyle 1m 43s root: The patch generated 27 new + 279 unchanged - 19 fixed = 306 total (was 298)
        +1 mvnsite 3m 5s the patch passed
        +1 mvneclipse 0m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 5m 4s the patch passed
        +1 javadoc 2m 16s the patch passed
        -1 unit 7m 38s hadoop-common in the patch failed.
        +1 unit 1m 1s hadoop-hdfs-client in the patch passed.
        -1 unit 57m 51s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        146m 2s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          hadoop.security.TestRefreshUserMappings
          hadoop.tracing.TestTracing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822434/HADOOP-13061.03.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f49c459473b7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 10ed06a
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 8m 30s trunk passed +1 compile 8m 19s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 46s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 6s trunk passed +1 javadoc 2m 26s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 21s the patch passed +1 compile 8m 22s the patch passed +1 javac 8m 22s the patch passed -0 checkstyle 1m 43s root: The patch generated 27 new + 279 unchanged - 19 fixed = 306 total (was 298) +1 mvnsite 3m 5s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 4s the patch passed +1 javadoc 2m 16s the patch passed -1 unit 7m 38s hadoop-common in the patch failed. +1 unit 1m 1s hadoop-hdfs-client in the patch passed. -1 unit 57m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 146m 2s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.TestRefreshUserMappings   hadoop.tracing.TestTracing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822434/HADOOP-13061.03.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f49c459473b7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 10ed06a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10194/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 40s trunk passed
        +1 compile 6m 56s trunk passed
        +1 checkstyle 1m 55s trunk passed
        +1 mvnsite 2m 24s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 4m 37s trunk passed
        +1 javadoc 2m 6s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 2m 3s the patch passed
        +1 compile 6m 44s the patch passed
        +1 javac 6m 44s the patch passed
        -0 checkstyle 1m 31s root: The patch generated 31 new + 279 unchanged - 19 fixed = 310 total (was 298)
        +1 mvnsite 2m 30s the patch passed
        +1 mvneclipse 0m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 5m 5s the patch passed
        +1 javadoc 2m 17s the patch passed
        -1 unit 7m 44s hadoop-common in the patch failed.
        +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
        -1 unit 57m 36s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        137m 43s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          hadoop.security.TestRefreshUserMappings
          hadoop.tracing.TestTracing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822455/HADOOP-13061.04.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7c91bc44f7be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 131d58a
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 6m 56s trunk passed +1 checkstyle 1m 55s trunk passed +1 mvnsite 2m 24s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 4m 37s trunk passed +1 javadoc 2m 6s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 3s the patch passed +1 compile 6m 44s the patch passed +1 javac 6m 44s the patch passed -0 checkstyle 1m 31s root: The patch generated 31 new + 279 unchanged - 19 fixed = 310 total (was 298) +1 mvnsite 2m 30s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 5s the patch passed +1 javadoc 2m 17s the patch passed -1 unit 7m 44s hadoop-common in the patch failed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 57m 36s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 137m 43s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.TestRefreshUserMappings   hadoop.tracing.TestTracing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822455/HADOOP-13061.04.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7c91bc44f7be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 131d58a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10195/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 18 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 7m 51s trunk passed
        +1 compile 6m 50s trunk passed
        +1 checkstyle 1m 26s trunk passed
        +1 mvnsite 2m 19s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 4m 33s trunk passed
        +1 javadoc 2m 5s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 2m 2s the patch passed
        +1 compile 6m 43s the patch passed
        +1 javac 6m 43s the patch passed
        -0 checkstyle 1m 31s root: The patch generated 10 new + 279 unchanged - 19 fixed = 289 total (was 298)
        +1 mvnsite 2m 29s the patch passed
        +1 mvneclipse 0m 49s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 5m 2s the patch passed
        +1 javadoc 2m 16s the patch passed
        +1 unit 8m 3s hadoop-common in the patch passed.
        +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
        -1 unit 57m 37s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        138m 25s



        Reason Tests
        Failed junit tests hadoop.security.TestRefreshUserMappings
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.tracing.TestTracing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822458/HADOOP-13061.05.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a2ccfd062af5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 131d58a
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 18 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 51s trunk passed +1 compile 6m 50s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 2m 19s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 4m 33s trunk passed +1 javadoc 2m 5s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 6m 43s the patch passed +1 javac 6m 43s the patch passed -0 checkstyle 1m 31s root: The patch generated 10 new + 279 unchanged - 19 fixed = 289 total (was 298) +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 2s the patch passed +1 javadoc 2m 16s the patch passed +1 unit 8m 3s hadoop-common in the patch passed. +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 57m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 138m 25s Reason Tests Failed junit tests hadoop.security.TestRefreshUserMappings   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822458/HADOOP-13061.05.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a2ccfd062af5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 131d58a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10197/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Though there is some checkstyle issues, some of the members of TestCoderBase cannot be private because they are inherited and override test method bahaviour. prepare method does this.

        I think we can keep it as it is otherwise it is necessary to modify TestCoderBase structure.

        Show
        lewuathe Kai Sasaki added a comment - Though there is some checkstyle issues, some of the members of TestCoderBase cannot be private because they are inherited and override test method bahaviour. prepare method does this. I think we can keep it as it is otherwise it is necessary to modify TestCoderBase structure.
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Could you review this? and what do you think about above check style issue?

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Could you review this? and what do you think about above check style issue?
        Hide
        drankye Kai Zheng added a comment -

        Hi Kai,

        Looks like you misunderstood what I meant. I meant to introduce a new class say ErasureCodecOptions. Could we avoid changing into HDFS side, since HDFS is still using the raw coder APIs.

        And also, would you help clean the following block in CommonConfigurationKeys? It's bad to mess the common place (not caused by this). Maybe the default values could be defined in CoderUtil place.

        import org.apache.hadoop.io.erasurecode.codec.HHXORErasureCodecFactory;
        import org.apache.hadoop.io.erasurecode.codec.RSErasureCodecFactory;
        import org.apache.hadoop.io.erasurecode.codec.XORErasureCodecFactory;
        import org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory;
        import org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy;
        import org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory;
        
        Show
        drankye Kai Zheng added a comment - Hi Kai, Looks like you misunderstood what I meant. I meant to introduce a new class say ErasureCodecOptions . Could we avoid changing into HDFS side, since HDFS is still using the raw coder APIs. And also, would you help clean the following block in CommonConfigurationKeys ? It's bad to mess the common place (not caused by this). Maybe the default values could be defined in CoderUtil place. import org.apache.hadoop.io.erasurecode.codec.HHXORErasureCodecFactory; import org.apache.hadoop.io.erasurecode.codec.RSErasureCodecFactory; import org.apache.hadoop.io.erasurecode.codec.XORErasureCodecFactory; import org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory; import org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy; import org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory;
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Thanks for checking and sorry for misunderstood.
        I'll make sure not have an influence to HDFS side by using ErasureCodecOptions.

        And also, would you help clean the following block in CommonConfigurationKeys?

        Sure, I'll do also.

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Thanks for checking and sorry for misunderstood. I'll make sure not have an influence to HDFS side by using ErasureCodecOptions . And also, would you help clean the following block in CommonConfigurationKeys? Sure, I'll do also.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 38s Maven dependency ordering for branch
        +1 mvninstall 7m 7s trunk passed
        +1 compile 8m 53s trunk passed
        +1 checkstyle 1m 28s trunk passed
        +1 mvnsite 2m 5s trunk passed
        +1 mvneclipse 0m 30s trunk passed
        +1 findbugs 3m 49s trunk passed
        +1 javadoc 1m 45s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 33s the patch passed
        +1 compile 7m 2s the patch passed
        +1 javac 7m 2s the patch passed
        -0 checkstyle 1m 35s root: The patch generated 38 new + 197 unchanged - 18 fixed = 235 total (was 215)
        +1 mvnsite 2m 4s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 45s the patch passed
        +1 javadoc 1m 57s the patch passed
        -1 unit 7m 29s hadoop-common in the patch failed.
        +1 unit 58m 23s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        134m 47s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823608/HADOOP-13061.06.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 11d3a8c1e187 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23c6e3c
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 38s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 8m 53s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 49s trunk passed +1 javadoc 1m 45s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 7m 2s the patch passed +1 javac 7m 2s the patch passed -0 checkstyle 1m 35s root: The patch generated 38 new + 197 unchanged - 18 fixed = 235 total (was 215) +1 mvnsite 2m 4s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 57s the patch passed -1 unit 7m 29s hadoop-common in the patch failed. +1 unit 58m 23s hadoop-hdfs in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 134m 47s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823608/HADOOP-13061.06.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11d3a8c1e187 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c6e3c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10240/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 6m 35s trunk passed
        +1 compile 6m 43s trunk passed
        +1 checkstyle 1m 25s trunk passed
        +1 mvnsite 1m 46s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 39s trunk passed
        +1 javadoc 1m 52s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 46s the patch passed
        +1 compile 8m 19s the patch passed
        +1 javac 8m 19s the patch passed
        -0 checkstyle 1m 35s root: The patch generated 1 new + 195 unchanged - 20 fixed = 196 total (was 215)
        +1 mvnsite 2m 11s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 50s the patch passed
        +1 javadoc 1m 44s the patch passed
        -1 unit 7m 44s hadoop-common in the patch failed.
        +1 unit 57m 19s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        131m 52s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823623/HADOOP-13061.07.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3b8c7c2a5378 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23c6e3c
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 6m 43s trunk passed +1 checkstyle 1m 25s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 39s trunk passed +1 javadoc 1m 52s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 46s the patch passed +1 compile 8m 19s the patch passed +1 javac 8m 19s the patch passed -0 checkstyle 1m 35s root: The patch generated 1 new + 195 unchanged - 20 fixed = 196 total (was 215) +1 mvnsite 2m 11s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 50s the patch passed +1 javadoc 1m 44s the patch passed -1 unit 7m 44s hadoop-common in the patch failed. +1 unit 57m 19s hadoop-hdfs in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 131m 52s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823623/HADOOP-13061.07.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3b8c7c2a5378 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c6e3c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10241/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Kai for the update! Looks like more changes need to be made according to previous refactoring thoughts and discussions.

        1. Could we get rid of codec factories? They don't seem to be needed.
        2. Could we change AbstractErasureCodec: 1) remove the interface ErasureCodec; 2) remove the parent of Configured.
        3. In AbstractErasureCodec, we could have both coderOptions and codecOptions as members. Please refine the get/set methods accordingly.
        4. I'm sorry, but could you move the new constants added to CoderUtil into CodecUtil? This will help keep CoderUtil internal.

        Show
        drankye Kai Zheng added a comment - Thanks Kai for the update! Looks like more changes need to be made according to previous refactoring thoughts and discussions. 1. Could we get rid of codec factories? They don't seem to be needed. 2. Could we change AbstractErasureCodec : 1) remove the interface ErasureCodec ; 2) remove the parent of Configured. 3. In AbstractErasureCodec, we could have both coderOptions and codecOptions as members. Please refine the get/set methods accordingly. 4. I'm sorry, but could you move the new constants added to CoderUtil into CodecUtil ? This will help keep CoderUtil internal.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 1m 31s Maven dependency ordering for branch
        +1 mvninstall 6m 36s trunk passed
        +1 compile 6m 48s trunk passed
        +1 checkstyle 1m 28s trunk passed
        +1 mvnsite 1m 46s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 4s trunk passed
        +1 javadoc 1m 44s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 6m 45s the patch passed
        +1 javac 6m 45s the patch passed
        -0 checkstyle 1m 31s root: The patch generated 3 new + 192 unchanged - 23 fixed = 195 total (was 215)
        +1 mvnsite 1m 52s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 26s the patch passed
        +1 javadoc 1m 51s the patch passed
        -1 unit 7m 44s hadoop-common in the patch failed.
        +1 unit 56m 30s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        129m 42s



        Reason Tests
        Failed junit tests hadoop.security.ssl.TestSSLFactory
          hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824684/HADOOP-13061.08.patch
        JIRA Issue HADOOP-13061
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f77f84a84c7a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 99603e9
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 1m 31s Maven dependency ordering for branch +1 mvninstall 6m 36s trunk passed +1 compile 6m 48s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 46s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 44s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 6m 45s the patch passed +1 javac 6m 45s the patch passed -0 checkstyle 1m 31s root: The patch generated 3 new + 192 unchanged - 23 fixed = 195 total (was 215) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 26s the patch passed +1 javadoc 1m 51s the patch passed -1 unit 7m 44s hadoop-common in the patch failed. +1 unit 56m 30s hadoop-hdfs in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 129m 42s Reason Tests Failed junit tests hadoop.security.ssl.TestSSLFactory   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824684/HADOOP-13061.08.patch JIRA Issue HADOOP-13061 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f77f84a84c7a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 99603e9 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10321/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 1m 22s Maven dependency ordering for branch
        +1 mvninstall 6m 38s trunk passed
        +1 compile 6m 54s trunk passed
        +1 checkstyle 1m 27s trunk passed
        +1 mvnsite 1m 48s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 3m 5s trunk passed
        +1 javadoc 1m 43s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 6m 51s the patch passed
        +1 javac 6m 51s the patch passed
        +1 checkstyle 1m 29s root: The patch generated 0 new + 190 unchanged - 23 fixed = 190 total (was 213)
        +1 mvnsite 1m 52s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 2s the patch passed
        +1 javadoc 1m 57s the patch passed
        -1 unit 17m 18s hadoop-common in the patch failed.
        +1 unit 60m 37s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        144m 3s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields
        Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827913/HADOOP-13061.10.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5f97a0ba3492 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bee9f57
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 1m 22s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 6m 54s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed +1 checkstyle 1m 29s root: The patch generated 0 new + 190 unchanged - 23 fixed = 190 total (was 213) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 2s the patch passed +1 javadoc 1m 57s the patch passed -1 unit 17m 18s hadoop-common in the patch failed. +1 unit 60m 37s hadoop-hdfs in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 144m 3s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827913/HADOOP-13061.10.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f97a0ba3492 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bee9f57 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10476/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        It looks much close now. Some minors:

        • AbstractErasureCodec => ErasureCodec, and in the class, ecCodecOptions => codecOptions, ecCoderOptions => coderOptions, also the related methods.
        • Maybe a package-info.java in the codec package?
        • Class comments for DummyErasureEncoder/DummyErasureDecoder could be refined;
        • Class comments for CodecUtil could be refined and more accurate now;
        • In CodecUtil#getCodecName, it should be getCodecClassName instead. It can just return the current codec class name (instead of from configuration), because we don't configure codec impl, instead we configure raw coder impl for a codec. Given a codec name, we can safely return the codec class.

        +1 once above addressed. Thanks!

        Show
        drankye Kai Zheng added a comment - It looks much close now. Some minors: AbstractErasureCodec => ErasureCodec, and in the class, ecCodecOptions => codecOptions, ecCoderOptions => coderOptions, also the related methods. Maybe a package-info.java in the codec package? Class comments for DummyErasureEncoder/DummyErasureDecoder could be refined; Class comments for CodecUtil could be refined and more accurate now; In CodecUtil#getCodecName, it should be getCodecClassName instead. It can just return the current codec class name (instead of from configuration), because we don't configure codec impl, instead we configure raw coder impl for a codec. Given a codec name, we can safely return the codec class. +1 once above addressed. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 55s trunk passed
        +1 compile 7m 31s trunk passed
        +1 checkstyle 1m 34s trunk passed
        +1 mvnsite 1m 54s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 15s trunk passed
        +1 javadoc 1m 48s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 34s the patch passed
        +1 compile 7m 24s the patch passed
        +1 javac 7m 24s the patch passed
        -0 checkstyle 1m 36s root: The patch generated 3 new + 188 unchanged - 24 fixed = 191 total (was 212)
        +1 mvnsite 2m 0s the patch passed
        +1 mvneclipse 0m 32s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 31s the patch passed
        +1 javadoc 1m 52s the patch passed
        -1 unit 6m 58s hadoop-common in the patch failed.
        -1 unit 59m 44s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        133m 3s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829162/HADOOP-13061.11.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a895b54cc307 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b8a30f2
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 7m 31s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 15s trunk passed +1 javadoc 1m 48s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 7m 24s the patch passed +1 javac 7m 24s the patch passed -0 checkstyle 1m 36s root: The patch generated 3 new + 188 unchanged - 24 fixed = 191 total (was 212) +1 mvnsite 2m 0s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 52s the patch passed -1 unit 6m 58s hadoop-common in the patch failed. -1 unit 59m 44s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 133m 3s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829162/HADOOP-13061.11.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a895b54cc307 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b8a30f2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
        0 mvndep 1m 32s Maven dependency ordering for branch
        +1 mvninstall 6m 49s trunk passed
        +1 compile 6m 51s trunk passed
        +1 checkstyle 1m 28s trunk passed
        +1 mvnsite 1m 48s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 10s trunk passed
        +1 javadoc 1m 43s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 6m 51s the patch passed
        +1 javac 6m 51s the patch passed
        +1 checkstyle 1m 32s root: The patch generated 0 new + 189 unchanged - 24 fixed = 189 total (was 213)
        +1 mvnsite 1m 51s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 27s the patch passed
        +1 javadoc 1m 51s the patch passed
        -1 unit 7m 54s hadoop-common in the patch failed.
        +1 unit 59m 16s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        133m 8s



        Reason Tests
        Failed junit tests hadoop.net.TestDNS
          hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831748/HADOOP-13061.13.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ee9d5970f316 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6cfe649
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 6m 51s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed +1 checkstyle 1m 32s root: The patch generated 0 new + 189 unchanged - 24 fixed = 189 total (was 213) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 27s the patch passed +1 javadoc 1m 51s the patch passed -1 unit 7m 54s hadoop-common in the patch failed. +1 unit 59m 16s hadoop-hdfs in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 133m 8s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831748/HADOOP-13061.13.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ee9d5970f316 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6cfe649 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10670/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 6m 43s trunk passed
        +1 compile 6m 44s trunk passed
        +1 checkstyle 1m 28s trunk passed
        +1 mvnsite 1m 49s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 17s trunk passed
        +1 javadoc 1m 27s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 49s the patch passed
        +1 compile 7m 49s the patch passed
        +1 javac 7m 49s the patch passed
        +1 checkstyle 1m 34s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218)
        +1 mvnsite 1m 57s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 1m 39s the patch passed
        +1 unit 8m 42s hadoop-common in the patch passed.
        -1 unit 63m 15s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        137m 8s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832130/HADOOP-13061.14.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c153befe397d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ebd4f39
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 6m 44s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 49s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 17s trunk passed +1 javadoc 1m 27s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 49s the patch passed +1 compile 7m 49s the patch passed +1 javac 7m 49s the patch passed +1 checkstyle 1m 34s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 57s the patch passed +1 javadoc 1m 39s the patch passed +1 unit 8m 42s hadoop-common in the patch passed. -1 unit 63m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 137m 8s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832130/HADOOP-13061.14.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c153befe397d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ebd4f39 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10700/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment - - edited

        The patch looks very close. Updated the patch to refine some minors further. Kai Sasaki would you help review the changes? Thanks.

        Show
        drankye Kai Zheng added a comment - - edited The patch looks very close. Updated the patch to refine some minors further. Kai Sasaki would you help review the changes? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
        0 mvndep 1m 32s Maven dependency ordering for branch
        +1 mvninstall 7m 8s trunk passed
        +1 compile 8m 22s trunk passed
        +1 checkstyle 1m 37s trunk passed
        +1 mvnsite 2m 7s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 30s trunk passed
        +1 javadoc 1m 31s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        -1 mvninstall 0m 37s hadoop-common in the patch failed.
        -1 mvninstall 0m 42s hadoop-hdfs in the patch failed.
        -1 compile 0m 57s root in the patch failed.
        -1 javac 0m 57s root in the patch failed.
        +1 checkstyle 1m 33s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218)
        -1 mvnsite 0m 43s hadoop-common in the patch failed.
        -1 mvnsite 0m 55s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 30s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 29s hadoop-common in the patch failed.
        -1 findbugs 0m 30s hadoop-hdfs in the patch failed.
        +1 javadoc 1m 38s the patch passed
        -1 unit 0m 55s hadoop-common in the patch failed.
        -1 unit 0m 49s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        60m 37s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832628/HADOOP-13061.15.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8c9888ce93bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 96b1266
        Default Java 1.8.0_101
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
        mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-compile-root.txt
        mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
        mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 7m 8s trunk passed +1 compile 8m 22s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 7s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 30s trunk passed +1 javadoc 1m 31s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch -1 mvninstall 0m 37s hadoop-common in the patch failed. -1 mvninstall 0m 42s hadoop-hdfs in the patch failed. -1 compile 0m 57s root in the patch failed. -1 javac 0m 57s root in the patch failed. +1 checkstyle 1m 33s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218) -1 mvnsite 0m 43s hadoop-common in the patch failed. -1 mvnsite 0m 55s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 30s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 29s hadoop-common in the patch failed. -1 findbugs 0m 30s hadoop-hdfs in the patch failed. +1 javadoc 1m 38s the patch passed -1 unit 0m 55s hadoop-common in the patch failed. -1 unit 0m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 60m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832628/HADOOP-13061.15.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8c9888ce93bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96b1266 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-compile-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10731/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Thank you so much for checking! Build seems to be failed due to missing configuration keys in CodecUtil. Could you check it?

        And one question.

         //TODO:rs-legacy should be handled differently.
        

        Does it mean the issue which will be solved in HADOOP-13685?

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Thank you so much for checking! Build seems to be failed due to missing configuration keys in CodecUtil . Could you check it? And one question. //TODO:rs-legacy should be handled differently. Does it mean the issue which will be solved in HADOOP-13685 ?
        Hide
        drankye Kai Zheng added a comment -

        Updated the patch adding the missed changes.

        Show
        drankye Kai Zheng added a comment - Updated the patch adding the missed changes.
        Hide
        drankye Kai Zheng added a comment -

        Does it mean the issue which will be solved in HADOOP-13685?

        Nope. rs-legacy is a different erasure codec but to be implemented using the corresponding RS legacy raw coder. We can fire a new issue to address this, considering this refactoring is already quite large.

        Show
        drankye Kai Zheng added a comment - Does it mean the issue which will be solved in HADOOP-13685 ? Nope. rs-legacy is a different erasure codec but to be implemented using the corresponding RS legacy raw coder. We can fire a new issue to address this, considering this refactoring is already quite large.
        Hide
        andrew.wang Andrew Wang added a comment -

        Somewhat unrelated question related to rs-legacy: it's from the Facebook HDFS-RAID implementation right? My recollection is that we added rs-legacy to support migration from HDFS-RAID to the HDFS-7285 implementation without copying.

        However, as it is right now, that zero-copy migration is not possible. HDFS-RAID is not striped, it's at the block level. Also, HDFS-RAID combines an entire directory of files, whereas HDFS-7285 is on a single file. So, I don't think we can natively handle HDFS-RAID files without a lot more work.

        Kai Zheng, if this is accurate, do you think it's still worth keeping the rs-legacy codec around?

        Show
        andrew.wang Andrew Wang added a comment - Somewhat unrelated question related to rs-legacy: it's from the Facebook HDFS-RAID implementation right? My recollection is that we added rs-legacy to support migration from HDFS-RAID to the HDFS-7285 implementation without copying. However, as it is right now, that zero-copy migration is not possible. HDFS-RAID is not striped, it's at the block level. Also, HDFS-RAID combines an entire directory of files, whereas HDFS-7285 is on a single file. So, I don't think we can natively handle HDFS-RAID files without a lot more work. Kai Zheng , if this is accurate, do you think it's still worth keeping the rs-legacy codec around?
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng
        Ah sorry I attached wrong JIRA ticket. I intended to paste HADOOP-13665. But anyway it seems unrelated with your explanation. Thanks.

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Ah sorry I attached wrong JIRA ticket. I intended to paste HADOOP-13665 . But anyway it seems unrelated with your explanation. Thanks.
        Hide
        drankye Kai Zheng added a comment -

        Hi Andrew Wang,

        Thanks for your thoughts! Yes the legacy coder is from HDFS-RAID implementation and we have it when starting with our HDFS-EC development at the very beginning. It's a good question to ask whether we should keep and maintain the coder or not. In the past I have discussed with Zhe Zhang for some times and my preference would be to keep the coder and make the codec work if it wouldn't involve too much overhead, for some reasons like: 1) having the coder surely doesn't mean we can migrate the HDFS-RAID file system data directly but it's possible with some quick-written tools using the coder. The coder logic doesn't couple with HDFS specific (either HDFS-RAID blocks or HDFS-EC strip) and what it can do is to encode/decode a group of input buffers (and thus a group of blocks if repeatedly called). 2) for performance comparison. AFAIK HDFS-RAID wasn't rare to be mentioned/discussed when talking about HDFS erasure coding things. 3) it'd be a good sample to illustrate that even for the most often mentioned RS algorithm, it's good to have different implementation and codecs for it. 4) if we don't want to use it in HDFS side, it's ok because all the coder/codec logics are in Hadoop common side. I'm wondering if it's good to consider that, Hadoop erasure coder/codec framework can develop independently and be used elsewhere.

        When I said we implement a new erasure codec for rs-legacy, it doesn't mean a lots of work since we already have the underlying raw coder implementations. It means to be consistent as we did for the xor, rs-default and hhxor codecs. The codec doesn't have to be used by HDFS or we can ignore it in HDFS side at all.

        Sound good? Thanks.

        Show
        drankye Kai Zheng added a comment - Hi Andrew Wang , Thanks for your thoughts! Yes the legacy coder is from HDFS-RAID implementation and we have it when starting with our HDFS-EC development at the very beginning. It's a good question to ask whether we should keep and maintain the coder or not. In the past I have discussed with Zhe Zhang for some times and my preference would be to keep the coder and make the codec work if it wouldn't involve too much overhead, for some reasons like: 1) having the coder surely doesn't mean we can migrate the HDFS-RAID file system data directly but it's possible with some quick-written tools using the coder. The coder logic doesn't couple with HDFS specific (either HDFS-RAID blocks or HDFS-EC strip) and what it can do is to encode/decode a group of input buffers (and thus a group of blocks if repeatedly called). 2) for performance comparison. AFAIK HDFS-RAID wasn't rare to be mentioned/discussed when talking about HDFS erasure coding things. 3) it'd be a good sample to illustrate that even for the most often mentioned RS algorithm, it's good to have different implementation and codecs for it. 4) if we don't want to use it in HDFS side, it's ok because all the coder/codec logics are in Hadoop common side. I'm wondering if it's good to consider that, Hadoop erasure coder/codec framework can develop independently and be used elsewhere. When I said we implement a new erasure codec for rs-legacy, it doesn't mean a lots of work since we already have the underlying raw coder implementations. It means to be consistent as we did for the xor, rs-default and hhxor codecs. The codec doesn't have to be used by HDFS or we can ignore it in HDFS side at all. Sound good? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
        0 mvndep 1m 32s Maven dependency ordering for branch
        +1 mvninstall 6m 41s trunk passed
        +1 compile 6m 47s trunk passed
        +1 checkstyle 1m 29s trunk passed
        +1 mvnsite 1m 47s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 2s trunk passed
        +1 javadoc 1m 26s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        -1 mvninstall 0m 42s hadoop-hdfs in the patch failed.
        -1 compile 1m 52s root in the patch failed.
        -1 javac 1m 52s root in the patch failed.
        +1 checkstyle 1m 32s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218)
        -1 mvnsite 0m 45s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 29s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 28s hadoop-hdfs in the patch failed.
        +1 javadoc 1m 29s the patch passed
        -1 unit 8m 9s hadoop-common in the patch failed.
        -1 unit 0m 52s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        66m 50s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832805/HADOOP-13061.16.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3ba8e75716bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b84c489
        Default Java 1.8.0_101
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-compile-root.txt
        mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 26s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch -1 mvninstall 0m 42s hadoop-hdfs in the patch failed. -1 compile 1m 52s root in the patch failed. -1 javac 1m 52s root in the patch failed. +1 checkstyle 1m 32s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218) -1 mvnsite 0m 45s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 28s hadoop-hdfs in the patch failed. +1 javadoc 1m 29s the patch passed -1 unit 8m 9s hadoop-common in the patch failed. -1 unit 0m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 66m 50s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832805/HADOOP-13061.16.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3ba8e75716bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b84c489 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-compile-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10736/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Hi Kai Sasaki could you help proceed with the following updates? Please let me know if anything I can help. Thanks!

        Show
        drankye Kai Zheng added a comment - Hi Kai Sasaki could you help proceed with the following updates? Please let me know if anything I can help. Thanks!
        Hide
        lewuathe Kai Sasaki added a comment -

        Kai Zheng Sure, I can. Thanks.

        Show
        lewuathe Kai Sasaki added a comment - Kai Zheng Sure, I can. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 8m 15s trunk passed
        +1 compile 8m 23s trunk passed
        +1 checkstyle 1m 36s trunk passed
        +1 mvnsite 2m 5s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 23s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 45s the patch passed
        +1 compile 8m 1s the patch passed
        +1 javac 8m 1s the patch passed
        +1 checkstyle 1m 36s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218)
        +1 mvnsite 2m 3s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 45s the patch passed
        +1 javadoc 1m 35s the patch passed
        -1 unit 8m 16s hadoop-common in the patch failed.
        -1 unit 65m 0s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        142m 51s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833111/HADOOP-13061.17.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 10919a2f50e4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 901eca0
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 15s trunk passed +1 compile 8m 23s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 45s the patch passed +1 compile 8m 1s the patch passed +1 javac 8m 1s the patch passed +1 checkstyle 1m 36s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218) +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 35s the patch passed -1 unit 8m 16s hadoop-common in the patch failed. -1 unit 65m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 142m 51s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833111/HADOOP-13061.17.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 10919a2f50e4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 901eca0 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10761/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Kai Sasaki for the update! Just find one issue, looks like the failure TestCommonConfigurationFields is related:

        java.lang.AssertionError: core-default.xml has 1 properties missing in  class org.apache.hadoop.fs.CommonConfigurationKeys  class org.apache.hadoop.fs.CommonConfigurationKeysPublic  class org.apache.hadoop.fs.local.LocalConfigKeys  class org.apache.hadoop.fs.ftp.FtpConfigKeys  class org.apache.hadoop.ha.SshFenceByTcpPort  class org.apache.hadoop.security.LdapGroupsMapping  class org.apache.hadoop.ha.ZKFailoverController  class org.apache.hadoop.security.ssl.SSLFactory  class org.apache.hadoop.security.CompositeGroupsMapping  class org.apache.hadoop.io.erasurecode.rawcoder.CoderUtil
        
        	at org.junit.Assert.fail(Assert.java:88)
        	at org.junit.Assert.assertTrue(Assert.java:41)
        	at org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:563)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:497)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
        	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
        	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:497)
        	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
        
        Show
        drankye Kai Zheng added a comment - Thanks Kai Sasaki for the update! Just find one issue, looks like the failure TestCommonConfigurationFields is related: java.lang.AssertionError: core-default.xml has 1 properties missing in class org.apache.hadoop.fs.CommonConfigurationKeys class org.apache.hadoop.fs.CommonConfigurationKeysPublic class org.apache.hadoop.fs.local.LocalConfigKeys class org.apache.hadoop.fs.ftp.FtpConfigKeys class org.apache.hadoop.ha.SshFenceByTcpPort class org.apache.hadoop.security.LdapGroupsMapping class org.apache.hadoop.ha.ZKFailoverController class org.apache.hadoop.security.ssl.SSLFactory class org.apache.hadoop.security.CompositeGroupsMapping class org.apache.hadoop.io.erasurecode.rawcoder.CoderUtil at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:563) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
        Hide
        drankye Kai Zheng added a comment -

        Another one is, we don't need to have this configuration because it's only for internal and very specific to raw coder implementation.

          public static final String IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_KEY =
              "io.erasurecode.codec.allow-change-inputs";
          public static final boolean IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_DEFAULT
              = false;
        
        Show
        drankye Kai Zheng added a comment - Another one is, we don't need to have this configuration because it's only for internal and very specific to raw coder implementation. public static final String IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_KEY = "io.erasurecode.codec.allow-change-inputs" ; public static final boolean IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_DEFAULT = false ;
        Hide
        drankye Kai Zheng added a comment -

        +1 once above two are addressed. Andrew Wang might also give a look? Thanks.

        Show
        drankye Kai Zheng added a comment - +1 once above two are addressed. Andrew Wang might also give a look? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
        0 mvndep 1m 39s Maven dependency ordering for branch
        +1 mvninstall 8m 7s trunk passed
        +1 compile 8m 15s trunk passed
        +1 checkstyle 1m 42s trunk passed
        +1 mvnsite 2m 18s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 3m 32s trunk passed
        +1 javadoc 1m 35s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 1m 44s the patch passed
        +1 compile 8m 9s the patch passed
        +1 javac 8m 9s the patch passed
        +1 checkstyle 1m 43s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218)
        +1 mvnsite 2m 12s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 5s the patch passed
        +1 javadoc 1m 41s the patch passed
        +1 unit 9m 6s hadoop-common in the patch passed.
        +1 unit 60m 40s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        142m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13061
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833271/HADOOP-13061.18.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 46a642489451 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0a85d07
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 8m 7s trunk passed +1 compile 8m 15s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 2m 18s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 32s trunk passed +1 javadoc 1m 35s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 8m 9s the patch passed +1 javac 8m 9s the patch passed +1 checkstyle 1m 43s root: The patch generated 0 new + 194 unchanged - 24 fixed = 194 total (was 218) +1 mvnsite 2m 12s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 5s the patch passed +1 javadoc 1m 41s the patch passed +1 unit 9m 6s hadoop-common in the patch passed. +1 unit 60m 40s hadoop-hdfs in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 142m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13061 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833271/HADOOP-13061.18.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 46a642489451 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a85d07 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        If it's fine with Kai, it's fine with me. One question (which we can address in a follow-on), do we need any doc or core-default.xml updates to go along with this change? If so, a release note would also be nice.

        Show
        andrew.wang Andrew Wang added a comment - If it's fine with Kai, it's fine with me. One question (which we can address in a follow-on), do we need any doc or core-default.xml updates to go along with this change? If so, a release note would also be nice.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Andrew for the confirm. This refactors ErasureCoder part and HDFS is still using the low level part RawErasureCoder. Even given this, it would take some time to happen for HDFS to switch to ErasureCoder API (for example, to use hhxor codec) for stability consideration. We don't need to change core-default.xml at this stage for this work.

        Show
        drankye Kai Zheng added a comment - Thanks Andrew for the confirm. This refactors ErasureCoder part and HDFS is still using the low level part RawErasureCoder . Even given this, it would take some time to happen for HDFS to switch to ErasureCoder API (for example, to use hhxor codec) for stability consideration. We don't need to change core-default.xml at this stage for this work.
        Hide
        drankye Kai Zheng added a comment -

        The latest patch LGTM and +1. Committed to trunk branch.

        Show
        drankye Kai Zheng added a comment - The latest patch LGTM and +1. Committed to trunk branch.
        Hide
        drankye Kai Zheng added a comment -

        Uploaded the committed revision (with a minor cleaning) in case needed.

        Show
        drankye Kai Zheng added a comment - Uploaded the committed revision (with a minor cleaning) in case needed.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10629 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10629/)
        HADOOP-13061. Refactor erasure coders. Contributed by Kai Sasaki (kai.zheng: rev c023c748869063fb67d14ea996569c42578d1cea)

        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/DummyErasureCodec.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractHHErasureCodingStep.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/DummyErasureDecoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureDecoder.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHErasureCodingStep.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/package-info.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XORErasureEncoder.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/codec/TestHHXORErasureCodec.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureEncoder.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/HHXORErasureCodec.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncodingStep.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/AbstractErasureCodec.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/XORErasureCodec.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/RSErasureCodec.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/ErasureCodec.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/CoderUtil.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecodingStep.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodeConstants.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodecOptions.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XORErasureDecoder.java
        • (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/package-info.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/DummyErasureEncoder.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10629 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10629/ ) HADOOP-13061 . Refactor erasure coders. Contributed by Kai Sasaki (kai.zheng: rev c023c748869063fb67d14ea996569c42578d1cea) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCodingStep.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureEncodingStep.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/DummyErasureCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureCoder.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractHHErasureCodingStep.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/DummyErasureDecoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecodingStep.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureDecoder.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHErasureCodingStep.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/package-info.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XORErasureEncoder.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/codec/TestHHXORErasureCodec.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureEncoder.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/ErasureDecoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/HHXORErasureCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncodingStep.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/AbstractErasureCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/XORErasureCodec.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureEncoder.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/RSErasureCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/codec/ErasureCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/CoderUtil.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCodingStep.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecodingStep.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodeConstants.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodecOptions.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/XORErasureDecoder.java (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureCoder.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/package-info.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/DummyErasureEncoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java

          People

          • Assignee:
            lewuathe Kai Sasaki
            Reporter:
            lirui Rui Li
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development