Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The "rs-default" codec has been renamed to simply "rs" for simplicity. Previous configuration keys like "io.erasurecode.codec.rs-default" have also been renamed to match.

      Description

      I think it's confusing (and verbose) that our RS code is named "rs-default" instead of just "rs", partially because we also have the idea of setting a default implementation for each codec. Let's rename it for simplicity.

      This is an incompatible change since it affects some configuration keys.

      1. HDFS-11427.001.patch
        32 kB
        Andrew Wang
      2. HDFS-11427.002.patch
        31 kB
        Andrew Wang

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11302 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11302/)
        HDFS-11427. Rename rs-default to rs. (wang: rev c1a52b04d0cc5ad5c86ae93043655f313386f7f9)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodeConstants.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecoder.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureEncoder.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedOutputStreamWithFailure.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureDecoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncoder.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedOutputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedInputStream.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11302 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11302/ ) HDFS-11427 . Rename rs-default to rs. (wang: rev c1a52b04d0cc5ad5c86ae93043655f313386f7f9) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ErasureCodeConstants.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureEncoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedOutputStreamWithFailure.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCodecRawCoderMapping.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/RSErasureDecoder.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncoder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestUnsetAndChangeDirectoryEcPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRSDefault10x4StripedInputStream.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestRSErasureCoder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the review Wei-chiu! Committed to trunk.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the review Wei-chiu! Committed to trunk.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Andrew Wang for the patch. Most of the changes are mechanical.

        The failed tests are unrelated. I've double checked and there's no mentioning of rs-default or RS_DEFAULT in the code, so I think we are good. +1.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Andrew Wang for the patch. Most of the changes are mechanical. The failed tests are unrelated. I've double checked and there's no mentioning of rs-default or RS_DEFAULT in the code, so I think we are good. +1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
        0 mvndep 1m 44s Maven dependency ordering for branch
        +1 mvninstall 13m 43s trunk passed
        +1 compile 13m 27s trunk passed
        +1 checkstyle 1m 58s trunk passed
        +1 mvnsite 2m 14s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 29s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 38s the patch passed
        +1 compile 12m 56s the patch passed
        +1 javac 12m 56s the patch passed
        +1 checkstyle 2m 5s the patch passed
        +1 mvnsite 2m 0s the patch passed
        +1 mvneclipse 0m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 3s The patch has no ill-formed XML file.
        +1 findbugs 3m 32s the patch passed
        +1 javadoc 1m 35s the patch passed
        -1 unit 8m 56s hadoop-common in the patch failed.
        -1 unit 68m 35s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        142m 59s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11427
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854327/HDFS-11427.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 7bfcc3c8edf9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 132f758
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18432/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18432/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18432/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18432/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 1m 44s Maven dependency ordering for branch +1 mvninstall 13m 43s trunk passed +1 compile 13m 27s trunk passed +1 checkstyle 1m 58s trunk passed +1 mvnsite 2m 14s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 12m 56s the patch passed +1 javac 12m 56s the patch passed +1 checkstyle 2m 5s the patch passed +1 mvnsite 2m 0s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 findbugs 3m 32s the patch passed +1 javadoc 1m 35s the patch passed -1 unit 8m 56s hadoop-common in the patch failed. -1 unit 68m 35s hadoop-hdfs in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 142m 59s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11427 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854327/HDFS-11427.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7bfcc3c8edf9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132f758 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18432/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18432/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18432/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18432/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Rebased patch attached, Kai Zheng / Wei-Chiu Chuang could you help review? Thanks!

        Show
        andrew.wang Andrew Wang added a comment - Rebased patch attached, Kai Zheng / Wei-Chiu Chuang could you help review? Thanks!
        Hide
        andrew.wang Andrew Wang added a comment -

        Realized I typo'd the JIRA summary, I meant "rs-default" not "rs-legacy".

        Show
        andrew.wang Andrew Wang added a comment - Realized I typo'd the JIRA summary, I meant "rs-default" not "rs-legacy".
        Hide
        aw Allen Wittenauer added a comment -

        History has shown that naming anything with "default" is extremely problematic when the default needs to change.

        Show
        aw Allen Wittenauer added a comment - History has shown that naming anything with "default" is extremely problematic when the default needs to change.
        Hide
        andrew.wang Andrew Wang added a comment -

        Depends on HDFS-11426, so not marking patch available for now.

        Show
        andrew.wang Andrew Wang added a comment - Depends on HDFS-11426 , so not marking patch available for now.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development