Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1, 0.24.0
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Tags:
      compression

      Description

      According to several benchmark sites, LZ4 seems to overtake other fast compression algorithms, especially in the decompression speed area. The interface is also trivial to integrate (http://code.google.com/p/lz4/source/browse/trunk/lz4.h) and there is no license issue.

      1. HADOOP-7657.lz4.v1.patch
        58 kB
        Binglin Chang
      2. HADOOP-7657.lz4.v2.patch
        60 kB
        Binglin Chang
      3. hadoop-7657.v3.txt
        60 kB
        Todd Lipcon

        Issue Links

          Activity

          Gavin made changes -
          Link This issue relates to HADOOP-8266 [ HADOOP-8266 ]
          Gavin made changes -
          Link This issue relates to HADOOP-8266 [ HADOOP-8266 ]
          Joep Rottinghuis made changes -
          Link This issue relates HADOOP-8266 [ HADOOP-8266 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Harsh J made changes -
          Link This issue relates to ACCUMULO-351 [ ACCUMULO-351 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #959 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/959/)
          HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #959 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/959/ ) HADOOP-7975 . Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/926/)
          HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #926 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/926/ ) HADOOP-7975 . Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1562/)
          HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1562/ ) HADOOP-7975 . Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1544 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1544/)
          HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1544 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1544/ ) HADOOP-7975 . Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1617/)
          HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1617 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1617/ ) HADOOP-7975 . Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1231627 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Harsh J made changes -
          Link This issue is superceded by HADOOP-7975 [ HADOOP-7975 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/931/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #131 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/131/)
          Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #131 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/131/ ) Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #111 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/111/)
          Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #111 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/111/ ) Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #898 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/898/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #315 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/315/)
          Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #315 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/315/ ) Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1472/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1472/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #314 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/314/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #314 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/314/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #292 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/292/)
          Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #292 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/292/ ) Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1449 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1449/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1449 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1449/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1522 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1522/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1522 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1522/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220314 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #303 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/303/)
          Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #303 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/303/ ) Amend previous commit of HADOOP-7657 (LZ4 compression). Fix a mistaken #include todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220313 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12507815/hadoop-7657.v3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/483//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/483//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507815/hadoop-7657.v3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 5 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/483//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/483//console This message is automatically generated.
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 0.23.1, 0.24.0 [ 12318884, 12317652 ] 0.24.0, 0.23.1 [ 12317652, 12318884 ]
          Assignee Binglin Chang [ decster ]
          Fix Version/s 0.24.0 [ 12317652 ]
          Fix Version/s 0.23.1 [ 12318884 ]
          Resolution Fixed [ 1 ]
          Hide
          Todd Lipcon added a comment -

          Committed for 0.23.1 and trunk. Thanks, Binglin!

          Show
          Todd Lipcon added a comment - Committed for 0.23.1 and trunk. Thanks, Binglin!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #302 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/302/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #302 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/302/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #291 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/291/)
          HADOOP-7657. Add support for LZ4 compression. Contributed by Binglin Chang.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #291 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/291/ ) HADOOP-7657 . Add support for LZ4 compression. Contributed by Binglin Chang. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1220312 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/LICENSE.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/package-info.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/Makefile.am /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
          Todd Lipcon made changes -
          Attachment hadoop-7657.v3.txt [ 12507815 ]
          Hide
          Todd Lipcon added a comment -

          Reattaching same patch, just updated to the newest rev (r43) from svn (fixes an endian bug). I'll commit this momentarily. Thanks, Binglin!

          Show
          Todd Lipcon added a comment - Reattaching same patch, just updated to the newest rev (r43) from svn (fixes an endian bug). I'll commit this momentarily. Thanks, Binglin!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12507765/HADOOP-7657.lz4.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/482//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/482//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507765/HADOOP-7657.lz4.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 5 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/482//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/482//console This message is automatically generated.
          Binglin Chang made changes -
          Attachment HADOOP-7657.lz4.v2.patch [ 12507765 ]
          Hide
          Binglin Chang added a comment -

          Thanks for review and comments, Todd!
          This patch add license notice to LICENSE.txt and svn revision info to
          hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c

          Also put revision info here:

          Path: lz4.c
          Name: lz4.c
          URL: http://lz4.googlecode.com/svn/trunk/lz4.c
          Repository Root: http://lz4.googlecode.com/svn
          Repository UUID: 650e7d94-2a16-8b24-b05c-7c0b3f6821cd
          Revision: 42
          Node Kind: file
          Schedule: normal
          Last Changed Author: yann.collet.73@gmail.com
          Last Changed Rev: 42
          Last Changed Date: 2011-12-09 21:59:59 +0800 (Fri, 09 Dec 2011)
          Text Last Updated: 2011-12-12 01:31:50 +0800 (Mon, 12 Dec 2011)
          Checksum: 636803a29da03eb68dbf1bd2c4451a41

          Show
          Binglin Chang added a comment - Thanks for review and comments, Todd! This patch add license notice to LICENSE.txt and svn revision info to hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c Also put revision info here: Path: lz4.c Name: lz4.c URL: http://lz4.googlecode.com/svn/trunk/lz4.c Repository Root: http://lz4.googlecode.com/svn Repository UUID: 650e7d94-2a16-8b24-b05c-7c0b3f6821cd Revision: 42 Node Kind: file Schedule: normal Last Changed Author: yann.collet.73@gmail.com Last Changed Rev: 42 Last Changed Date: 2011-12-09 21:59:59 +0800 (Fri, 09 Dec 2011) Text Last Updated: 2011-12-12 01:31:50 +0800 (Mon, 12 Dec 2011) Checksum: 636803a29da03eb68dbf1bd2c4451a41
          Hide
          Todd Lipcon added a comment -

          Actually, two small comments:

          • Can you please indicate in lz4.c which SVN revision or release it is copied from on the main repository? This will help if we want to update it in the future.
          • Can you please add text like the following to LICENSE.txt in the hadoop-common project root?
            For src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c:
            
            /*
               LZ4 - Fast LZ compression algorithm
               Copyright (C) 2011, Yann Collet.
               BSD License
            
               Redistribution and use in source and binary forms, with or without
               modification, are permitted provided that the following conditions are
               met:
              
                   * Redistributions of source code must retain the above copyright
               notice, this list of conditions and the following disclaimer.
                   * Redistributions in binary form must reproduce the above
               copyright notice, this list of conditions and the following disclaimer
               in the documentation and/or other materials provided with the
               distribution.
              
               THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
               "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
               LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
               A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
               OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
               SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
               LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
               DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
               THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
               (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
               OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
            */
            

            so that we comply correctly with the restriction that binary distributions will include the license text and copyright?

          Show
          Todd Lipcon added a comment - Actually, two small comments: Can you please indicate in lz4.c which SVN revision or release it is copied from on the main repository? This will help if we want to update it in the future. Can you please add text like the following to LICENSE.txt in the hadoop-common project root? For src/main/ native /src/org/apache/hadoop/io/compress/lz4/lz4.c: /* LZ4 - Fast LZ compression algorithm Copyright (C) 2011, Yann Collet. BSD License Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ so that we comply correctly with the restriction that binary distributions will include the license text and copyright?
          Hide
          Todd Lipcon added a comment -

          This patch looks good to me - straightforward translation of the Snappy code. +1. I'll commit this in a few days if no one else has any comments.

          Show
          Todd Lipcon added a comment - This patch looks good to me - straightforward translation of the Snappy code. +1. I'll commit this in a few days if no one else has any comments.
          Hide
          Bert Sanders added a comment -

          Thanks Chang.
          Your patch gets excellent results. This is even better than expected.
          Here are some figures from a test system :

          FP.LOG :
          ZLIB: [b 1M] bytes 20617071 -> 1451989 07.1% comp 55.3 MB/s uncomp 355.0 MB/s
          LZ4: [b 1M] bytes 20617071 -> 2412432 11.7% comp 859.9 MB/s uncomp 1874.9 MB/s
          SNAPPY: [b 4M] bytes 20617071 -> 3033623 14.7% comp 606.6 MB/s uncomp 1145.3 MB/s

          Show
          Bert Sanders added a comment - Thanks Chang. Your patch gets excellent results. This is even better than expected. Here are some figures from a test system : FP.LOG : ZLIB: [b 1M] bytes 20617071 -> 1451989 07.1% comp 55.3 MB/s uncomp 355.0 MB/s LZ4: [b 1M] bytes 20617071 -> 2412432 11.7% comp 859.9 MB/s uncomp 1874.9 MB/s SNAPPY: [b 4M] bytes 20617071 -> 3033623 14.7% comp 606.6 MB/s uncomp 1145.3 MB/s
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12507659/HADOOP-7657.lz4.v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/479//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/479//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507659/HADOOP-7657.lz4.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 5 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/479//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/479//console This message is automatically generated.
          Binglin Chang made changes -
          Attachment HADOOP-7657.lz4.v1.patch [ 12507659 ]
          Hide
          Binglin Chang added a comment -

          Lz4 java codec, just copy & paste from SnappyCodec.

          Show
          Binglin Chang added a comment - Lz4 java codec, just copy & paste from SnappyCodec.
          Binglin Chang made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317652, 12318884 ]
          Hide
          Binglin Chang added a comment -

          testdata/alice29.txt :
          ZLIB: [b 1M] bytes 152089 -> 54404 35.8% comp 12.4 MB/s uncomp 189.3 MB/s
          LZ4: [b 1M] bytes 152089 -> 90735 59.7% comp 188.3 MB/s uncomp 1055.6 MB/s
          SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 133.0 MB/s uncomp 594.3 MB/s
          testdata/asyoulik.txt :
          ZLIB: [b 1M] bytes 125179 -> 48897 39.1% comp 11.3 MB/s uncomp 177.4 MB/s
          LZ4: [b 1M] bytes 125179 -> 79026 63.1% comp 180.7 MB/s uncomp 1062.6 MB/s
          SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 130.0 MB/s uncomp 568.6 MB/s
          testdata/baddata1.snappy :
          ZLIB: [b 1M] bytes 27512 -> 22941 83.4% comp 24.0 MB/s uncomp 113.8 MB/s
          LZ4: [b 1M] bytes 27512 -> 26486 96.3% comp 500.4 MB/s uncomp 3963.1 MB/s
          SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 387.4 MB/s uncomp 3445.4 MB/s
          testdata/baddata2.snappy :
          ZLIB: [b 1M] bytes 27483 -> 23023 83.8% comp 24.1 MB/s uncomp 113.0 MB/s
          LZ4: [b 1M] bytes 27483 -> 26633 96.9% comp 549.6 MB/s uncomp 4310.0 MB/s
          SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 423.3 MB/s uncomp 3647.4 MB/s
          testdata/baddata3.snappy :
          ZLIB: [b 1M] bytes 28384 -> 23718 83.6% comp 23.9 MB/s uncomp 112.6 MB/s
          LZ4: [b 1M] bytes 28384 -> 27529 97.0% comp 542.0 MB/s uncomp 4256.5 MB/s
          SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 342.9 MB/s uncomp 3005.0 MB/s
          testdata/cp.html :
          ZLIB: [b 1M] bytes 24603 -> 7961 32.4% comp 27.8 MB/s uncomp 198.8 MB/s
          LZ4: [b 1M] bytes 24603 -> 12039 48.9% comp 327.8 MB/s uncomp 1561.8 MB/s
          SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 195.6 MB/s uncomp 1002.1 MB/s
          testdata/fields.c :
          ZLIB: [b 1M] bytes 11150 -> 3122 28.0% comp 29.0 MB/s uncomp 237.6 MB/s
          LZ4: [b 1M] bytes 11150 -> 5285 47.4% comp 453.1 MB/s uncomp 1356.9 MB/s
          SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 224.1 MB/s uncomp 990.1 MB/s
          testdata/geo.protodata :
          ZLIB: [b 1M] bytes 118588 -> 15131 12.8% comp 56.0 MB/s uncomp 396.9 MB/s
          LZ4: [b 1M] bytes 118588 -> 19862 16.7% comp 627.9 MB/s uncomp 2526.5 MB/s
          SNAPPY: [b 4M] bytes 118588 -> 27459 23.2% comp 422.5 MB/s uncomp 1640.4 MB/s
          testdata/grammar.lsp :
          ZLIB: [b 1M] bytes 3721 -> 1222 32.8% comp 30.9 MB/s uncomp 208.9 MB/s
          LZ4: [b 1M] bytes 3721 -> 1947 52.3% comp 461.7 MB/s uncomp 1556.7 MB/s
          SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 281.0 MB/s uncomp 1118.8 MB/s
          testdata/house.jpg :
          ZLIB: [b 1M] bytes 126958 -> 126513 99.6% comp 24.7 MB/s uncomp 224.6 MB/s
          LZ4: [b 1M] bytes 126958 -> 127272 100.2% comp 2339.8 MB/s uncomp 7389.2 MB/s
          SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2866.3 MB/s uncomp 14627.9 MB/s
          testdata/html :
          ZLIB: [b 1M] bytes 102400 -> 13699 13.4% comp 44.6 MB/s uncomp 356.4 MB/s
          LZ4: [b 1M] bytes 102400 -> 21835 21.3% comp 501.6 MB/s uncomp 1990.6 MB/s
          SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 356.6 MB/s uncomp 1381.3 MB/s
          testdata/html_x_4 :
          ZLIB: [b 1M] bytes 409600 -> 53367 13.0% comp 41.4 MB/s uncomp 356.1 MB/s
          LZ4: [b 1M] bytes 409600 -> 85864 21.0% comp 507.6 MB/s uncomp 1962.6 MB/s
          SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 350.9 MB/s uncomp 1346.3 MB/s
          testdata/kennedy.xls :
          ZLIB: [b 1M] bytes 1029744 -> 203992 19.8% comp 18.8 MB/s uncomp 305.2 MB/s
          LZ4: [b 1M] bytes 1029744 -> 375713 36.5% comp 416.4 MB/s uncomp 1665.8 MB/s
          SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 304.5 MB/s uncomp 879.3 MB/s
          testdata/kppkn.gtb :
          ZLIB: [b 1M] bytes 184320 -> 38751 21.0% comp 8.1 MB/s uncomp 248.7 MB/s
          LZ4: [b 1M] bytes 184320 -> 84897 46.1% comp 325.8 MB/s uncomp 933.0 MB/s
          SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 190.9 MB/s uncomp 690.2 MB/s
          testdata/lcet10.txt :
          ZLIB: [b 1M] bytes 426754 -> 144904 34.0% comp 12.8 MB/s uncomp 196.0 MB/s
          LZ4: [b 1M] bytes 426754 -> 237316 55.6% comp 191.6 MB/s uncomp 1050.0 MB/s
          SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 138.7 MB/s uncomp 621.7 MB/s
          testdata/mapreduce-osdi-1.pdf :
          ZLIB: [b 1M] bytes 94330 -> 74928 79.4% comp 28.8 MB/s uncomp 181.6 MB/s
          LZ4: [b 1M] bytes 94330 -> 77760 82.4% comp 794.6 MB/s uncomp 4056.2 MB/s
          SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 761.5 MB/s uncomp 3864.8 MB/s
          testdata/plrabn12.txt :
          ZLIB: [b 1M] bytes 481861 -> 195261 40.5% comp 9.6 MB/s uncomp 175.4 MB/s
          LZ4: [b 1M] bytes 481861 -> 323373 67.1% comp 172.4 MB/s uncomp 997.3 MB/s
          SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 122.3 MB/s uncomp 518.8 MB/s
          testdata/ptt5 :
          ZLIB: [b 1M] bytes 513216 -> 56465 11.0% comp 31.9 MB/s uncomp 331.6 MB/s
          LZ4: [b 1M] bytes 513216 -> 86327 16.8% comp 659.1 MB/s uncomp 1793.4 MB/s
          SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 488.2 MB/s uncomp 1342.0 MB/s
          testdata/sum :
          ZLIB: [b 1M] bytes 38240 -> 12990 34.0% comp 14.8 MB/s uncomp 186.6 MB/s
          LZ4: [b 1M] bytes 38240 -> 18916 49.5% comp 317.7 MB/s uncomp 1582.3 MB/s
          SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 195.5 MB/s uncomp 853.2 MB/s
          testdata/urls.10K :
          ZLIB: [b 1M] bytes 702087 -> 222613 31.7% comp 23.9 MB/s uncomp 207.0 MB/s
          LZ4: [b 1M] bytes 702087 -> 329450 46.9% comp 306.6 MB/s uncomp 1448.9 MB/s
          SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 209.3 MB/s uncomp 948.3 MB/s
          testdata/xargs.1 :
          ZLIB: [b 1M] bytes 4227 -> 1736 41.1% comp 31.0 MB/s uncomp 197.2 MB/s
          LZ4: [b 1M] bytes 4227 -> 2671 63.2% comp 401.6 MB/s uncomp 1459.2 MB/s
          SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 237.4 MB/s uncomp 950.0 MB/s

          The speed really rocks...

          Show
          Binglin Chang added a comment - testdata/alice29.txt : ZLIB: [b 1M] bytes 152089 -> 54404 35.8% comp 12.4 MB/s uncomp 189.3 MB/s LZ4: [b 1M] bytes 152089 -> 90735 59.7% comp 188.3 MB/s uncomp 1055.6 MB/s SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 133.0 MB/s uncomp 594.3 MB/s testdata/asyoulik.txt : ZLIB: [b 1M] bytes 125179 -> 48897 39.1% comp 11.3 MB/s uncomp 177.4 MB/s LZ4: [b 1M] bytes 125179 -> 79026 63.1% comp 180.7 MB/s uncomp 1062.6 MB/s SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 130.0 MB/s uncomp 568.6 MB/s testdata/baddata1.snappy : ZLIB: [b 1M] bytes 27512 -> 22941 83.4% comp 24.0 MB/s uncomp 113.8 MB/s LZ4: [b 1M] bytes 27512 -> 26486 96.3% comp 500.4 MB/s uncomp 3963.1 MB/s SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 387.4 MB/s uncomp 3445.4 MB/s testdata/baddata2.snappy : ZLIB: [b 1M] bytes 27483 -> 23023 83.8% comp 24.1 MB/s uncomp 113.0 MB/s LZ4: [b 1M] bytes 27483 -> 26633 96.9% comp 549.6 MB/s uncomp 4310.0 MB/s SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 423.3 MB/s uncomp 3647.4 MB/s testdata/baddata3.snappy : ZLIB: [b 1M] bytes 28384 -> 23718 83.6% comp 23.9 MB/s uncomp 112.6 MB/s LZ4: [b 1M] bytes 28384 -> 27529 97.0% comp 542.0 MB/s uncomp 4256.5 MB/s SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 342.9 MB/s uncomp 3005.0 MB/s testdata/cp.html : ZLIB: [b 1M] bytes 24603 -> 7961 32.4% comp 27.8 MB/s uncomp 198.8 MB/s LZ4: [b 1M] bytes 24603 -> 12039 48.9% comp 327.8 MB/s uncomp 1561.8 MB/s SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 195.6 MB/s uncomp 1002.1 MB/s testdata/fields.c : ZLIB: [b 1M] bytes 11150 -> 3122 28.0% comp 29.0 MB/s uncomp 237.6 MB/s LZ4: [b 1M] bytes 11150 -> 5285 47.4% comp 453.1 MB/s uncomp 1356.9 MB/s SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 224.1 MB/s uncomp 990.1 MB/s testdata/geo.protodata : ZLIB: [b 1M] bytes 118588 -> 15131 12.8% comp 56.0 MB/s uncomp 396.9 MB/s LZ4: [b 1M] bytes 118588 -> 19862 16.7% comp 627.9 MB/s uncomp 2526.5 MB/s SNAPPY: [b 4M] bytes 118588 -> 27459 23.2% comp 422.5 MB/s uncomp 1640.4 MB/s testdata/grammar.lsp : ZLIB: [b 1M] bytes 3721 -> 1222 32.8% comp 30.9 MB/s uncomp 208.9 MB/s LZ4: [b 1M] bytes 3721 -> 1947 52.3% comp 461.7 MB/s uncomp 1556.7 MB/s SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 281.0 MB/s uncomp 1118.8 MB/s testdata/house.jpg : ZLIB: [b 1M] bytes 126958 -> 126513 99.6% comp 24.7 MB/s uncomp 224.6 MB/s LZ4: [b 1M] bytes 126958 -> 127272 100.2% comp 2339.8 MB/s uncomp 7389.2 MB/s SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2866.3 MB/s uncomp 14627.9 MB/s testdata/html : ZLIB: [b 1M] bytes 102400 -> 13699 13.4% comp 44.6 MB/s uncomp 356.4 MB/s LZ4: [b 1M] bytes 102400 -> 21835 21.3% comp 501.6 MB/s uncomp 1990.6 MB/s SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 356.6 MB/s uncomp 1381.3 MB/s testdata/html_x_4 : ZLIB: [b 1M] bytes 409600 -> 53367 13.0% comp 41.4 MB/s uncomp 356.1 MB/s LZ4: [b 1M] bytes 409600 -> 85864 21.0% comp 507.6 MB/s uncomp 1962.6 MB/s SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 350.9 MB/s uncomp 1346.3 MB/s testdata/kennedy.xls : ZLIB: [b 1M] bytes 1029744 -> 203992 19.8% comp 18.8 MB/s uncomp 305.2 MB/s LZ4: [b 1M] bytes 1029744 -> 375713 36.5% comp 416.4 MB/s uncomp 1665.8 MB/s SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 304.5 MB/s uncomp 879.3 MB/s testdata/kppkn.gtb : ZLIB: [b 1M] bytes 184320 -> 38751 21.0% comp 8.1 MB/s uncomp 248.7 MB/s LZ4: [b 1M] bytes 184320 -> 84897 46.1% comp 325.8 MB/s uncomp 933.0 MB/s SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 190.9 MB/s uncomp 690.2 MB/s testdata/lcet10.txt : ZLIB: [b 1M] bytes 426754 -> 144904 34.0% comp 12.8 MB/s uncomp 196.0 MB/s LZ4: [b 1M] bytes 426754 -> 237316 55.6% comp 191.6 MB/s uncomp 1050.0 MB/s SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 138.7 MB/s uncomp 621.7 MB/s testdata/mapreduce-osdi-1.pdf : ZLIB: [b 1M] bytes 94330 -> 74928 79.4% comp 28.8 MB/s uncomp 181.6 MB/s LZ4: [b 1M] bytes 94330 -> 77760 82.4% comp 794.6 MB/s uncomp 4056.2 MB/s SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 761.5 MB/s uncomp 3864.8 MB/s testdata/plrabn12.txt : ZLIB: [b 1M] bytes 481861 -> 195261 40.5% comp 9.6 MB/s uncomp 175.4 MB/s LZ4: [b 1M] bytes 481861 -> 323373 67.1% comp 172.4 MB/s uncomp 997.3 MB/s SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 122.3 MB/s uncomp 518.8 MB/s testdata/ptt5 : ZLIB: [b 1M] bytes 513216 -> 56465 11.0% comp 31.9 MB/s uncomp 331.6 MB/s LZ4: [b 1M] bytes 513216 -> 86327 16.8% comp 659.1 MB/s uncomp 1793.4 MB/s SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 488.2 MB/s uncomp 1342.0 MB/s testdata/sum : ZLIB: [b 1M] bytes 38240 -> 12990 34.0% comp 14.8 MB/s uncomp 186.6 MB/s LZ4: [b 1M] bytes 38240 -> 18916 49.5% comp 317.7 MB/s uncomp 1582.3 MB/s SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 195.5 MB/s uncomp 853.2 MB/s testdata/urls.10K : ZLIB: [b 1M] bytes 702087 -> 222613 31.7% comp 23.9 MB/s uncomp 207.0 MB/s LZ4: [b 1M] bytes 702087 -> 329450 46.9% comp 306.6 MB/s uncomp 1448.9 MB/s SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 209.3 MB/s uncomp 948.3 MB/s testdata/xargs.1 : ZLIB: [b 1M] bytes 4227 -> 1736 41.1% comp 31.0 MB/s uncomp 197.2 MB/s LZ4: [b 1M] bytes 4227 -> 2671 63.2% comp 401.6 MB/s uncomp 1459.2 MB/s SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 237.4 MB/s uncomp 950.0 MB/s The speed really rocks...
          Hide
          Tim Broberg added a comment -

          It would be interesting to see unit test results for lz4 as well when you get to it.

          Show
          Tim Broberg added a comment - It would be interesting to see unit test results for lz4 as well when you get to it.
          Hide
          Binglin Chang added a comment -

          I'm afraid it's not typical, Terasort input data has lots of repeated strings:
          .t^#|v$2\ 0AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEEFFFFFFFFFFGGGGGGGGGGHHHHHHHH
          75@~?'WdUF 1IIIIIIIIIIJJJJJJJJJJKKKKKKKKKKLLLLLLLLLLMMMMMMMMMMNNNNNNNNNNOOOOOOOOOOPPPPPPPP
          w[o||:N&H, 2QQQQQQQQQQRRRRRRRRRRSSSSSSSSSSTTTTTTTTTTUUUUUUUUUUVVVVVVVVVVWWWWWWWWWWXXXXXXXX
          ^Eu)<n#kdP 3YYYYYYYYYYZZZZZZZZZZAAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEEFFFFFFFF

          Normally log data, web page collection and structured data(table, record set) have higher compression ratios(which is very common in Hadoop usecase), about 20%-30% in my experience, plain text not so good, around 50%.

          Just for reference, here are the test results from snappy unittest:
          BM_ZFlat/0 270628 270443 737 361.1MB/s html (23.57 %)
          BM_ZFlat/1 3215660 3214530 100 208.3MB/s urls (50.89 %)
          BM_ZFlat/2 43917 43870 4533 2.7GB/s jpg (99.88 %)
          BM_ZFlat/3 123527 123369 1593 729.2MB/s pdf (82.13 %)
          BM_ZFlat/4 1098281 1097779 181 355.8MB/s html4 (23.55 %)
          BM_ZFlat/5 117902 117862 1669 199.1MB/s cp (48.12 %)
          BM_ZFlat/6 49592 49591 3824 214.4MB/s c (42.40 %)
          BM_ZFlat/7 12214 12202 14461 290.8MB/s lsp (48.37 %)
          BM_ZFlat/8 2898230 2895690 100 339.1MB/s xls (41.34 %)
          BM_ZFlat/9 1053508 1052727 187 137.8MB/s txt1 (59.81 %)
          BM_ZFlat/10 894211 893702 222 133.6MB/s txt2 (64.07 %)
          BM_ZFlat/11 2811850 2810680 100 144.8MB/s txt3 (57.11 %)
          BM_ZFlat/12 3594620 3592880 100 127.9MB/s txt4 (68.35 %)
          BM_ZFlat/13 991489 990943 194 493.9MB/s bin (18.21 %)
          BM_ZFlat/14 186471 186407 1076 195.6MB/s sum (51.88 %)
          BM_ZFlat/15 17664 17648 10672 228.4MB/s man (59.36 %)
          BM_ZFlat/16 259190 259137 770 436.4MB/s pb (23.15 %)
          BM_ZFlat/17 897617 896724 225 196.0MB/s gaviota (38.27 %)

          There is a test result from http://code.google.com/p/lz4/, using a more standard corpus with compression ratio around 50%.

          Show
          Binglin Chang added a comment - I'm afraid it's not typical, Terasort input data has lots of repeated strings: .t^#|v$2\ 0AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEEFFFFFFFFFFGGGGGGGGGGHHHHHHHH 75@~?'WdUF 1IIIIIIIIIIJJJJJJJJJJKKKKKKKKKKLLLLLLLLLLMMMMMMMMMMNNNNNNNNNNOOOOOOOOOOPPPPPPPP w[o||:N&H, 2QQQQQQQQQQRRRRRRRRRRSSSSSSSSSSTTTTTTTTTTUUUUUUUUUUVVVVVVVVVVWWWWWWWWWWXXXXXXXX ^Eu)<n#kdP 3YYYYYYYYYYZZZZZZZZZZAAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEEFFFFFFFF Normally log data, web page collection and structured data(table, record set) have higher compression ratios(which is very common in Hadoop usecase), about 20%-30% in my experience, plain text not so good, around 50%. Just for reference, here are the test results from snappy unittest: BM_ZFlat/0 270628 270443 737 361.1MB/s html (23.57 %) BM_ZFlat/1 3215660 3214530 100 208.3MB/s urls (50.89 %) BM_ZFlat/2 43917 43870 4533 2.7GB/s jpg (99.88 %) BM_ZFlat/3 123527 123369 1593 729.2MB/s pdf (82.13 %) BM_ZFlat/4 1098281 1097779 181 355.8MB/s html4 (23.55 %) BM_ZFlat/5 117902 117862 1669 199.1MB/s cp (48.12 %) BM_ZFlat/6 49592 49591 3824 214.4MB/s c (42.40 %) BM_ZFlat/7 12214 12202 14461 290.8MB/s lsp (48.37 %) BM_ZFlat/8 2898230 2895690 100 339.1MB/s xls (41.34 %) BM_ZFlat/9 1053508 1052727 187 137.8MB/s txt1 (59.81 %) BM_ZFlat/10 894211 893702 222 133.6MB/s txt2 (64.07 %) BM_ZFlat/11 2811850 2810680 100 144.8MB/s txt3 (57.11 %) BM_ZFlat/12 3594620 3592880 100 127.9MB/s txt4 (68.35 %) BM_ZFlat/13 991489 990943 194 493.9MB/s bin (18.21 %) BM_ZFlat/14 186471 186407 1076 195.6MB/s sum (51.88 %) BM_ZFlat/15 17664 17648 10672 228.4MB/s man (59.36 %) BM_ZFlat/16 259190 259137 770 436.4MB/s pb (23.15 %) BM_ZFlat/17 897617 896724 225 196.0MB/s gaviota (38.27 %) There is a test result from http://code.google.com/p/lz4/ , using a more standard corpus with compression ratio around 50%.
          Hide
          Tim Broberg added a comment -

          I think I'm seeing ~5:1 compression ratios here, is that right?

          Compression performance is generally highly sensitive to input data, decompression even more so.

          Is it at all typical to see 5:1 compression with live Hadoop data?

          If not, is there an appropriate corpus of data that reflects normal usage?

          Show
          Tim Broberg added a comment - I think I'm seeing ~5:1 compression ratios here, is that right? Compression performance is generally highly sensitive to input data, decompression even more so. Is it at all typical to see 5:1 compression with live Hadoop data? If not, is there an appropriate corpus of data that reflects normal usage?
          Hide
          Binglin Chang added a comment -

          This test is all c++ code, I don't implement the java version...

          Show
          Binglin Chang added a comment - This test is all c++ code, I don't implement the java version...
          Hide
          Todd Lipcon added a comment -

          great results - looks like about 15% fsater compression and 60% faster decompression than snappy, along with a bit better ratio?

          Do you have the codec up in a patch somewhere for others to try? Also, I'm assuming you ran the test in a loop several times in the same JVM to let the JIT warm up equally for both algorithms?

          Show
          Todd Lipcon added a comment - great results - looks like about 15% fsater compression and 60% faster decompression than snappy, along with a bit better ratio? Do you have the codec up in a patch somewhere for others to try? Also, I'm assuming you ran the test in a loop several times in the same JVM to let the JIT warm up equally for both algorithms?
          Hide
          Binglin Chang added a comment -

          I have integrated lz4 in native task optimizition(MAPREDUCE-2841), and it did show better results compare to snappy. Here is some test results for terasort input data:

          [ RUN ] Perf.Compressions
          11/12/15 13:04:24 INFO Generate data time: 2.12431s
          11/12/15 13:04:26 INFO memset buffer to prevent missing page time: 2.08189s
          11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.SnappyCodec
          11/12/15 13:04:26 INFO snappy compress origin/compressed time: 0.16029s size: 100.000M/21.944M speed: 623.86M/136.90M
          11/12/15 13:04:26 INFO snappy decompress orig/uncompressed time: 0.05860s size: 21.944M/100.000M speed: 374.49M/1706.57M
          11/12/15 13:04:26 INFO ratio: 0.219
          11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.Lz4Codec
          11/12/15 13:04:27 INFO snappy compress origin/compressed time: 0.13551s size: 100.000M/18.952M speed: 737.96M/139.86M
          11/12/15 13:04:27 INFO snappy decompress orig/uncompressed time: 0.03126s size: 18.952M/100.000M speed: 606.24M/3198.87M
          11/12/15 13:04:27 INFO ratio: 0.190
          [ OK ] Perf.Compressions (4665 ms)
          [----------] 1 test from Perf (4665 ms total)

          Show
          Binglin Chang added a comment - I have integrated lz4 in native task optimizition( MAPREDUCE-2841 ), and it did show better results compare to snappy. Here is some test results for terasort input data: [ RUN ] Perf.Compressions 11/12/15 13:04:24 INFO Generate data time: 2.12431s 11/12/15 13:04:26 INFO memset buffer to prevent missing page time: 2.08189s 11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.SnappyCodec 11/12/15 13:04:26 INFO snappy compress origin/compressed time: 0.16029s size: 100.000M/21.944M speed: 623.86M/136.90M 11/12/15 13:04:26 INFO snappy decompress orig/uncompressed time: 0.05860s size: 21.944M/100.000M speed: 374.49M/1706.57M 11/12/15 13:04:26 INFO ratio: 0.219 11/12/15 13:04:26 INFO org.apache.hadoop.io.compress.Lz4Codec 11/12/15 13:04:27 INFO snappy compress origin/compressed time: 0.13551s size: 100.000M/18.952M speed: 737.96M/139.86M 11/12/15 13:04:27 INFO snappy decompress orig/uncompressed time: 0.03126s size: 18.952M/100.000M speed: 606.24M/3198.87M 11/12/15 13:04:27 INFO ratio: 0.190 [ OK ] Perf.Compressions (4665 ms) [----------] 1 test from Perf (4665 ms total)
          Hide
          Evergreen added a comment -

          Glad you noticed !

          Show
          Evergreen added a comment - Glad you noticed !
          Hide
          Yann Collet added a comment -

          Hi

          I'm the author of LZ4.

          Since it is mentionned in this issue, i feel compelled to provide some more information.
          There are in fact two container modes created for LZ4, but i only cared publishing the first one (stream mode), since there was very little interest expressed in the second one (seekable mode).

          The seekable mode has these properties :

          • Selectable block size (as a power of 2, from 512 bytes to 16 MB; extension reserved up to 1GB)
          • Handle uncompressed blocks
          • Blocks can be forced to start at aligned positions (2 bytes, 4 bytes, 8 bytes, etc.)
          • Compressed file starts with an array which provides the start position of each block
          • Array value can be stored as direct position or block size (the second solution can reduce the size of array)
          • Automatic sizing of array values (16, 32 or 64 bits).
          • File size limited to 256 TB

          Since the array must be written at the beginning of the compressed file, it is not compatible with "pipe mode".

          The only use-case for seekable mode so far has been compressed ISO. So i've not spent much time on it.

          I'm a bit surprised it would be usefull for Hadoop since i expect Handle to handle its own container format, and a probably much adapted one.

          Anyway, since r35 is out, i would advise to use something like 64KB blocks. This way, each block is independant, and can be fetched/stored/duplicated/decoded independantly, in its own thread, CPU, machine or cluster.

          Hope this helps
          Regards

          Show
          Yann Collet added a comment - Hi I'm the author of LZ4. Since it is mentionned in this issue, i feel compelled to provide some more information. There are in fact two container modes created for LZ4, but i only cared publishing the first one (stream mode), since there was very little interest expressed in the second one (seekable mode). The seekable mode has these properties : Selectable block size (as a power of 2, from 512 bytes to 16 MB; extension reserved up to 1GB) Handle uncompressed blocks Blocks can be forced to start at aligned positions (2 bytes, 4 bytes, 8 bytes, etc.) Compressed file starts with an array which provides the start position of each block Array value can be stored as direct position or block size (the second solution can reduce the size of array) Automatic sizing of array values (16, 32 or 64 bits). File size limited to 256 TB Since the array must be written at the beginning of the compressed file, it is not compatible with "pipe mode". The only use-case for seekable mode so far has been compressed ISO. So i've not spent much time on it. I'm a bit surprised it would be usefull for Hadoop since i expect Handle to handle its own container format, and a probably much adapted one. Anyway, since r35 is out, i would advise to use something like 64KB blocks. This way, each block is independant, and can be fetched/stored/duplicated/decoded independantly, in its own thread, CPU, machine or cluster. Hope this helps Regards
          Hide
          Bert Sanders added a comment -

          As far as i understand, LZ4 is just a compression algorithm, and does not define an output format.
          The one proposed in the open-source file "LZ4Demo" is just an example, and obviously made to work with pipe, i.e. unseekable input and output.

          Show
          Bert Sanders added a comment - As far as i understand, LZ4 is just a compression algorithm, and does not define an output format. The one proposed in the open-source file "LZ4Demo" is just an example, and obviously made to work with pipe, i.e. unseekable input and output.
          Hide
          Steve Loughran added a comment -

          Couple of Q's related to Hadoop use

          1. How well can you seek in it, so that when you work against a large file you can start work inside it> .lzo works better than gzip here, for example
          2. how well can you recover from corrupted LZ4 blocks? That is if a 128MB block has lost a 64KB segment due to an HDD problem, is the whole 128MB lost, or can the tooling extract everything other than the compressed areas that the lost 64KB sector straddles.
            issue #1 is important, because if you can't read the data so easily, decompression time matters less. handling file corruption is less critical, but it is something I'm starting to worry about.
          Show
          Steve Loughran added a comment - Couple of Q's related to Hadoop use How well can you seek in it, so that when you work against a large file you can start work inside it> .lzo works better than gzip here, for example how well can you recover from corrupted LZ4 blocks? That is if a 128MB block has lost a 64KB segment due to an HDD problem, is the whole 128MB lost, or can the tooling extract everything other than the compressed areas that the lost 64KB sector straddles. issue #1 is important, because if you can't read the data so easily, decompression time matters less. handling file corruption is less critical, but it is something I'm starting to worry about.
          Hide
          Bert Sanders added a comment -

          As far as i know, it's correct, i've tested it on my system.
          But these figures compare C version of compression algorithms, not Java ones.

          Show
          Bert Sanders added a comment - As far as i know, it's correct, i've tested it on my system. But these figures compare C version of compression algorithms, not Java ones.
          Hide
          Evergreen added a comment -

          Are the benchmark figure at the author's website correct ?
          http://code.google.com/p/lz4/
          Because if they are, it looks like a win.
          Anyone having tried the benchmark program yet ?

          Show
          Evergreen added a comment - Are the benchmark figure at the author's website correct ? http://code.google.com/p/lz4/ Because if they are, it looks like a win. Anyone having tried the benchmark program yet ?
          Hide
          Harsh J added a comment -

          A good article's here (from author) - wherein he directly compares with snappy: http://fastcompression.blogspot.com/2011/06/lz4-improved-performance.html

          Does seem to have an edge.

          Show
          Harsh J added a comment - A good article's here (from author) - wherein he directly compares with snappy: http://fastcompression.blogspot.com/2011/06/lz4-improved-performance.html Does seem to have an edge.
          Bert Sanders created issue -

            People

            • Assignee:
              Binglin Chang
              Reporter:
              Bert Sanders
            • Votes:
              2 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development