Avro
  1. Avro
  2. AVRO-1522

SortedKeyValueFile Writer should support compression codecs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.6
    • Fix Version/s: 1.7.7
    • Component/s: java
    • Labels:
      None

      Description

      There's no option to set the compression codec on a SortedKeyValueFile.Writer. And since the Writer creates the output file in the constructor and it provides no public access to the underlying DataFileWriter anyway, the user can't set a codec themselves.

      I have a patch that will add this functionality through the SortedKeyValueFile.Writer.Options class. But we might want to consider if it also makes sense to delay creating the data and index DataFileWriters until a create() method is called, and allow public access to these writers so a consumer of the API can modify settings of the underlying writers themselves.

      1. AVRO-1522.patch
        6 kB
        Steven Willis

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in AvroJava #454 (See https://builds.apache.org/job/AvroJava/454/)
        AVRO-1522. Java: Add support for compression codecs to SortedKeyValueFile. Contributed by Steven Willis. (cutting: rev 1601014)

        • /avro/trunk/CHANGES.txt
        • /avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/hadoop/file/SortedKeyValueFile.java
        • /avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/hadoop/file/TestSortedKeyValueFile.java
        Show
        Hudson added a comment - SUCCESS: Integrated in AvroJava #454 (See https://builds.apache.org/job/AvroJava/454/ ) AVRO-1522 . Java: Add support for compression codecs to SortedKeyValueFile. Contributed by Steven Willis. (cutting: rev 1601014) /avro/trunk/CHANGES.txt /avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/hadoop/file/SortedKeyValueFile.java /avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/hadoop/file/TestSortedKeyValueFile.java
        Hide
        Doug Cutting added a comment -

        I committed this. Thanks, Steven!

        Show
        Doug Cutting added a comment - I committed this. Thanks, Steven!
        Hide
        ASF subversion and git services added a comment -

        Commit 1601014 from Doug Cutting in branch 'avro/trunk'
        [ https://svn.apache.org/r1601014 ]

        AVRO-1522. Java: Add support for compression codecs to SortedKeyValueFile. Contributed by Steven Willis.

        Show
        ASF subversion and git services added a comment - Commit 1601014 from Doug Cutting in branch 'avro/trunk' [ https://svn.apache.org/r1601014 ] AVRO-1522 . Java: Add support for compression codecs to SortedKeyValueFile. Contributed by Steven Willis.
        Hide
        Steven Willis added a comment -

        Can I get a code review on this patch? Also, how does one get Jenkins to automatically test this?

        Show
        Steven Willis added a comment - Can I get a code review on this patch? Also, how does one get Jenkins to automatically test this?
        Hide
        Steven Willis added a comment -

        This patch introduces withCodec(String codec), withCodec(CodecFactory codec), and CodecFactory getCodec() methods on the SortedKeyValue.Writer.Options class. And sets the codec on the writer's underlying DataFileWriter with this new setting. Tests are included.

        Note that this patch doesn't set the codec on the index file. Perhaps it should also be set to the same codec, or alternatively, have a separate option (perhaps setIndexCodec() and setDataCodec()).

        Show
        Steven Willis added a comment - This patch introduces withCodec(String codec) , withCodec(CodecFactory codec) , and CodecFactory getCodec() methods on the SortedKeyValue.Writer.Options class. And sets the codec on the writer's underlying DataFileWriter with this new setting. Tests are included. Note that this patch doesn't set the codec on the index file. Perhaps it should also be set to the same codec, or alternatively, have a separate option (perhaps setIndexCodec() and setDataCodec() ).

          People

          • Assignee:
            Steven Willis
            Reporter:
            Steven Willis
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development