Flume
  1. Flume
  2. FLUME-1117

Support output to files in Avro container format

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Support output of events to a file in Avro container format.

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.
        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs


        flume-ng-core/pom.xml 37fb112
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing
        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/#review6901
        -----------------------------------------------------------

        I think we should log a separate jira to address cleanly integrating the Avro serializer with HDFS sink.
        Got a few minor comments, mainly calling flush/sync of the serializer and the stream is bit inconsistent. Rest look fine to me.

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
        <https://reviews.apache.org/r/4708/#comment15339>

        Do we need another outputStream flush after seriallizer.flush() ?

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
        <https://reviews.apache.org/r/4708/#comment15340>

        should it call serializer flush and beforeClose here ?

        • Prasad

        On 2012-04-12 13:06:04, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4708/

        -----------------------------------------------------------

        (Updated 2012-04-12 13:06:04)

        Review request for Flume.

        Summary

        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.

        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs

        -----

        flume-ng-core/pom.xml 37fb112

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing

        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6901 ----------------------------------------------------------- I think we should log a separate jira to address cleanly integrating the Avro serializer with HDFS sink. Got a few minor comments, mainly calling flush/sync of the serializer and the stream is bit inconsistent. Rest look fine to me. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java < https://reviews.apache.org/r/4708/#comment15339 > Do we need another outputStream flush after seriallizer.flush() ? flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java < https://reviews.apache.org/r/4708/#comment15340 > should it call serializer flush and beforeClose here ? Prasad On 2012-04-12 13:06:04, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-12 13:06:04) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/#review6928
        -----------------------------------------------------------

        I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

        The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

        • Hari

        On 2012-04-12 13:06:04, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4708/

        -----------------------------------------------------------

        (Updated 2012-04-12 13:06:04)

        Review request for Flume.

        Summary

        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.

        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs

        -----

        flume-ng-core/pom.xml 37fb112

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing

        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6928 ----------------------------------------------------------- I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc. The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written. Hari On 2012-04-12 13:06:04, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-12 13:06:04) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/
        -----------------------------------------------------------

        (Updated 2012-04-16 04:09:32.452242)

        Review request for Flume.

        Changes
        -------

        1. Added missing callbacks per feedback from Prasad
        2. Clarified interface semantics in Javadoc and made callback implementations consistent with it
        3. Calling flush() then sync() for HDFS output (per convo w/ HDFS devs)

        Summary
        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.
        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs (updated)


        flume-ng-core/pom.xml 37fb112
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing
        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32.452242) Review request for Flume. Changes ------- 1. Added missing callbacks per feedback from Prasad 2. Clarified interface semantics in Javadoc and made callback implementations consistent with it 3. Calling flush() then sync() for HDFS output (per convo w/ HDFS devs) Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs (updated) flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-04-16 01:49:48, Hari Shreedharan wrote:

        > I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

        >

        > The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

        Hey - I agree with you that implementations should not be calling flush() on the output stream. I've incorporated that feedback in the latest patch (removed that from the BodyTextEventSerializer and documented it in the interface javadocs).

        The main reason that flush() exists is so that serializer implementations can buffer output. This allows us to implement things like block compression, as well as batching writes on transaction boundaries. We write to a stream instead of returning a byte array for the same reason.

        • Mike

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/#review6928
        -----------------------------------------------------------

        On 2012-04-16 04:09:32, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4708/

        -----------------------------------------------------------

        (Updated 2012-04-16 04:09:32)

        Review request for Flume.

        Summary

        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.

        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs

        -----

        flume-ng-core/pom.xml 37fb112

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing

        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-04-16 01:49:48, Hari Shreedharan wrote: > I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc. > > The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written. Hey - I agree with you that implementations should not be calling flush() on the output stream. I've incorporated that feedback in the latest patch (removed that from the BodyTextEventSerializer and documented it in the interface javadocs). The main reason that flush() exists is so that serializer implementations can buffer output. This allows us to implement things like block compression, as well as batching writes on transaction boundaries. We write to a stream instead of returning a byte array for the same reason. Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6928 ----------------------------------------------------------- On 2012-04-16 04:09:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

        > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 224

        > <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line224>

        >

        > should it call serializer flush and beforeClose here ?

        Thanks for the catch! Added that.

        On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

        > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 146

        > <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line146>

        >

        > Do we need another outputStream flush after seriallizer.flush() ?

        I've just clarified the interface contract. So yes, now outputStream.flush() should always follow serializer.flush() in order to get durability semantics.

        • Mike

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/#review6901
        -----------------------------------------------------------

        On 2012-04-16 04:09:32, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4708/

        -----------------------------------------------------------

        (Updated 2012-04-16 04:09:32)

        Review request for Flume.

        Summary

        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.

        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs

        -----

        flume-ng-core/pom.xml 37fb112

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

        pom.xml 3a9bc42

        Diff: https://reviews.apache.org/r/4708/diff

        Testing

        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-04-12 21:51:03, Prasad Mujumdar wrote: > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 224 > < https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line224 > > > should it call serializer flush and beforeClose here ? Thanks for the catch! Added that. On 2012-04-12 21:51:03, Prasad Mujumdar wrote: > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 146 > < https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line146 > > > Do we need another outputStream flush after seriallizer.flush() ? I've just clarified the interface contract. So yes, now outputStream.flush() should always follow serializer.flush() in order to get durability semantics. Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6901 ----------------------------------------------------------- On 2012-04-16 04:09:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/
        -----------------------------------------------------------

        (Updated 2012-04-16 06:40:32.734329)

        Review request for Flume.

        Changes
        -------

        I accidentally a rebase

        Summary
        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.
        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs (updated)


        flume-ng-core/pom.xml 37fb112
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
        pom.xml 32320d8

        Diff: https://reviews.apache.org/r/4708/diff

        Testing
        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 06:40:32.734329) Review request for Flume. Changes ------- I accidentally a rebase Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs (updated) flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 32320d8 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4708/#review6938
        -----------------------------------------------------------

        Ship it!

        lgtm

        • Prasad

        On 2012-04-16 06:40:32, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4708/

        -----------------------------------------------------------

        (Updated 2012-04-16 06:40:32)

        Review request for Flume.

        Summary

        -------

        Patch with support for Avro Container File format.

        This addresses bug FLUME-1117.

        https://issues.apache.org/jira/browse/FLUME-1117

        Diffs

        -----

        flume-ng-core/pom.xml 37fb112

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

        flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

        flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

        flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

        flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

        pom.xml 32320d8

        Diff: https://reviews.apache.org/r/4708/diff

        Testing

        -------

        Unit tests pass. New unit tests added for new functionality.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6938 ----------------------------------------------------------- Ship it! lgtm Prasad On 2012-04-16 06:40:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 06:40:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 32320d8 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
        Hide
        Mike Percy added a comment -

        Clicked the wrong license button

        Show
        Mike Percy added a comment - Clicked the wrong license button
        Hide
        Prasad Mujumdar added a comment -

        Patch committed to trunk.
        Thanks Mike!

        Show
        Prasad Mujumdar added a comment - Patch committed to trunk. Thanks Mike!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #172 (See https://builds.apache.org/job/flume-trunk/172/)
        FLUME-1117: Support output to files in Avro container format
        (Mike Percy via Prasad Mujumdar) (Revision 1326827)

        Result = SUCCESS
        prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1326827
        Files :

        • /incubator/flume/trunk/flume-ng-core/pom.xml
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
        • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization
        • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java
        • /incubator/flume/trunk/flume-ng-core/src/test/resources/syslog_event.avsc
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        • /incubator/flume/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in flume-trunk #172 (See https://builds.apache.org/job/flume-trunk/172/ ) FLUME-1117 : Support output to files in Avro container format (Mike Percy via Prasad Mujumdar) (Revision 1326827) Result = SUCCESS prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1326827 Files : /incubator/flume/trunk/flume-ng-core/pom.xml /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/resources/syslog_event.avsc /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java /incubator/flume/trunk/pom.xml
        Hide
        Mubarak Seyed added a comment -

        Hi Mike,

        I am using trunk and trying to persist events in avro data file format.
        This is how i configured the serializer

        agent.sinks.hdfsSink1.type = hdfs
        agent.sinks.hdfsSink1.hdfs.fileType = DataStream
        agent.sinks.hdfsSink1.hdfs.path = hdfs://nn_host/dir
        agent.sinks.hdfsSink1.serializer = avro_event 
        (or) 
        agent.sinks.hdfsSink1.serializer = org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder 
        

        I could not see the avro data file in hdfs. Am i missing any configurations specific to avro data file in serializer?

        Thanks.
        Mubarak

        Show
        Mubarak Seyed added a comment - Hi Mike, I am using trunk and trying to persist events in avro data file format. This is how i configured the serializer agent.sinks.hdfsSink1.type = hdfs agent.sinks.hdfsSink1.hdfs.fileType = DataStream agent.sinks.hdfsSink1.hdfs.path = hdfs: //nn_host/dir agent.sinks.hdfsSink1.serializer = avro_event (or) agent.sinks.hdfsSink1.serializer = org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder I could not see the avro data file in hdfs. Am i missing any configurations specific to avro data file in serializer? Thanks. Mubarak
        Hide
        Mike Percy added a comment -

        Hi Mubarak,

        What build of FlumeNG are you using? Also, is any data showing up in HDFS? What format is it in? Are you getting any error messages in your flume.log?

        Here is an example config file I am using. This should work for you, in FlumeNG on trunk. It's using a Syslog source so you may want to change the source.


        agent.sources = svc_0_src
        agent.channels = svc_0_chan
        agent.sinks = svc_0_sink

        agent.channels.svc_0_chan.type = memory
        agent.channels.svc_0_chan.capacity = 100000
        agent.channels.svc_0_chan.transactionCapacity = 1000

        agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource
        agent.sources.svc_0_src.port = 10001
        agent.sources.svc_0_src.channels = svc_0_chan

        agent.sinks.svc_0_sink.type = hdfs
        agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxx.cloudera.com/flume/20120427/flow0
        agent.sinks.svc_0_sink.hdfs.fileType = DataStream
        agent.sinks.svc_0_sink.hdfs.rollInterval = 300
        agent.sinks.svc_0_sink.hdfs.rollSize = 0
        agent.sinks.svc_0_sink.hdfs.rollCount = 0
        agent.sinks.svc_0_sink.hdfs.batchSize = 1000
        agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000
        agent.sinks.svc_0_sink.serializer = avro_event
        agent.sinks.svc_0_sink.serializer.compressionCodec = snappy
        agent.sinks.svc_0_sink.serializer.syncIntervalBytes = 2048000
        agent.sinks.svc_0_sink.channel = svc_0_chan

        Please let me know whether this works for you.

        Thanks,
        Mike

        Show
        Mike Percy added a comment - Hi Mubarak, What build of FlumeNG are you using? Also, is any data showing up in HDFS? What format is it in? Are you getting any error messages in your flume.log? Here is an example config file I am using. This should work for you, in FlumeNG on trunk. It's using a Syslog source so you may want to change the source. – agent.sources = svc_0_src agent.channels = svc_0_chan agent.sinks = svc_0_sink agent.channels.svc_0_chan.type = memory agent.channels.svc_0_chan.capacity = 100000 agent.channels.svc_0_chan.transactionCapacity = 1000 agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource agent.sources.svc_0_src.port = 10001 agent.sources.svc_0_src.channels = svc_0_chan agent.sinks.svc_0_sink.type = hdfs agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxx.cloudera.com/flume/20120427/flow0 agent.sinks.svc_0_sink.hdfs.fileType = DataStream agent.sinks.svc_0_sink.hdfs.rollInterval = 300 agent.sinks.svc_0_sink.hdfs.rollSize = 0 agent.sinks.svc_0_sink.hdfs.rollCount = 0 agent.sinks.svc_0_sink.hdfs.batchSize = 1000 agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000 agent.sinks.svc_0_sink.serializer = avro_event agent.sinks.svc_0_sink.serializer.compressionCodec = snappy agent.sinks.svc_0_sink.serializer.syncIntervalBytes = 2048000 agent.sinks.svc_0_sink.channel = svc_0_chan – Please let me know whether this works for you. Thanks, Mike
        Hide
        Mubarak Seyed added a comment -

        Thanks Mike. It works now.

        Show
        Mubarak Seyed added a comment - Thanks Mike. It works now.

          People

          • Assignee:
            Mike Percy
            Reporter:
            Mike Percy
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development