Flume
  1. Flume
  2. FLUME-1117

Support output to files in Avro container format

        Details

        • Type: Improvement Improvement
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: v1.2.0
        • Fix Version/s: v1.2.0
        • Component/s: None
        • Labels:
          None

          Description

          Support output of events to a file in Avro container format.

            Activity

            Descending order - Click to sort in ascending order
            Hide
            Permalink
            Mubarak Seyed added a comment -

            Thanks Mike. It works now.

            Show
            Mubarak Seyed added a comment - Thanks Mike. It works now.
            Hide
            Permalink
            Mike Percy added a comment -

            Hi Mubarak,

            What build of FlumeNG are you using? Also, is any data showing up in HDFS? What format is it in? Are you getting any error messages in your flume.log?

            Here is an example config file I am using. This should work for you, in FlumeNG on trunk. It's using a Syslog source so you may want to change the source.

            –
            agent.sources = svc_0_src
            agent.channels = svc_0_chan
            agent.sinks = svc_0_sink

            agent.channels.svc_0_chan.type = memory
            agent.channels.svc_0_chan.capacity = 100000
            agent.channels.svc_0_chan.transactionCapacity = 1000

            agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource
            agent.sources.svc_0_src.port = 10001
            agent.sources.svc_0_src.channels = svc_0_chan

            agent.sinks.svc_0_sink.type = hdfs
            agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxx.cloudera.com/flume/20120427/flow0
            agent.sinks.svc_0_sink.hdfs.fileType = DataStream
            agent.sinks.svc_0_sink.hdfs.rollInterval = 300
            agent.sinks.svc_0_sink.hdfs.rollSize = 0
            agent.sinks.svc_0_sink.hdfs.rollCount = 0
            agent.sinks.svc_0_sink.hdfs.batchSize = 1000
            agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000
            agent.sinks.svc_0_sink.serializer = avro_event
            agent.sinks.svc_0_sink.serializer.compressionCodec = snappy
            agent.sinks.svc_0_sink.serializer.syncIntervalBytes = 2048000
            agent.sinks.svc_0_sink.channel = svc_0_chan
            –

            Please let me know whether this works for you.

            Thanks,
            Mike

            Show
            Mike Percy added a comment - Hi Mubarak, What build of FlumeNG are you using? Also, is any data showing up in HDFS? What format is it in? Are you getting any error messages in your flume.log? Here is an example config file I am using. This should work for you, in FlumeNG on trunk. It's using a Syslog source so you may want to change the source. – agent.sources = svc_0_src agent.channels = svc_0_chan agent.sinks = svc_0_sink agent.channels.svc_0_chan.type = memory agent.channels.svc_0_chan.capacity = 100000 agent.channels.svc_0_chan.transactionCapacity = 1000 agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource agent.sources.svc_0_src.port = 10001 agent.sources.svc_0_src.channels = svc_0_chan agent.sinks.svc_0_sink.type = hdfs agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxx.cloudera.com/flume/20120427/flow0 agent.sinks.svc_0_sink.hdfs.fileType = DataStream agent.sinks.svc_0_sink.hdfs.rollInterval = 300 agent.sinks.svc_0_sink.hdfs.rollSize = 0 agent.sinks.svc_0_sink.hdfs.rollCount = 0 agent.sinks.svc_0_sink.hdfs.batchSize = 1000 agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000 agent.sinks.svc_0_sink.serializer = avro_event agent.sinks.svc_0_sink.serializer.compressionCodec = snappy agent.sinks.svc_0_sink.serializer.syncIntervalBytes = 2048000 agent.sinks.svc_0_sink.channel = svc_0_chan – Please let me know whether this works for you. Thanks, Mike
            Hide
            Permalink
            Mubarak Seyed added a comment -

            Hi Mike,

            I am using trunk and trying to persist events in avro data file format.
            This is how i configured the serializer

            agent.sinks.hdfsSink1.type = hdfs
agent.sinks.hdfsSink1.hdfs.fileType = DataStream
agent.sinks.hdfsSink1.hdfs.path = hdfs://nn_host/dir
agent.sinks.hdfsSink1.serializer = avro_event 
(or) 
agent.sinks.hdfsSink1.serializer = org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder

            I could not see the avro data file in hdfs. Am i missing any configurations specific to avro data file in serializer?

            Thanks.
            Mubarak

            Show
            Mubarak Seyed added a comment - Hi Mike, I am using trunk and trying to persist events in avro data file format. This is how i configured the serializer agent.sinks.hdfsSink1.type = hdfs agent.sinks.hdfsSink1.hdfs.fileType = DataStream agent.sinks.hdfsSink1.hdfs.path = hdfs: //nn_host/dir agent.sinks.hdfsSink1.serializer = avro_event (or) agent.sinks.hdfsSink1.serializer = org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder I could not see the avro data file in hdfs. Am i missing any configurations specific to avro data file in serializer? Thanks. Mubarak
            Hide
            Permalink
            Hudson added a comment -

            Integrated in flume-trunk #172 (See https://builds.apache.org/job/flume-trunk/172/)
            FLUME-1117: Support output to files in Avro container format
            (Mike Percy via Prasad Mujumdar) (Revision 1326827)

            Result = SUCCESS
            prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1326827
            Files :

            • /incubator/flume/trunk/flume-ng-core/pom.xml
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
            • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization
            • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java
            • /incubator/flume/trunk/flume-ng-core/src/test/resources/syslog_event.avsc
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
            • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
            • /incubator/flume/trunk/pom.xml
            Show
            Hudson added a comment - Integrated in flume-trunk #172 (See https://builds.apache.org/job/flume-trunk/172/ ) FLUME-1117 : Support output to files in Avro container format (Mike Percy via Prasad Mujumdar) (Revision 1326827) Result = SUCCESS prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1326827 Files : /incubator/flume/trunk/flume-ng-core/pom.xml /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java /incubator/flume/trunk/flume-ng-core/src/test/resources/syslog_event.avsc /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java /incubator/flume/trunk/pom.xml
            Hide
            Permalink
            Prasad Mujumdar added a comment -

            Patch committed to trunk.
            Thanks Mike!

            Show
            Prasad Mujumdar added a comment - Patch committed to trunk. Thanks Mike!
            Hide
            Permalink
            Mike Percy added a comment -

            Clicked the wrong license button

            Show
            Mike Percy added a comment - Clicked the wrong license button
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/#review6938
            -----------------------------------------------------------

            Ship it!

            lgtm

            • Prasad

            On 2012-04-16 06:40:32, Mike Percy wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/4708/

            -----------------------------------------------------------

            (Updated 2012-04-16 06:40:32)

            Review request for Flume.

            Summary

            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.

            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            -----

            flume-ng-core/pom.xml 37fb112

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

            pom.xml 32320d8

            Diff: https://reviews.apache.org/r/4708/diff

            Testing

            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6938 ----------------------------------------------------------- Ship it! lgtm Prasad On 2012-04-16 06:40:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 06:40:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 32320d8 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/
            -----------------------------------------------------------

            (Updated 2012-04-16 06:40:32.734329)

            Review request for Flume.

            Changes
            -------

            I accidentally a rebase

            Summary
            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.
            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs (updated)

            flume-ng-core/pom.xml 37fb112
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
            pom.xml 32320d8

            Diff: https://reviews.apache.org/r/4708/diff

            Testing
            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 06:40:32.734329) Review request for Flume. Changes ------- I accidentally a rebase Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs (updated) flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 32320d8 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

            > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 224

            > <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line224>

            >

            > should it call serializer flush and beforeClose here ?

            Thanks for the catch! Added that.

            On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

            > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 146

            > <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line146>

            >

            > Do we need another outputStream flush after seriallizer.flush() ?

            I've just clarified the interface contract. So yes, now outputStream.flush() should always follow serializer.flush() in order to get durability semantics.

            • Mike

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/#review6901
            -----------------------------------------------------------

            On 2012-04-16 04:09:32, Mike Percy wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/4708/

            -----------------------------------------------------------

            (Updated 2012-04-16 04:09:32)

            Review request for Flume.

            Summary

            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.

            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            -----

            flume-ng-core/pom.xml 37fb112

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing

            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - On 2012-04-12 21:51:03, Prasad Mujumdar wrote: > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 224 > < https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line224 > > > should it call serializer flush and beforeClose here ? Thanks for the catch! Added that. On 2012-04-12 21:51:03, Prasad Mujumdar wrote: > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 146 > < https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line146 > > > Do we need another outputStream flush after seriallizer.flush() ? I've just clarified the interface contract. So yes, now outputStream.flush() should always follow serializer.flush() in order to get durability semantics. Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6901 ----------------------------------------------------------- On 2012-04-16 04:09:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            On 2012-04-16 01:49:48, Hari Shreedharan wrote:

            > I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

            >

            > The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

            Hey - I agree with you that implementations should not be calling flush() on the output stream. I've incorporated that feedback in the latest patch (removed that from the BodyTextEventSerializer and documented it in the interface javadocs).

            The main reason that flush() exists is so that serializer implementations can buffer output. This allows us to implement things like block compression, as well as batching writes on transaction boundaries. We write to a stream instead of returning a byte array for the same reason.

            • Mike

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/#review6928
            -----------------------------------------------------------

            On 2012-04-16 04:09:32, Mike Percy wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/4708/

            -----------------------------------------------------------

            (Updated 2012-04-16 04:09:32)

            Review request for Flume.

            Summary

            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.

            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            -----

            flume-ng-core/pom.xml 37fb112

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing

            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - On 2012-04-16 01:49:48, Hari Shreedharan wrote: > I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc. > > The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written. Hey - I agree with you that implementations should not be calling flush() on the output stream. I've incorporated that feedback in the latest patch (removed that from the BodyTextEventSerializer and documented it in the interface javadocs). The main reason that flush() exists is so that serializer implementations can buffer output. This allows us to implement things like block compression, as well as batching writes on transaction boundaries. We write to a stream instead of returning a byte array for the same reason. Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6928 ----------------------------------------------------------- On 2012-04-16 04:09:32, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/
            -----------------------------------------------------------

            (Updated 2012-04-16 04:09:32.452242)

            Review request for Flume.

            Changes
            -------

            1. Added missing callbacks per feedback from Prasad
            2. Clarified interface semantics in Javadoc and made callback implementations consistent with it
            3. Calling flush() then sync() for HDFS output (per convo w/ HDFS devs)

            Summary
            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.
            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs (updated)

            flume-ng-core/pom.xml 37fb112
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing
            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-16 04:09:32.452242) Review request for Flume. Changes ------- 1. Added missing callbacks per feedback from Prasad 2. Clarified interface semantics in Javadoc and made callback implementations consistent with it 3. Calling flush() then sync() for HDFS output (per convo w/ HDFS devs) Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs (updated) flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/#review6928
            -----------------------------------------------------------

            I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

            The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

            • Hari

            On 2012-04-12 13:06:04, Mike Percy wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/4708/

            -----------------------------------------------------------

            (Updated 2012-04-12 13:06:04)

            Review request for Flume.

            Summary

            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.

            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            -----

            flume-ng-core/pom.xml 37fb112

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing

            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6928 ----------------------------------------------------------- I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc. The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written. Hari On 2012-04-12 13:06:04, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-12 13:06:04) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/#review6901
            -----------------------------------------------------------

            I think we should log a separate jira to address cleanly integrating the Avro serializer with HDFS sink.
            Got a few minor comments, mainly calling flush/sync of the serializer and the stream is bit inconsistent. Rest look fine to me.

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
            <https://reviews.apache.org/r/4708/#comment15339>

            Do we need another outputStream flush after seriallizer.flush() ?

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
            <https://reviews.apache.org/r/4708/#comment15340>

            should it call serializer flush and beforeClose here ?

            • Prasad

            On 2012-04-12 13:06:04, Mike Percy wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/4708/

            -----------------------------------------------------------

            (Updated 2012-04-12 13:06:04)

            Review request for Flume.

            Summary

            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.

            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            -----

            flume-ng-core/pom.xml 37fb112

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing

            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/#review6901 ----------------------------------------------------------- I think we should log a separate jira to address cleanly integrating the Avro serializer with HDFS sink. Got a few minor comments, mainly calling flush/sync of the serializer and the stream is bit inconsistent. Rest look fine to me. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java < https://reviews.apache.org/r/4708/#comment15339 > Do we need another outputStream flush after seriallizer.flush() ? flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java < https://reviews.apache.org/r/4708/#comment15340 > should it call serializer flush and beforeClose here ? Prasad On 2012-04-12 13:06:04, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- (Updated 2012-04-12 13:06:04) Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs ----- flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike
            Hide
            Permalink
            jiraposter@reviews.apache.org added a comment -

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/4708/
            -----------------------------------------------------------

            Review request for Flume.

            Summary
            -------

            Patch with support for Avro Container File format.

            This addresses bug FLUME-1117.
            https://issues.apache.org/jira/browse/FLUME-1117

            Diffs

            flume-ng-core/pom.xml 37fb112
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
            flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
            flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
            flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
            flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
            flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
            flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
            pom.xml 3a9bc42

            Diff: https://reviews.apache.org/r/4708/diff

            Testing
            -------

            Unit tests pass. New unit tests added for new functionality.

            Thanks,

            Mike

            Show
            jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4708/ ----------------------------------------------------------- Review request for Flume. Summary ------- Patch with support for Avro Container File format. This addresses bug FLUME-1117 . https://issues.apache.org/jira/browse/FLUME-1117 Diffs flume-ng-core/pom.xml 37fb112 flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3 flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66 flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47 pom.xml 3a9bc42 Diff: https://reviews.apache.org/r/4708/diff Testing ------- Unit tests pass. New unit tests added for new functionality. Thanks, Mike

              People

              • Assignee:
                Mike Percy
                Reporter:
                Mike Percy
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development