-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/
-----------------------------------------------------------

Review request for Flume.

Summary
-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.
https://issues.apache.org/jira/browse/FLUME-1117

Diffs

flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing
-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/#review6901
-----------------------------------------------------------

I think we should log a separate jira to address cleanly integrating the Avro serializer with HDFS sink.
Got a few minor comments, mainly calling flush/sync of the serializer and the stream is bit inconsistent. Rest look fine to me.

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
<https://reviews.apache.org/r/4708/#comment15339>

Do we need another outputStream flush after seriallizer.flush() ?

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
<https://reviews.apache.org/r/4708/#comment15340>

should it call serializer flush and beforeClose here ?

• Prasad

On 2012-04-12 13:06:04, Mike Percy wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/4708/

-----------------------------------------------------------

(Updated 2012-04-12 13:06:04)

Review request for Flume.

Summary

-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.

https://issues.apache.org/jira/browse/FLUME-1117

Diffs

-----

flume-ng-core/pom.xml 37fb112

flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing

-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/#review6928
-----------------------------------------------------------

I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

• Hari

On 2012-04-12 13:06:04, Mike Percy wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/4708/

-----------------------------------------------------------

(Updated 2012-04-12 13:06:04)

Review request for Flume.

Summary

-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.

https://issues.apache.org/jira/browse/FLUME-1117

Diffs

-----

flume-ng-core/pom.xml 37fb112

flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing

-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/
-----------------------------------------------------------

(Updated 2012-04-16 04:09:32.452242)

Review request for Flume.

Changes
-------

1. Added missing callbacks per feedback from Prasad
2. Clarified interface semantics in Javadoc and made callback implementations consistent with it
3. Calling flush() then sync() for HDFS output (per convo w/ HDFS devs)

Summary
-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.
https://issues.apache.org/jira/browse/FLUME-1117

Diffs (updated)

flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing
-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

On 2012-04-16 01:49:48, Hari Shreedharan wrote:

> I have one general question about this. I am not entirely sure that the serializer should handle the stream directly. I think it would be better if we simply return the serialized event as a byte array from the EventSerializer, which the class that deals with the output streams(the sinks etc) can deal with directly. This would in general remove the requirement of the serializer component to have a bunch of functions which are not associated with serializing at all, but with writing/flushing etc of the output stream. The serializer should simply return a serialized event which the sink can write to the stream. The serializer should not really be concerned about flushing the stream etc.

>

> The event serializer in my opinion should be simple. It knows the schema and simply returns the byte array which can be written.

Hey - I agree with you that implementations should not be calling flush() on the output stream. I've incorporated that feedback in the latest patch (removed that from the BodyTextEventSerializer and documented it in the interface javadocs).

The main reason that flush() exists is so that serializer implementations can buffer output. This allows us to implement things like block compression, as well as batching writes on transaction boundaries. We write to a stream instead of returning a byte array for the same reason.

• Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/#review6928
-----------------------------------------------------------

On 2012-04-16 04:09:32, Mike Percy wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/4708/

-----------------------------------------------------------

(Updated 2012-04-16 04:09:32)

Review request for Flume.

Summary

-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.

https://issues.apache.org/jira/browse/FLUME-1117

Diffs

-----

flume-ng-core/pom.xml 37fb112

flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing

-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 224

> <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line224>

>

> should it call serializer flush and beforeClose here ?

Thanks for the catch! Added that.

On 2012-04-12 21:51:03, Prasad Mujumdar wrote:

> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java, line 146

> <https://reviews.apache.org/r/4708/diff/1/?file=101449#file101449line146>

>

> Do we need another outputStream flush after seriallizer.flush() ?

I've just clarified the interface contract. So yes, now outputStream.flush() should always follow serializer.flush() in order to get durability semantics.

• Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/#review6901
-----------------------------------------------------------

On 2012-04-16 04:09:32, Mike Percy wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/4708/

-----------------------------------------------------------

(Updated 2012-04-16 04:09:32)

Review request for Flume.

Summary

-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.

https://issues.apache.org/jira/browse/FLUME-1117

Diffs

-----

flume-ng-core/pom.xml 37fb112

flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

pom.xml 3a9bc42

Diff: https://reviews.apache.org/r/4708/diff

Testing

-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/
-----------------------------------------------------------

(Updated 2012-04-16 06:40:32.734329)

Review request for Flume.

Changes
-------

I accidentally a rebase

Summary
-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.
https://issues.apache.org/jira/browse/FLUME-1117

Diffs (updated)

flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d
flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3
flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66
flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION
flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47
pom.xml 32320d8

Diff: https://reviews.apache.org/r/4708/diff

Testing
-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4708/#review6938
-----------------------------------------------------------

Ship it!

lgtm

• Prasad

On 2012-04-16 06:40:32, Mike Percy wrote:

-----------------------------------------------------------

This is an automatically generated e-mail. To reply, visit:

https://reviews.apache.org/r/4708/

-----------------------------------------------------------

(Updated 2012-04-16 06:40:32)

Review request for Flume.

Summary

-------

Patch with support for Avro Container File format.

This addresses bug FLUME-1117.

https://issues.apache.org/jira/browse/FLUME-1117

Diffs

-----

flume-ng-core/pom.xml 37fb112

flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java c8b953d

flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java a12afd3

flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java a2f4f66

flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java PRE-CREATION

flume-ng-core/src/test/resources/syslog_event.avsc PRE-CREATION

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java fdad75b

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java c8e1df9

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 39a4456

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9f28d82

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 49a62df

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 2e5470e

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 73d3284

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java fb61092

flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 5a6ad47

pom.xml 32320d8

Diff: https://reviews.apache.org/r/4708/diff

Testing

-------

Unit tests pass. New unit tests added for new functionality.

Thanks,

Mike

Clicked the wrong license button

Patch committed to trunk.
Thanks Mike!

Integrated in flume-trunk #172 (See https://builds.apache.org/job/flume-trunk/172/)
FLUME-1117: Support output to files in Avro container format
(Mike Percy via Prasad Mujumdar) (Revision 1326827)

Result = SUCCESS
prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1326827
Files :

• /incubator/flume/trunk/flume-ng-core/pom.xml
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/EventFormatter.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/TextDelimitedOutputFormatter.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/AbstractAvroEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/BodyTextEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/serialization/FlumeEventAvroEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
• /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization
• /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestBodyTextEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestFlumeEventAvroEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/TestSyslogAvroEventSerializer.java
• /incubator/flume/trunk/flume-ng-core/src/test/resources/syslog_event.avsc
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
• /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
• /incubator/flume/trunk/pom.xml

Hi Mike,

I am using trunk and trying to persist events in avro data file format.
This is how i configured the serializer

agent.sinks.hdfsSink1.type = hdfs
agent.sinks.hdfsSink1.hdfs.fileType = DataStream
agent.sinks.hdfsSink1.hdfs.path = hdfs://nn_host/dir
agent.sinks.hdfsSink1.serializer = avro_event 
(or) 
agent.sinks.hdfsSink1.serializer = org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder 

I could not see the avro data file in hdfs. Am i missing any configurations specific to avro data file in serializer?

Thanks.
Mubarak

Hi Mubarak,

What build of FlumeNG are you using? Also, is any data showing up in HDFS? What format is it in? Are you getting any error messages in your flume.log?

Here is an example config file I am using. This should work for you, in FlumeNG on trunk. It's using a Syslog source so you may want to change the source.

–
agent.sources = svc_0_src
agent.channels = svc_0_chan
agent.sinks = svc_0_sink

agent.channels.svc_0_chan.type = memory
agent.channels.svc_0_chan.capacity = 100000
agent.channels.svc_0_chan.transactionCapacity = 1000

agent.sources.svc_0_src.type = org.apache.flume.source.SyslogTcpSource
agent.sources.svc_0_src.port = 10001
agent.sources.svc_0_src.channels = svc_0_chan

agent.sinks.svc_0_sink.type = hdfs
agent.sinks.svc_0_sink.hdfs.path = hdfs://xxxxxx.cloudera.com/flume/20120427/flow0
agent.sinks.svc_0_sink.hdfs.fileType = DataStream
agent.sinks.svc_0_sink.hdfs.rollInterval = 300
agent.sinks.svc_0_sink.hdfs.rollSize = 0
agent.sinks.svc_0_sink.hdfs.rollCount = 0
agent.sinks.svc_0_sink.hdfs.batchSize = 1000
agent.sinks.svc_0_sink.hdfs.txnEventMax = 1000
agent.sinks.svc_0_sink.serializer = avro_event
agent.sinks.svc_0_sink.serializer.compressionCodec = snappy
agent.sinks.svc_0_sink.serializer.syncIntervalBytes = 2048000
agent.sinks.svc_0_sink.channel = svc_0_chan
–

Please let me know whether this works for you.

Thanks,
Mike

Thanks Mike. It works now.