[FLUME-2852] Kafka Source/Sink should optionally read/write Flume records

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: Sinks+Sources
• Release Note: Added ability to preserve event headers when using Kafka Sink and Kafka Source together by using Avro serialisation (useFlumeEventFormat property).
• Assignee: Tristan Stevens
• Reporter: Tristan Stevens
• Created: 08/Dec/15 14:15
• Updated: 18/Jun/16 17:21
• Resolved: 21/Apr/16 20:38

Description

Currently the Kafka Sink only writes the event body to Kafka rather than an Avro Datum. This works fine when being used with a Kafka Source, or when being used with Kafka Channel, however it does mean that any Flume headers are lost when transported via Kafka.

Request is to implement an equivalent of the Kafka Channel's parseAsFlumeEvent parameter to the sink/source so that optionally Avro Datums can be read from and written to Kafka.

Issue Links

• depends upon: FLUME-2820 Support New Kafka APIs
• links to: Review Board