[FLUME-3238] TailDir Source + Kafka Sink + Memory Channel - Couldn't identify event source filename

• Type: Question
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: Kafka Channel
• Assignee: Unassigned
• Reporter: r2d2
• Created: 03/Jun/18 18:27
• Updated: 16/Jul/18 10:44

Description

I am using Taildir to consume my log event and send it to kafka sink for further processing before its indexed to a table for any analytics.

The events received in kafka topic doesn't carry which file the event belongs to and also couldn't trace back to the actual file the event really occurred.

i tried so many config attribute as per the documentation, but the output doesnt change, it carries the raw event only to kafka topic.

Did anyone had similar requirement and able to achieve it?

As per the ticket# https://issues.cloudera.org/browse/FLUME-258 they say its resolved. But doesnt seem to be available in new versions.

 

I have included my sample config and input files, if anyone tried similar implementation, please let me know how to achieve this.