Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3238

TailDir Source + Kafka Sink + Memory Channel - Couldn't identify event source filename

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.8.0
    • Fix Version/s: None
    • Component/s: Kafka Channel
    • Labels:
      None

      Description

      I am using Taildir to consume my log event and send it to kafka sink for further processing before its indexed to a table for any analytics.

      The events received in kafka topic doesn't carry which file the event belongs to and also couldn't trace back to the actual file the event really occurred.

      i tried so many config attribute as per the documentation, but the output doesnt change, it carries the raw event only to kafka topic.

      Did anyone had similar requirement and able to achieve it?

      As per the ticket# https://issues.cloudera.org/browse/FLUME-258 they say its resolved. But doesnt seem to be available in new versions.

       

      I have included my sample config and input files, if anyone tried similar implementation, please let me know how to achieve this.

        Attachments

        1. exmple1.log
          0.3 kB
          r2d2
        2. exmple2.log
          0.5 kB
          r2d2
        3. taildir_flume.conf.txt
          0.7 kB
          r2d2

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ragu86 r2d2
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: