Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2781

A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.0
    • 1.7.0
    • None
    • When Flume writes to a channel defined as parseAsFlumeEvent=false, use text instead of Avro

    Description

      When a Kafka channel is configured as parseAsFlumeEvent=false, the channel will read events from the topic as text instead of serialized Avro Flume events.
      This is useful so Flume can read from an existing Kafka topic, where other Kafka clients publish as text.

      However, if you use a Flume source on that channel, it will still write the events as Avro so it will create an inconsistency and those events will fail to be read correctly.

      Also, this would allow a Flume source to write to a Kafka channel and any Kafka subscriber to listen to Flume events passing through without binary dependencies.

      Attachments

        1. FLUME-2781.patch
          4 kB
          Gonzalo Herreros

        Activity

          People

            gherreros Gonzalo Herreros
            gherreros Gonzalo Herreros
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m