Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2852

Kafka Source/Sink should optionally read/write Flume records

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      Added ability to preserve event headers when using Kafka Sink and Kafka Source together by using Avro serialisation (useFlumeEventFormat property).

      Description

      Currently the Kafka Sink only writes the event body to Kafka rather than an Avro Datum. This works fine when being used with a Kafka Source, or when being used with Kafka Channel, however it does mean that any Flume headers are lost when transported via Kafka.

      Request is to implement an equivalent of the Kafka Channel's parseAsFlumeEvent parameter to the sink/source so that optionally Avro Datums can be read from and written to Kafka.

        Attachments

        1. FLUME-2852.patch
          34 kB
          Tristan Stevens
        2. FLUME-2852v4.patch
          28 kB
          Tristan Stevens
        3. FLUME-2852v5.patch
          29 kB
          Tristan Stevens

          Issue Links

            Activity

              People

              • Assignee:
                tmgstev Tristan Stevens
                Reporter:
                tmgstev Tristan Stevens
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: