Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3081

KafkaSource outputs offset in header

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • 1.7.0, 1.8.0
    • None
    • None
    • None
    • Adding message offset to KafkaSource headers.
    • Patch

    Description

      In FLUME-2799 the original request included (amongst others) Kafka offset to be part of the message headers from the KafkaSource. However the ticket was resolved without it. In affected versions the possible headers are:

      • topic
      • partition
      • key
      • timestamp (agent JVM system time)

      I propose to add offset in order to make a traceable system key (topic+partition+offset) in order to synchronise between several consumers (something necessary for my case).

      Attachments

        1. FLUME-3081-0.patch
          2 kB
          Viktor Gerdin

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. FLUME-2799 Kafka Source - Message Offset and Partition add to headers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              viktor.gerdin Viktor Gerdin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: