Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3081

KafkaSource outputs offset in header

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.7.0, 1.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Adding message offset to KafkaSource headers.
    • Flags:
      Patch

      Description

      In FLUME-2799 the original request included (amongst others) Kafka offset to be part of the message headers from the KafkaSource. However the ticket was resolved without it. In affected versions the possible headers are:

      • topic
      • partition
      • key
      • timestamp (agent JVM system time)

      I propose to add offset in order to make a traceable system key (topic+partition+offset) in order to synchronise between several consumers (something necessary for my case).

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              viktor.gerdin Viktor Gerdin

              Dates

              • Created:
                Updated:

                Issue deployment