Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2499

Include Kafka Message Key in Event Header, Updated Comments

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.5.0.1
    • 1.6.0
    • Sinks+Sources
    • None

    Description

      I believe it would be useful to include a message's key in the event header, as users will often want to partition based on this key when ingesting into a sink.

      Example:

      sfdc.sinks.hdfs.hdfs.path = /data/testing/kafka_flume/dataset=%{key}/year=%Y/month=%m/day=%d/
      

      In this case, the first part of the partition key is the Kafka message key, which will ensure messages are organized by key.

      I also updated some of the comments and cleaned up the code slightly.

      Cheers,
      Ricky

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rickysaltzer Ricky Saltzer
            rickysaltzer Ricky Saltzer
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment