Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2868

Kafka Channel partition topic by key

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.6.0
    • Fix Version/s: v1.7.0
    • Component/s: Channel
    • Labels:
      None

      Description

      Add the ability to partition a Kafka channel topic by key. Users can currently partition topics by key using a file/memory channel with a Kafka sink. Although this approach adds unnecessary latency and the Kafka sink drops Flume event headers, which can be undesirable.

        Issue Links

          Activity

          Hide
          jrkinley James Kinley added a comment - - edited

          This has been fixed by FLUME-2823.

          Users can now specify a partition key directly with a Kafka Channel by using an Interceptor to generate a "key" event header.

          Show
          jrkinley James Kinley added a comment - - edited This has been fixed by FLUME-2823 . Users can now specify a partition key directly with a Kafka Channel by using an Interceptor to generate a "key" event header.
          Hide
          tmgstev Tristan Stevens added a comment -

          FLUME-2823 allows you to specify the key, but not the partition. We should do this in the channel and in the sink.

          Show
          tmgstev Tristan Stevens added a comment - FLUME-2823 allows you to specify the key, but not the partition. We should do this in the channel and in the sink.
          Hide
          tmgstev Tristan Stevens added a comment - - edited

          You can specify the key and the Kafka Producer will hash partition based on the key. There should be a separate JIRA to directly specify the partition.

          Show
          tmgstev Tristan Stevens added a comment - - edited You can specify the key and the Kafka Producer will hash partition based on the key. There should be a separate JIRA to directly specify the partition.

            People

            • Assignee:
              Unassigned
              Reporter:
              jrkinley James Kinley
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development