Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2872

Kafka Sink should be able to select which header as the key

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
    • Release Note:
      A new property called keyHeader is added for Kafka Sink. It can be used to specify which header is used as the key of Kafka message. Its default value is "key", so this patch is compatible with Flume 1.6.

      Description

      In Flume 1.6.0, Kafka Sink uses the topic and key properties from the FlumeEvent headers to send events to Kafka. If topic exists in the headers, the event will be sent to that specific topic, overriding the topic configured for the Sink. If key exists in the headers, the key will used by Kafka to partition the data between the topic partitions. Events with same key will be sent to the same partition. If the key is null, events will be sent to random partitions.
      But if Kafka Sink is able to select which header as the key, much convieience will be brought to Flume users.

      1. FLUME-2872-2.patch
        7 kB
        Li Ye
      2. FLUME-2872-1.patch
        7 kB
        Li Ye

        Issue Links

          Activity

          Hide
          l90006322 Li Ye added a comment -

          A new configuration item called keyHeader is added for Kafka Sink. It can be used to specify which header is used as the key of Kafka message. Its default value is "key", so this patch is compatible with Flume 1.6.

          Show
          l90006322 Li Ye added a comment - A new configuration item called keyHeader is added for Kafka Sink. It can be used to specify which header is used as the key of Kafka message. Its default value is "key", so this patch is compatible with Flume 1.6.
          Hide
          l90006322 Li Ye added a comment -

          A new configuration item called keyHeader is added for Kafka Sink. It can be used to specify which header is used as the key of Kafka message. Its default value is "key", so this patch is compatible with Flume 1.6.

          Show
          l90006322 Li Ye added a comment - A new configuration item called keyHeader is added for Kafka Sink. It can be used to specify which header is used as the key of Kafka message. Its default value is "key", so this patch is compatible with Flume 1.6.
          Hide
          l90006322 Li Ye added a comment - - edited

          It was my misoperation to close issue. It should be reopened.

          Show
          l90006322 Li Ye added a comment - - edited It was my misoperation to close issue. It should be reopened.
          Hide
          l90006322 Li Ye added a comment -

          Unused code is deleted.

          Show
          l90006322 Li Ye added a comment - Unused code is deleted.

            People

            • Assignee:
              Unassigned
              Reporter:
              l90006322 Li Ye
            • Votes:
              5 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development