Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2499

Include Kafka Message Key in Event Header, Updated Comments

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I believe it would be useful to include a message's key in the event header, as users will often want to partition based on this key when ingesting into a sink.

      Example:

      sfdc.sinks.hdfs.hdfs.path = /data/testing/kafka_flume/dataset=%{key}/year=%Y/month=%m/day=%d/
      

      In this case, the first part of the partition key is the Kafka message key, which will ensure messages are organized by key.

      I also updated some of the comments and cleaned up the code slightly.

      Cheers,
      Ricky

      1. FLUME-2499.diff
        6 kB
        Ricky Saltzer
      2. FLUME-2499.diff
        12 kB
        Ricky Saltzer

        Activity

        Hide
        hshreedharan Hari Shreedharan added a comment -

        Ricky:

        The patch does not apply. Could you please rebase the patch?

        Show
        hshreedharan Hari Shreedharan added a comment - Ricky: The patch does not apply. Could you please rebase the patch?
        Hide
        rickysaltzer Ricky Saltzer added a comment -

        Yes, I'll do that as soon as I get time. Thanks for letting me know

        Ricky

        Show
        rickysaltzer Ricky Saltzer added a comment - Yes, I'll do that as soon as I get time. Thanks for letting me know Ricky
        Hide
        rickysaltzer Ricky Saltzer added a comment -

        Fresh off the rebase. Let me know if there are any problems.

        Show
        rickysaltzer Ricky Saltzer added a comment - Fresh off the rebase. Let me know if there are any problems.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Running tests and committing.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Running tests and committing.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 622faa6f8812cd751dfad22deaa21bf0aa613111 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=622faa6 ]

        FLUME-2499. Include Kafka Message Key in Event Headers.

        (Ricky Saltzer via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 622faa6f8812cd751dfad22deaa21bf0aa613111 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=622faa6 ] FLUME-2499 . Include Kafka Message Key in Event Headers. (Ricky Saltzer via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 02973a98a223c4b72126b6db3b1b6c1dbdc9239a in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=02973a9 ]

        FLUME-2499. Include Kafka Message Key in Event Headers.

        (Ricky Saltzer via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 02973a98a223c4b72126b6db3b1b6c1dbdc9239a in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=02973a9 ] FLUME-2499 . Include Kafka Message Key in Event Headers. (Ricky Saltzer via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Ricky!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Ricky!
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #678 (See https://builds.apache.org/job/flume-trunk/678/)
        FLUME-2499. Include Kafka Message Key in Event Headers. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=622faa6f8812cd751dfad22deaa21bf0aa613111)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #678 (See https://builds.apache.org/job/flume-trunk/678/ ) FLUME-2499 . Include Kafka Message Key in Event Headers. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=622faa6f8812cd751dfad22deaa21bf0aa613111 ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #37 (See https://builds.apache.org/job/Flume-trunk-hbase-98/37/)
        FLUME-2499. Include Kafka Message Key in Event Headers. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=622faa6f8812cd751dfad22deaa21bf0aa613111)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #37 (See https://builds.apache.org/job/Flume-trunk-hbase-98/37/ ) FLUME-2499 . Include Kafka Message Key in Event Headers. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=622faa6f8812cd751dfad22deaa21bf0aa613111 ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java

          People

          • Assignee:
            rickysaltzer Ricky Saltzer
            Reporter:
            rickysaltzer Ricky Saltzer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development