Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3100

Support arbitrary header substitution for topic of Kafka

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Kafka Sink supports arbitrary header substitution which replace a substring $

      {A}

      in "kafka.topic" property with a value of header named “A”.
      Using the substitution become possible to dynamically select the topic by combining one or more values of event headers.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #249 (See https://builds.apache.org/job/Flume-trunk-hbase-1/249/)
          FLUME-3100. Support arbitrary header substitution for topic of Kafka (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=83e25691dc5f32d020b122d679b6f124162e4aef)

          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestConstants.java
          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Flume-trunk-hbase-1 #249 (See https://builds.apache.org/job/Flume-trunk-hbase-1/249/ ) FLUME-3100 . Support arbitrary header substitution for topic of Kafka (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=83e25691dc5f32d020b122d679b6f124162e4aef ) (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestConstants.java (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flume/pull/137

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/137
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 83e25691dc5f32d020b122d679b6f124162e4aef in flume's branch refs/heads/trunk from Takafumi Saito
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=83e2569 ]

          FLUME-3100. Support arbitrary header substitution for topic of Kafka Sink

          This patch adds the ability of header substitution n Kafka Sink's
          kafka.topic configuration variable.

          This closes #137.

          Reviewers: Denes Arvay

          (Takafumi Saito via Denes Arvay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 83e25691dc5f32d020b122d679b6f124162e4aef in flume's branch refs/heads/trunk from Takafumi Saito [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=83e2569 ] FLUME-3100 . Support arbitrary header substitution for topic of Kafka Sink This patch adds the ability of header substitution n Kafka Sink's kafka.topic configuration variable. This closes #137. Reviewers: Denes Arvay (Takafumi Saito via Denes Arvay)
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user stakafum opened a pull request:

          https://github.com/apache/flume/pull/137

          FLUME-3100 Support arbitrary header substitution for topic of Kafka

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/stakafum/flume FLUME-3100

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flume/pull/137.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #137


          commit 09d59d761d300aa683bf39a7c58578d1daafdedd
          Author: stakafum <s.takafum@gmail.com>
          Date: 2017-05-25T06:23:44Z

          FLUME-3100 Support arbitrary header substitution for topic of Kafka


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user stakafum opened a pull request: https://github.com/apache/flume/pull/137 FLUME-3100 Support arbitrary header substitution for topic of Kafka You can merge this pull request into a Git repository by running: $ git pull https://github.com/stakafum/flume FLUME-3100 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #137 commit 09d59d761d300aa683bf39a7c58578d1daafdedd Author: stakafum <s.takafum@gmail.com> Date: 2017-05-25T06:23:44Z FLUME-3100 Support arbitrary header substitution for topic of Kafka

            People

            • Assignee:
              Unassigned
              Reporter:
              Takafumi Saito Takafumi Saito
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development