Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3046

Kafka Sink and Source Configuration Improvements

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently the Kafka Source sets the header for the topic. The sink reads this value, rather than the statically defined topic value. We should fix this so that you can either change the topic header that is used, or just choose to prefer the statically defined topic in the sink.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #323 (See https://builds.apache.org/job/Flume-trunk-hbase-1/323/)
          FLUME-3046. Kafka Sink and Source Configuration Improvements (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=54e2728a8e141ee63704018c4497bbe083c0f75f)

          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java
          • (edit) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
          • (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
          • (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #323 (See https://builds.apache.org/job/Flume-trunk-hbase-1/323/ ) FLUME-3046 . Kafka Sink and Source Configuration Improvements (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=54e2728a8e141ee63704018c4497bbe083c0f75f ) (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java (edit) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          Hide
          denes Denes Arvay added a comment -

          Thank you Tristan Stevens for the patch and Attila Simon for the review, I have committed the change.

          Show
          denes Denes Arvay added a comment - Thank you Tristan Stevens for the patch and Attila Simon for the review, I have committed the change.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flume/pull/105

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/105
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 54e2728a8e141ee63704018c4497bbe083c0f75f in flume's branch refs/heads/trunk from Tristan Stevens
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=54e2728 ]

          FLUME-3046. Kafka Sink and Source Configuration Improvements

          This patch fixes the infinite loop between Kafka source and Kafka sink
          by introducing the following configuration parameters in those components:

          • topicHeader in Kafka source to specify the name of the header where it
            stores the topic name where the event comes from.
          • setTopicHeader in Kafka source to control whether the topic name is stored
            in the given header.
          • topicHeader in Kafka sink to configure the name of the header which
            is used to specify in which topic to send the event.
          • allowTopicOverride in Kafka sink to control whether the target topic's name
            can be overridden by the specified header.

          This closes #105

          Reviewers: Attila Simon

          (Tristan Stevens via Denes Arvay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 54e2728a8e141ee63704018c4497bbe083c0f75f in flume's branch refs/heads/trunk from Tristan Stevens [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=54e2728 ] FLUME-3046 . Kafka Sink and Source Configuration Improvements This patch fixes the infinite loop between Kafka source and Kafka sink by introducing the following configuration parameters in those components: topicHeader in Kafka source to specify the name of the header where it stores the topic name where the event comes from. setTopicHeader in Kafka source to control whether the topic name is stored in the given header. topicHeader in Kafka sink to configure the name of the header which is used to specify in which topic to send the event. allowTopicOverride in Kafka sink to control whether the target topic's name can be overridden by the specified header. This closes #105 Reviewers: Attila Simon (Tristan Stevens via Denes Arvay)
          Hide
          tmgstev Tristan Stevens added a comment -

          Jeff Holoman Denes Arvay This PR has been open for a while and is actually quite a big issue (infinite loops when using Kakfa Sources with Kafka Sinks).

          Could we get a re-review and commit please (I've rebased on trunk and merged in FLUME-3100).

          Show
          tmgstev Tristan Stevens added a comment - Jeff Holoman Denes Arvay This PR has been open for a while and is actually quite a big issue (infinite loops when using Kakfa Sources with Kafka Sinks). Could we get a re-review and commit please (I've rebased on trunk and merged in FLUME-3100 ).
          Hide
          jholoman Jeff Holoman added a comment -

          Go for it.

          Show
          jholoman Jeff Holoman added a comment - Go for it.
          Hide
          tmgstev Tristan Stevens added a comment -

          Jeff Holoman I've been digging into this issue a little bit - mind if I take this, or have you got a fix in mind?

          Show
          tmgstev Tristan Stevens added a comment - Jeff Holoman I've been digging into this issue a little bit - mind if I take this, or have you got a fix in mind?

            People

            • Assignee:
              tmgstev Tristan Stevens
              Reporter:
              jholoman Jeff Holoman
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development