Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2999

Kafka channel and sink should enable statically assigned partition per event via header

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v1.7.0
    • Fix Version/s: v1.7.0
    • Component/s: Channel, Sinks+Sources
    • Labels:
      None

      Description

      This feature is useful for anyone who needs greater control of which partitions are being written to - normally in a situation where multiple Flume agents are being deployed in order to horizontally scale, or alternatively if there is a scenario where there is a skew in data that might lead to one or more partitions hotspotting.

      We also have the ability to specify custom partitions on to the Kafka Producer itself using the kafka.* configuration properties.

      The Kafka Producer provides the ability to set the partition ID using the following constructor (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord(java.lang.String,%20java.lang.Integer,%20K,%20V%29 ), this is just a matter of providing the option to use this constructor.

      1. FLUME-2999-4.patch
        51 kB
        Tristan Stevens

        Issue Links

          Activity

          Hide
          mpercy Mike Percy added a comment -

          +1. I am about to commit the latest rev from FLUME-2999.

          Show
          mpercy Mike Percy added a comment - +1. I am about to commit the latest rev from FLUME-2999 .
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 585c4c92ed65c529871765ab64a5b3637fbdb07d in flume's branch refs/heads/trunk from Tristan Stevens
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=585c4c9 ]

          FLUME-2999. Kafka channel and sink should enable statically assigned partition per event via header

          This feature is useful for anyone who needs greater control of which
          partitions are being written to - normally in a situation where multiple
          Flume agents are being deployed in order to horizontally scale, or
          alternatively if there is a scenario where there is a skew in data that
          might lead to one or more partitions hotspotting.

          We also have the ability to specify custom partitions on to the Kafka
          Producer itself using the kafka.* configuration properties. The Kafka
          Producer provides the ability to set the partition ID using a particular
          ProducerRecord constructor, this is just a matter of providing the
          option to use this constructor.

          Reviewers: Attila Simon, Mike Percy

          (Tristan Stevens via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 585c4c92ed65c529871765ab64a5b3637fbdb07d in flume's branch refs/heads/trunk from Tristan Stevens [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=585c4c9 ] FLUME-2999 . Kafka channel and sink should enable statically assigned partition per event via header This feature is useful for anyone who needs greater control of which partitions are being written to - normally in a situation where multiple Flume agents are being deployed in order to horizontally scale, or alternatively if there is a scenario where there is a skew in data that might lead to one or more partitions hotspotting. We also have the ability to specify custom partitions on to the Kafka Producer itself using the kafka.* configuration properties. The Kafka Producer provides the ability to set the partition ID using a particular ProducerRecord constructor, this is just a matter of providing the option to use this constructor. Reviewers: Attila Simon, Mike Percy (Tristan Stevens via Mike Percy)
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk. Thanks for the patch Tristan!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk. Thanks for the patch Tristan!
          Hide
          tmgstev Tristan Stevens added a comment -

          Thanks for the support Mike Percy.
          Attached patch.

          Show
          tmgstev Tristan Stevens added a comment - Thanks for the support Mike Percy . Attached patch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #209 (See https://builds.apache.org/job/Flume-trunk-hbase-1/209/)
          FLUME-2999. Kafka channel and sink should enable statically assigned (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=585c4c92ed65c529871765ab64a5b3637fbdb07d)

          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          • (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java
          • (add) flume-shared/pom.xml
          • (edit) flume-ng-sources/flume-kafka-source/pom.xml
          • (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
          • (edit) pom.xml
          • (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
          • (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
          • (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java
          • (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java
          • (add) flume-shared/flume-shared-kafka-test/pom.xml
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/pom.xml
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java
          • (edit) flume-ng-channels/flume-kafka-channel/pom.xml
          • (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #209 (See https://builds.apache.org/job/Flume-trunk-hbase-1/209/ ) FLUME-2999 . Kafka channel and sink should enable statically assigned (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=585c4c92ed65c529871765ab64a5b3637fbdb07d ) (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java (add) flume-shared/pom.xml (edit) flume-ng-sources/flume-kafka-source/pom.xml (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java (edit) pom.xml (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java (add) flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java (add) flume-shared/flume-shared-kafka-test/pom.xml (edit) flume-ng-sinks/flume-ng-kafka-sink/pom.xml (edit) flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java (edit) flume-ng-channels/flume-kafka-channel/pom.xml (edit) flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java

            People

            • Assignee:
              tmgstev Tristan Stevens
              Reporter:
              tmgstev Tristan Stevens
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development