Hey Jeff,

Can I have this one?

Sure thing.

Hi,

Here is an implementation for new Kafka Source.
Could you please review my code.

Thanks,
Grigoriy

Ted Malaska Is it ok if we assign this to Grigoriy Rozhkov ?

Grigoriy Rozhkov would you mind posting your patch on Review Board? Please use flume-git when uploading the patch. You can add me as the reviewer.

Thanks

Jeff

Jeff,
Could you please explain in more details how to submit a patch on review board?
Thanks.

Review Board Updated

https://reviews.apache.org/r/41702/diff/5/

GitHub user GRpro opened a pull request:

https://github.com/apache/flume/pull/44

FLUME-2821 Add Kafka Source that supports new Kafka API

Support for reading multiple topics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/GRpro/flume FLUME-2821

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44

commit e709e0dfe1bd9ae7bd9533ec41963bfb5af46157
Author: GRpro <grigoriyroghkov@gmail.com>
Date: 2016-03-14T16:51:01Z

Add new implementation of Kafka Source

Kafka introduces new consumer API in
release 0.9.0.0 and Kafka Source
is rewritten in order to support new API.

Can you please upload the latest patch from the review board to this JIRA as well Grigoriy Rozhkov? I need that because by attaching the patch here, you're implicitly giving ASF rights to include it (which is not the case on review board). After that I'll go ahead and commit it as discussed on FLUME-2820.

The latest version of the patch from review board.

Added the latest patch.

Commit f8abaf78fb98e91b7a228aaa231f4164d8dcfc97 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f8abaf7 ]

FLUME-2821: Flume-Kafka Source with new Consumer

(Grigoriy Rozhkov via Jarek Jarcec Cecho)

Commit dd1232b6f1b4551f7909c43e693a694f5251b399 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dd1232b ]

FLUME-2821: Flume-Kafka Source with new Consumer

(Grigoriy Rozhkov via Jarek Jarcec Cecho)

Thank you for your contribution Grigoriy Rozhkov!

Github user GRpro closed the pull request at:

https://github.com/apache/flume/pull/44

FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/)
FLUME-2821: Flume-Kafka Source with new Consumer (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f8abaf78fb98e91b7a228aaa231f4164d8dcfc97)

• flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
• flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
• flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java
• flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
• flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java
• flume-ng-doc/sphinx/FlumeUserGuide.rst
• flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java
• flume-ng-sources/flume-kafka-source/pom.xml
• pom.xml
• flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java

Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from Grant Henke
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ]

FLUME-2983. Handle offset migration in the new Kafka Source

Similar to FLUME-2972, offsets tracking the position in Kafka consumers
change from using zookeeper for offset storage to Kafka when moving from
0.8.x to 0.9.x.

FLUME-2821 makes the client change in the Kafka Source but does not
ensure existing offsets get migrated in order to continue consuming
where it left off. Flume should have some automated logic on startup to
check if Kafka offsets exist, if not and migration is enabled (by
default) then copy the offsets from Zookeeper and commit them to Kafka.

This change should also fix the backwards incompatibility caused by
removing the zookeeperConnect property. The bootstrap can be looked up
if zookeeperConnect is used.

Reviewers: Denes Arvay, Mike Percy

(Grant Henke via Mike Percy)