Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: None
    • Labels:
      None
    1. FLUME-2821.patch
      66 kB
      Grigoriy Rozhkov

      Issue Links

        Activity

        Hide
        Theodore michael Malaska added a comment -

        Hey Jeff,

        Can I have this one?

        Show
        Theodore michael Malaska added a comment - Hey Jeff, Can I have this one?
        Hide
        Jeff Holoman added a comment -

        Sure thing.

        Show
        Jeff Holoman added a comment - Sure thing.
        Hide
        Grigoriy Rozhkov added a comment -

        Hi,

        Here is an implementation for new Kafka Source.
        Could you please review my code.

        Thanks,
        Grigoriy

        Show
        Grigoriy Rozhkov added a comment - Hi, Here is an implementation for new Kafka Source. Could you please review my code. Thanks, Grigoriy
        Hide
        Jeff Holoman added a comment -

        Ted Malaska Is it ok if we assign this to Grigoriy Rozhkov ?

        Grigoriy Rozhkov would you mind posting your patch on Review Board? Please use flume-git when uploading the patch. You can add me as the reviewer.

        Thanks

        Jeff

        Show
        Jeff Holoman added a comment - Ted Malaska Is it ok if we assign this to Grigoriy Rozhkov ? Grigoriy Rozhkov would you mind posting your patch on Review Board? Please use flume-git when uploading the patch. You can add me as the reviewer. Thanks Jeff
        Hide
        Grigoriy Rozhkov added a comment -

        Jeff,
        Could you please explain in more details how to submit a patch on review board?
        Thanks.

        Show
        Grigoriy Rozhkov added a comment - Jeff, Could you please explain in more details how to submit a patch on review board? Thanks.
        Hide
        Grigoriy Rozhkov added a comment -
        Show
        Grigoriy Rozhkov added a comment - Review Board Updated https://reviews.apache.org/r/41702/diff/5/
        Hide
        ASF GitHub Bot added a comment -

        GitHub user GRpro opened a pull request:

        https://github.com/apache/flume/pull/44

        FLUME-2821 Add Kafka Source that supports new Kafka API

        Support for reading multiple topics.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/GRpro/flume FLUME-2821

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/flume/pull/44.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #44


        commit e709e0dfe1bd9ae7bd9533ec41963bfb5af46157
        Author: GRpro <grigoriyroghkov@gmail.com>
        Date: 2016-03-14T16:51:01Z

        Add new implementation of Kafka Source

        Kafka introduces new consumer API in
        release 0.9.0.0 and Kafka Source
        is rewritten in order to support new API.


        Show
        ASF GitHub Bot added a comment - GitHub user GRpro opened a pull request: https://github.com/apache/flume/pull/44 FLUME-2821 Add Kafka Source that supports new Kafka API Support for reading multiple topics. You can merge this pull request into a Git repository by running: $ git pull https://github.com/GRpro/flume FLUME-2821 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/44.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #44 commit e709e0dfe1bd9ae7bd9533ec41963bfb5af46157 Author: GRpro <grigoriyroghkov@gmail.com> Date: 2016-03-14T16:51:01Z Add new implementation of Kafka Source Kafka introduces new consumer API in release 0.9.0.0 and Kafka Source is rewritten in order to support new API.
        Hide
        Jarek Jarcec Cecho added a comment -

        Can you please upload the latest patch from the review board to this JIRA as well Grigoriy Rozhkov? I need that because by attaching the patch here, you're implicitly giving ASF rights to include it (which is not the case on review board). After that I'll go ahead and commit it as discussed on FLUME-2820.

        Show
        Jarek Jarcec Cecho added a comment - Can you please upload the latest patch from the review board to this JIRA as well Grigoriy Rozhkov ? I need that because by attaching the patch here, you're implicitly giving ASF rights to include it (which is not the case on review board). After that I'll go ahead and commit it as discussed on FLUME-2820 .
        Hide
        Grigoriy Rozhkov added a comment -

        The latest version of the patch from review board.

        Show
        Grigoriy Rozhkov added a comment - The latest version of the patch from review board.
        Hide
        Grigoriy Rozhkov added a comment -

        Added the latest patch.

        Show
        Grigoriy Rozhkov added a comment - Added the latest patch.
        Hide
        ASF subversion and git services added a comment -

        Commit f8abaf78fb98e91b7a228aaa231f4164d8dcfc97 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f8abaf7 ]

        FLUME-2821: Flume-Kafka Source with new Consumer

        (Grigoriy Rozhkov via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit f8abaf78fb98e91b7a228aaa231f4164d8dcfc97 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=f8abaf7 ] FLUME-2821 : Flume-Kafka Source with new Consumer (Grigoriy Rozhkov via Jarek Jarcec Cecho)
        Hide
        ASF subversion and git services added a comment -

        Commit dd1232b6f1b4551f7909c43e693a694f5251b399 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dd1232b ]

        FLUME-2821: Flume-Kafka Source with new Consumer

        (Grigoriy Rozhkov via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit dd1232b6f1b4551f7909c43e693a694f5251b399 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dd1232b ] FLUME-2821 : Flume-Kafka Source with new Consumer (Grigoriy Rozhkov via Jarek Jarcec Cecho)
        Hide
        Jarek Jarcec Cecho added a comment -

        Thank you for your contribution Grigoriy Rozhkov!

        Show
        Jarek Jarcec Cecho added a comment - Thank you for your contribution Grigoriy Rozhkov !
        Hide
        ASF GitHub Bot added a comment -

        Github user GRpro closed the pull request at:

        https://github.com/apache/flume/pull/44

        Show
        ASF GitHub Bot added a comment - Github user GRpro closed the pull request at: https://github.com/apache/flume/pull/44
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/)
        FLUME-2821: Flume-Kafka Source with new Consumer (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f8abaf78fb98e91b7a228aaa231f4164d8dcfc97)

        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java
        • flume-ng-sources/flume-kafka-source/pom.xml
        • pom.xml
        • flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java
        Show
        Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/ ) FLUME-2821 : Flume-Kafka Source with new Consumer (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=f8abaf78fb98e91b7a228aaa231f4164d8dcfc97 ) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java flume-ng-sources/flume-kafka-source/pom.xml pom.xml flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java
        Hide
        ASF subversion and git services added a comment -

        Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from Grant Henke
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ]

        FLUME-2983. Handle offset migration in the new Kafka Source

        Similar to FLUME-2972, offsets tracking the position in Kafka consumers
        change from using zookeeper for offset storage to Kafka when moving from
        0.8.x to 0.9.x.

        FLUME-2821 makes the client change in the Kafka Source but does not
        ensure existing offsets get migrated in order to continue consuming
        where it left off. Flume should have some automated logic on startup to
        check if Kafka offsets exist, if not and migration is enabled (by
        default) then copy the offsets from Zookeeper and commit them to Kafka.

        This change should also fix the backwards incompatibility caused by
        removing the zookeeperConnect property. The bootstrap can be looked up
        if zookeeperConnect is used.

        Reviewers: Denes Arvay, Mike Percy

        (Grant Henke via Mike Percy)

        Show
        ASF subversion and git services added a comment - Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from Grant Henke [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ] FLUME-2983 . Handle offset migration in the new Kafka Source Similar to FLUME-2972 , offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x. FLUME-2821 makes the client change in the Kafka Source but does not ensure existing offsets get migrated in order to continue consuming where it left off. Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. This change should also fix the backwards incompatibility caused by removing the zookeeperConnect property. The bootstrap can be looked up if zookeeperConnect is used. Reviewers: Denes Arvay, Mike Percy (Grant Henke via Mike Percy)

          People

          • Assignee:
            Grigoriy Rozhkov
            Reporter:
            Jeff Holoman
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development