Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: None
    • Labels:
      None
    1. FLUME-2823v4.patch
      70 kB
      Jeff Holoman
    2. FLUME-2823v6.patch
      69 kB
      Jeff Holoman

      Issue Links

        Activity

        Hide
        jholoman Jeff Holoman added a comment -

        Review Board Updated

        Show
        jholoman Jeff Holoman added a comment - Review Board Updated
        Hide
        hshreedharan Hari Shreedharan added a comment -

        This looks good, but the tests are hanging. There is at least 1 incomplete test as well. Could you please fix that as well?

        Show
        hshreedharan Hari Shreedharan added a comment - This looks good, but the tests are hanging. There is at least 1 incomplete test as well. Could you please fix that as well?
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e8c4a7b ]

        FLUME-2823: Flume-Kafka-Channel with new APIs

        (Jeff Holoman via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e8c4a7b ] FLUME-2823 : Flume-Kafka-Channel with new APIs (Jeff Holoman via Jarek Jarcec Cecho)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 49680d6bc2b28e5e8b55e32254ebaf86dafcaa60 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49680d6 ]

        FLUME-2823: Flume-Kafka-Channel with new APIs

        (Jeff Holoman via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 49680d6bc2b28e5e8b55e32254ebaf86dafcaa60 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49680d6 ] FLUME-2823 : Flume-Kafka-Channel with new APIs (Jeff Holoman via Jarek Jarcec Cecho)
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thank you for your contribution Jeff Holoman!

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Jeff Holoman !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/)
        FLUME-2823: Flume-Kafka-Channel with new APIs (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186)

        • flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
        • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
        • flume-ng-channels/flume-kafka-channel/src/test/resources/kafka-server.properties
        • flume-ng-channels/flume-kafka-channel/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/ ) FLUME-2823 : Flume-Kafka-Channel with new APIs (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 ) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java flume-ng-channels/flume-kafka-channel/src/test/resources/kafka-server.properties flume-ng-channels/flume-kafka-channel/pom.xml
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ]

        FLUME-2972. Handle offset migration in the new Kafka Channel

        Offsets tracking the position in Kafka consumers change from using
        Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to
        0.9.x.

        FLUME-2823 makes the client change in the Kafka Channel but does not
        ensure existing offsets get migrated in order to continue consuming
        where it left off.

        Flume should have some automated logic on startup to check if Kafka
        offsets exist, if not and migration is enabled (by default) then copy
        the offsets from Zookeeper and commit them to Kafka.

        Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy

        (Grant Henke via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ] FLUME-2972 . Handle offset migration in the new Kafka Channel Offsets tracking the position in Kafka consumers change from using Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to 0.9.x. FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off. Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy (Grant Henke via Mike Percy)

          People

          • Assignee:
            jholoman Jeff Holoman
            Reporter:
            jholoman Jeff Holoman
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development