Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2820 Support New Kafka APIs
  3. FLUME-2823

Flume-Kafka-Channel with new APIs

        Details

        • Type: Sub-task
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 1.7.0
        • Component/s: None
        • Labels:
          None
        1. Text File
          FLUME-2823v4.patch
          70 kB
          Jeff Holoman
        2. Text File
          FLUME-2823v6.patch
          69 kB
          Jeff Holoman

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLUME-2972 Handle offset migration in the new Kafka Channel

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. FLUME-2868 Kafka Channel partition topic by key

          • Major - Major loss of function.
          • Closed
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. FLUME-2726 Remove * imports from Kafka Channel

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          links to

          Web Link Review Board

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jholoman Jeff Holoman added a comment -

            Review Board Updated

            Show
            jholoman Jeff Holoman added a comment - Review Board Updated
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            This looks good, but the tests are hanging. There is at least 1 incomplete test as well. Could you please fix that as well?

            Show
            hshreedharan Hari Shreedharan added a comment - This looks good, but the tests are hanging. There is at least 1 incomplete test as well. Could you please fix that as well?
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e8c4a7b ]

            FLUME-2823: Flume-Kafka-Channel with new APIs

            (Jeff Holoman via Jarek Jarcec Cecho)

            Show
            jira-bot ASF subversion and git services added a comment - Commit e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e8c4a7b ] FLUME-2823 : Flume-Kafka-Channel with new APIs (Jeff Holoman via Jarek Jarcec Cecho)
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit 49680d6bc2b28e5e8b55e32254ebaf86dafcaa60 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49680d6 ]

            FLUME-2823: Flume-Kafka-Channel with new APIs

            (Jeff Holoman via Jarek Jarcec Cecho)

            Show
            jira-bot ASF subversion and git services added a comment - Commit 49680d6bc2b28e5e8b55e32254ebaf86dafcaa60 in flume's branch refs/heads/flume-1.7 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=49680d6 ] FLUME-2823 : Flume-Kafka-Channel with new APIs (Jeff Holoman via Jarek Jarcec Cecho)
            Hide
            Permalink
            jarcec Jarek Jarcec Cecho added a comment -

            Thank you for your contribution Jeff Holoman!

            Show
            jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Jeff Holoman !
            Hide
            Permalink
            hudson Hudson added a comment -

            FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/)
            FLUME-2823: Flume-Kafka-Channel with new APIs (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186)

            • flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
            • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
            • flume-ng-doc/sphinx/FlumeUserGuide.rst
            • flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
            • flume-ng-channels/flume-kafka-channel/src/test/resources/kafka-server.properties
            • flume-ng-channels/flume-kafka-channel/pom.xml
            Show
            hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #154 (See https://builds.apache.org/job/Flume-trunk-hbase-1/154/ ) FLUME-2823 : Flume-Kafka-Channel with new APIs (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e8c4a7bffc74f6ea10ae6cc45adbaf4919f45186 ) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java flume-ng-channels/flume-kafka-channel/src/test/resources/kafka-server.properties flume-ng-channels/flume-kafka-channel/pom.xml
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ]

            FLUME-2972. Handle offset migration in the new Kafka Channel

            Offsets tracking the position in Kafka consumers change from using
            Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to
            0.9.x.

            FLUME-2823 makes the client change in the Kafka Channel but does not
            ensure existing offsets get migrated in order to continue consuming
            where it left off.

            Flume should have some automated logic on startup to check if Kafka
            offsets exist, if not and migration is enabled (by default) then copy
            the offsets from Zookeeper and commit them to Kafka.

            Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy

            (Grant Henke via Mike Percy)

            Show
            jira-bot ASF subversion and git services added a comment - Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ] FLUME-2972 . Handle offset migration in the new Kafka Channel Offsets tracking the position in Kafka consumers change from using Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to 0.9.x. FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off. Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy (Grant Henke via Mike Percy)

              People

              • Assignee:
                jholoman Jeff Holoman
                Reporter:
                jholoman Jeff Holoman
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development