Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2972

Handle offset migration in the new Kafka Channel

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.7.0
    • Component/s: Channel
    • Labels:
      None

      Description

      Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.

      FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off.

      Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka.

      1. FLUME-2972.patch
        22 kB
        Grant Henke
      2. FLUME-2972-1.patch
        21 kB
        Grant Henke
      3. FLUME-2972-2.patch
        21 kB
        Grant Henke

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from Grant Henke
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ]

          FLUME-2983. Handle offset migration in the new Kafka Source

          Similar to FLUME-2972, offsets tracking the position in Kafka consumers
          change from using zookeeper for offset storage to Kafka when moving from
          0.8.x to 0.9.x.

          FLUME-2821 makes the client change in the Kafka Source but does not
          ensure existing offsets get migrated in order to continue consuming
          where it left off. Flume should have some automated logic on startup to
          check if Kafka offsets exist, if not and migration is enabled (by
          default) then copy the offsets from Zookeeper and commit them to Kafka.

          This change should also fix the backwards incompatibility caused by
          removing the zookeeperConnect property. The bootstrap can be looked up
          if zookeeperConnect is used.

          Reviewers: Denes Arvay, Mike Percy

          (Grant Henke via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from Grant Henke [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ] FLUME-2983 . Handle offset migration in the new Kafka Source Similar to FLUME-2972 , offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x. FLUME-2821 makes the client change in the Kafka Source but does not ensure existing offsets get migrated in order to continue consuming where it left off. Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. This change should also fix the backwards incompatibility caused by removing the zookeeperConnect property. The bootstrap can be looked up if zookeeperConnect is used. Reviewers: Denes Arvay, Mike Percy (Grant Henke via Mike Percy)
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #199 (See https://builds.apache.org/job/Flume-trunk-hbase-1/199/)
          FLUME-2972. Handle offset migration in the new Kafka Channel (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=1e8f2651dacf5daef55d75c7b9b12492962e7921)

          • (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
          • (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
          • (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          • (edit) flume-ng-channels/flume-kafka-channel/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #199 (See https://builds.apache.org/job/Flume-trunk-hbase-1/199/ ) FLUME-2972 . Handle offset migration in the new Kafka Channel (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=1e8f2651dacf5daef55d75c7b9b12492962e7921 ) (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst (edit) flume-ng-channels/flume-kafka-channel/pom.xml
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk. Thanks Grant!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk. Thanks Grant!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ]

          FLUME-2972. Handle offset migration in the new Kafka Channel

          Offsets tracking the position in Kafka consumers change from using
          Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to
          0.9.x.

          FLUME-2823 makes the client change in the Kafka Channel but does not
          ensure existing offsets get migrated in order to continue consuming
          where it left off.

          Flume should have some automated logic on startup to check if Kafka
          offsets exist, if not and migration is enabled (by default) then copy
          the offsets from Zookeeper and commit them to Kafka.

          Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy

          (Grant Henke via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1e8f2651dacf5daef55d75c7b9b12492962e7921 in flume's branch refs/heads/trunk from Grant Henke [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=1e8f265 ] FLUME-2972 . Handle offset migration in the new Kafka Channel Offsets tracking the position in Kafka consumers change from using Zookeeper for offset storage to Kafka when moving from Kafka 0.8.x to 0.9.x. FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off. Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. Reviewers: Balázs Donát Bessenyei, Denes Arvay, Mike Percy (Grant Henke via Mike Percy)
          Hide
          mpercy Mike Percy added a comment -

          +1. I am going to commit the latest version from ReviewBoard.

          Show
          mpercy Mike Percy added a comment - +1. I am going to commit the latest version from ReviewBoard.

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              granthenke Grant Henke
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development