Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2972

Handle offset migration in the new Kafka Channel

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.7.0
    • 1.7.0
    • Channel
    • None

    Description

      Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.

      FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off.

      Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka.

      Attachments

        1. FLUME-2972.patch
          22 kB
          Grant Henke
        2. FLUME-2972-1.patch
          21 kB
          Grant Henke
        3. FLUME-2972-2.patch
          21 kB
          Grant Henke

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            granthenke Grant Henke
            granthenke Grant Henke
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment