Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2983

Handle offset migration in the new Kafka Source

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Similar to FLUME-2972, Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.
      FLUME-2821 makes the client change in the Kafka Source but does not ensure existing offsets get migrated in order to continue consuming where it left off.
      Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka.

      This change should also fix the backwards incompatibility caused by removing the zookeeperConnect property. The bootstrap can be looked up if zookeeperConnect is used.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              granthenke Grant Henke

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment