Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9076

MirrorMaker 2.0 automated consumer offset sync

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      To calculate the translated consumer offset in the target cluster, currently `Mirror-client` provides a function called "remoteConsumerOffsets()" that is used by "RemoteClusterUtils" for one-time purpose.

      In order to make the consumer and stream applications migrate from source to target cluster transparently and conveniently, e.g. in event of source cluster failure, a background job is proposed to periodically sync the consumer offsets from the source to target cluster, so that when the consumer and stream applications switch to the target cluster, it will resume to consume from where it left off at source cluster.

       KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

      https://github.com/apache/kafka/pull/7577

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yangguo1220 Ning Zhang
            yangguo1220 Ning Zhang
            Votes:
            3 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment