Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9076

MirrorMaker 2.0 automated consumer offset sync

    XMLWordPrintableJSON

    Details

      Description

      To calculate the translated consumer offset in the target cluster, currently `Mirror-client` provides a function called "remoteConsumerOffsets()" that is used by "RemoteClusterUtils" for one-time purpose.

      In order to make the consumer and stream applications migrate from source to target cluster transparently and conveniently, e.g. in event of source cluster failure, a background job is proposed to periodically sync the consumer offsets from the source to target cluster, so that when the consumer and stream applications switch to the target cluster, it will resume to consume from where it left off at source cluster.

       KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

      https://github.com/apache/kafka/pull/7577

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yangguo1220 Ning Zhang
                Reporter:
                yangguo1220 Ning Zhang
              • Votes:
                3 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: