Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12937

Mirrormaker2 can only start from the beginning of a topic

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.8.0
    • None
    • mirrormaker
    • None
    • Dockerized environment

    Description

      Goal:
      I want to replace Mirrormaker version 1 with Mirrormaker2.
      To do this I want to:
      start Mirrormaker2 from the latest offset of every topic
      stop Mirrormaker1 
      There should only be a couple of double messages.

      What happened:
      Mirrormaker2 starts replicating from the start of all topics

      How to reproduce:
      Start two Kafka clusters, A and B

      I produce 3000 messages to cluster A on a topic (TOPIC1)
      Kafka Connect is running and connected to cluster B
      Start a Mirrormaker2 task in connect to replicate messages from cluster A. Wit the option:
      consumer auto.offset.reset to latest
      Produce another 3000 messages to cluster A on the same topic (TOPIC1)

      Expected result:
      Cluster B will only contain the messages produced the second time (3000 in total) on TOPIC1

      Actual result:
      The mirror picks up all messages from the start (6000 in total) and replicates them to cluster B

      Additional logs:
      Logs from the consumer of the Mirrormaker task:

      mirrormaker.log:7581:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7583:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7585:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7587:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7589:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7591:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7593:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7595:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
      mirrormaker.log:7597:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition perf-test-4 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are trying to seek to a position and thus overriding the latest offset

       

      You can see it is doing a seek to position 0 for every partition. which is not what I expected

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              henkdesurinaamsepotvis Daan Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: