Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13196

MirrorMaker 2 not always start tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.7.1
    • None
    • mirrormaker
    • None

    Description

      I am using MirrorMaker 2.0 and running it via  MirrorMaker.java class. This method will start up `DistributedHerder` and will use non-functional `advertisedUrl`, and therefore workers can not talk to each other and coordinate.

      After upgrading from version `2.4.0` to `2.7.1` I noticed that when I am starting up mirror maker it does not always start tasks - just connector is running. Doing some amount of stop/starts will eventually start tasks too.

      After a bit of digging I did notice that in attempt to configure connector's task, code ends up in this [1] branch, where configure request is being forwarded to the leader. For some reason, task configuration is not done on leader. However, MirrorMaker does not pack RestServer and therefore that request will never succeed.

      I am not sure what is going no or why it does seem to work better on 2.4.0. I noticed that connector start procedure did involve less callbacks on 2.4.0 in connector start sequence [2].

       

       

      [1] https://github.com/apache/kafka/blob/2.7/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1494

      [2] https://github.com/apache/kafka/blob/2.4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1236

      Attachments

        Activity

          People

            Unassigned Unassigned
            jvilcek Jozef Vilcek
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: