Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6703

MirrorMaker cannot make progress when any matched topic from a whitelist regexp has -1 leader

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Scenario:

      • MM whitelabel regexp matches multiple topics
      • destination cluster has 5 brokers with multiple topics replication factor 3
      • without partition reassign shut down 2 brokers
      • suppose a topic has no leader any more because it was off-sync and the leader and the rest of the replicas are hosted on the downed brokers.
      • so we have 1 topic with some partitions with leader -1
      • the rest of the matching topics has 3 replicas with leaders

      MM will not produce into any of the matched topics until:

      • the "orphaned" topic removed or
      • the partition reassign carried out from the downed brokers (suppose you can turn these back on)

      In the MirrorMaker logs, there are a lot of messages like the following ones:

      [2018-03-22 19:55:32,522] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054] Coordinator discovery failed, refreshing metadata (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      
      [2018-03-22 19:55:32,522] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054] Sending metadata request (type=MetadataRequest, topics=<ALL>) to node 192.168.1.102:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
      
      [2018-03-22 19:55:32,525] DEBUG Updated cluster metadata version 10 to Cluster(id = Y-qtoFP-RMq2uuVnkEKAAw, nodes = [192.168.1.102:9092 (id: 0 rack: null)], partitions = [Partition(topic = testR1P2, partition = 1, leader = none, replicas = [42], isr = [], offlineReplicas = [42]), Partition(topic = testR1P1, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = []), Partition(topic = testAlive, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = []), Partition(topic = testERRR, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = []), Partition(topic = testR1P2, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = [])]) (org.apache.kafka.clients.Metadata)
      
      [2018-03-22 19:55:32,525] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054] Sending FindCoordinator request to broker 192.168.1.102:9092 (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      
      [2018-03-22 19:55:32,525] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054] Received FindCoordinator response ClientResponse(receivedTimeMs=1521744932525, latencyMs=0, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, clientId=consumer-1, correlationId=19), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=COORDINATOR_NOT_AVAILABLE, node=:-1 (id: -1 rack: null))) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      
      [2018-03-22 19:55:32,526] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054] Group coordinator lookup failed: The coordinator is not available. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      

      Interestingly, if MirrorMaker uses zookeeper.connect in its consumer properties file, then an OldConsumer is created, and it can make progress.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              asasvari Attila Sasvári
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: