Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7238

Producer can't update Metadata when two brokers break down and restart the first one

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.11.0.1, 0.11.0.3
    • None
    • clients
    • None
    • os: 14.04.1-Ubuntu
      java: 1.8.0_151-b12

    Description

      There is a kafka cluster with two brokers( broker ids are 70 and 67),I create a topic named topic2 with 2 partitions and 1 replica, and partition 0  is on the broker 70,and partition 1  is on the broker 67.

       While sending datas， I stop broker 70 first, and wait 5 ms， stop broker 67 . And then ,I restart  broker 70,  the  producer client can't update metadata  and don't  switch to connet broker 70，otherwise， it always try to connect broker 67.

      The log before stop the broker:

      ================send
      The message size: 12, key: null, partition:0, offset:29
      [2018-08-02 19:59:10,180] DEBUG Sending metadata request (type=MetadataRequest, topics=topic2) to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 19:59:10,184] DEBUG Updated cluster metadata version 11 to Cluster(id = 3fL1MXQtRo6Ujmxa9dS3eA, nodes = [10.93.132.70:9092 (id: 70 rack: null), 10.93.132.67:9092 (id: 67 rack: null)], partitions = [Partition(topic = topic2, partition = 1, leader = 70, replicas = [70], isr = [70]), Partition(topic = topic2, partition = 0, leader = 67, replicas = [67], isr = [67])]) (org.apache.kafka.clients.Metadata)
      ================send
      The message size: 12, key: null, partition:1, offset:69
      ================send
      The message size: 12, key: null, partition:0, offset:30
      ================send
      The message size: 12, key: null, partition:1, offset:70
      ================send
      The message size: 12, key: null, partition:0, offset:31
      ================send
      The message size: 12, key: null, partition:1, offset:71
      ================send
      The message size: 12, key: null, partition:0, offset:32

       

       

      The log after stop the broker 70  :

      [2018-08-02 20:02:50,049] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:50,049] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:50,049] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:50,100] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:50,151] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:50,151] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient)
      ================send
      The message size: 12, key: null, partition:0, offset:95
      [2018-08-02 20:02:51,156] DEBUG Connection with /10.93.132.70 disconnected (org.apache.kafka.common.network.Selector)
      java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
      at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:954)
      at java.lang.Thread.run(Thread.java:745)
      [2018-08-02 20:02:51,157] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:51,157] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:51,157] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:51,208] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:51,208] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,216] DEBUG Connection with /10.93.132.70 disconnected (org.apache.kafka.common.network.Selector)
      java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
      at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:954)
      at java.lang.Thread.run(Thread.java:745)
      [2018-08-02 20:02:52,216] DEBUG Node 70 disconnected. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,216] WARN Connection to node 70 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,216] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,267] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,318] DEBUG Initialize connection to node 10.93.132.70:9092 (id: 70 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:02:52,318] DEBUG Initiating connection to node 10.93.132.70:9092 (id: 70 rack: null) (org.apache.kafka.clients.NetworkClient)
      ================send
      The message size: 12, key: null, partition:0, offset:96

       

      The broker 67 is working!

       

      The log after stop the broker 70 ,broker 67 ,and restart broker 70 :

      [2018-08-02 20:05:15,136] DEBUG Node 67 disconnected. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:15,136] WARN Connection to node 67 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:15,137] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:15,177] DEBUG Initiating connection to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:16,183] DEBUG Connection with /10.93.132.67 disconnected (org.apache.kafka.common.network.Selector)
      java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
      at java.lang.Thread.run(Thread.java:745)
      [2018-08-02 20:05:16,183] DEBUG Node 67 disconnected. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:16,183] WARN Connection to node 67 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:16,183] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:16,233] DEBUG Give up sending metadata request since no node is available (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:16,242] DEBUG Initiating connection to node 10.93.132.67:9092 (id: 67 rack: null) (org.apache.kafka.clients.NetworkClient)
      [2018-08-02 20:05:17,247] DEBUG Connection with /10.93.132.67 disconnected (org.apache.kafka.common.network.Selector)
      java.net.ConnectException: Connection refused: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:361)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
      at java.lang.Thread.run(Thread.java:745)

       

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            little brother ma little brother ma
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: