Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8000

Java client uses IP address for different kafka broker, preventing reconnection

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.1.1
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None

      Description

      I've been observing this issue occurring most times I kill a Kafka Broker (for sake of exposing bugs, I periodically kill one of my kafka brokers). This bug results in large latency spikes. And sometimes prevents the java Kafka Client from ever being able to resume transmitting messages to a partition.

      VERSIONS:

      • I'm using Kafka 2.1.1, in confluentinc/cp-kafka:5.1.2. I also saw this in 5.0.1.

      OBSERVED ISSUE:

      • Create kafka cluster with three brokers. 
      • Setup topic with replication factor of three
      • Create example service that reads and writes from Kafka partition
      • Kill brokers periodically.

      In the following log, you see a situation where Node 30 has been killed intentionally. As expected, you'll see that the Kafka Client cannot find Node 30 any longer.

      However, what isn't expected is that when the Kafka Client attempts to connect to a different node it becomes confused about which Node has which IP address. You'll see lines such as "Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b)", even though previous metadata updates state "10.20.2.109" is the IP address for node 30, not Node 20!

      You'll notice that this behavior continues for a long period of time, even once the Kafka Client observes that node 30 is offline in MetadataCache update the following

       

      ```

      MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      ```

      FULL LOGS

      ```

      ordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.11.109:9092 (id: 30 rack: us-west2-c)
      17:47:44.355 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 34 to 34 for partition exampletopic3-0
      17:47:44.355 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 334 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.11.109:9092 (id: 30 rack: us-west2-c), 10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 30, replicas = [30,20,10], isr = [30,20], offlineReplicas = [10])], controller = 10.20.11.109:9092 (id: 30 rack: us-west2-c))}

      poll emptiness = false
      17:47:44.380 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30053 for partition exampletopic3-0
      17:47:44.585 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30053 to node 10.20.11.109:9092 (id: 30 rack: us-west2-c)
      17:47:44.585 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=622931687, epoch=787) for node 30. Added 0 partition(s), altered 1 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:44.585 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(exampletopic3-0), toForget=(), implied=()) to broker 10.20.11.109:9092 (id: 30 rack: us-west2-c)
      17:47:44.605 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 sent an incremental fetch response for session 622931687 with 1 response partition(s)
      17:47:44.605 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Fetch READ_UNCOMMITTED at offset 30053 for partition exampletopic3-0 returned fetch data (error=NONE, highWaterMark=30059, lastStableOffset = 30059, logStartOffset = 0, abortedTransactions = null, recordsSizeInBytes=846)
      17:47:44.605 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30059 to node 10.20.11.109:9092 (id: 30 rack: us-west2-c)
      17:47:44.605 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=622931687, epoch=788) for node 30. Added 0 partition(s), altered 1 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:44.605 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(exampletopic3-0), toForget=(), implied=()) to broker 10.20.11.109:9092 (id: 30 rack: us-west2-c)
      poll emptiness = false
      17:47:44.631 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30059 for partition exampletopic3-0
      17:47:44.708 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 sent an incremental fetch response for session 622931687 with 1 response partition(s)
      17:47:44.708 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Fetch READ_UNCOMMITTED at offset 30059 for partition exampletopic3-0 returned fetch data (error=NONE, highWaterMark=30060, lastStableOffset = 30060, logStartOffset = 0, abortedTransactions = null, recordsSizeInBytes=141)
      17:47:44.769 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.io.EOFException: null
      at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
      at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
      at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
      at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:644)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:575)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:44.769 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:44.770 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:44.770 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)

      NOTE: Node 30 disconnected! I think Node 30 was the group coordinator

      17:47:44.793 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:44.793 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.

      NOTE: Tries to connect to node 20 now? But uses 10.20.2.109.

      TRIES TO CONNECT TO NODE 20 NOW!!!
      FAILS!!! Because it tries to use the same address as the address from node 30.

      17:47:44.793 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:44.793 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:44.812 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.io.EOFException: null
      at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
      at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
      at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
      at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:644)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:575)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:304)
      at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1027)
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.io.EOFException: null
      at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
      at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
      at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
      at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:644)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:575)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:304)
      at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1027)
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 2147483617 disconnected.
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 disconnected.
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:44.813 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:44.822 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:44.836 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 34 to 34 for partition exampletopic3-0
      17:47:44.836 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 335 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.11.109:9092 (id: 30 rack: us-west2-c), 10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 30, replicas = [30,20,10], isr = [30,20], offlineReplicas = [10])], controller = 10.20.11.109:9092 (id: 30 rack: us-west2-c))}

      17:47:44.839 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:44.839 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:44.839 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:44.839 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:44.855 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972864855, latencyMs=42, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3618), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:44.855 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:44.855 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:44.893 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true

      TRIES TO CONNECT TO NODE 30, USING THE ADDRESS OF NODE 20.
      Absurd, because NODE 30's IP address is 10.20.11.110 not 10.20.11.109.

      17:47:44.935 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node
      10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:44.935 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:44.953 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:44.953 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:44.953 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:44.953 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:44.954 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972864954, latencyMs=19, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3620), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:44.954 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:44.954 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 2147483617 rack: null) using address /10.20.11.109
      connect(id=2147483617, address=/10.20.11.109:9092...)
      17:47:44.974 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
      at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:661)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1341)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1304)
      at org.apache.kafka.Main$1.run(Main.java:130)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:44.975 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 2147483617 disconnected.
      17:47:44.975 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 2147483617 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:44.975 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Cancelled request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=6, clientId=consumer-1, correlationId=3621) due to node 2147483617 being disconnected
      17:47:44.975 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:45.004 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.037 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initialize connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) for sending metadata request
      17:47:45.037 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:45.038 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.055 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.079 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.080 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
      at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235)
      at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:656)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1341)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1304)
      at org.apache.kafka.Main$1.run(Main.java:130)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.080 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 disconnected.
      17:47:45.080 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.098 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 34 to 34 for partition exampletopic3-0
      17:47:45.098 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 336 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.11.109:9092 (id: 30 rack: us-west2-c), 10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 30, replicas = [30,20,10], isr = [30,20], offlineReplicas = [10])], controller = 10.20.11.109:9092 (id: 30 rack: us-west2-c))}

      NOTE: This is a successful metadata refresh request.

      17:47:45.099 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972865099, latencyMs=61, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3622), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:45.099 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:45.100 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 2147483617 rack: null) using address /10.20.11.109
      connect(id=2147483617, address=/10.20.11.109:9092...)
      17:47:45.105 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.122 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
      at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:661)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1341)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1304)
      at org.apache.kafka.Main$1.run(Main.java:130)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.122 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 2147483617 disconnected.
      17:47:45.122 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 2147483617 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.122 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Cancelled request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=6, clientId=consumer-1, correlationId=3624) due to node 2147483617 being disconnected
      17:47:45.122 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:45.143 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.159 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.173 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:45.191 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.191 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:45.192 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.192 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.227 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initialize connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) for sending metadata request
      17:47:45.227 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:45.227 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972865227, latencyMs=84, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3625), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:45.227 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:45.228 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:45.244 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.298 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.332 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.332 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.333 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
      at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235)
      at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:656)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1341)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1304)
      at org.apache.kafka.Main$1.run(Main.java:130)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.333 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 disconnected.
      17:47:45.333 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.352 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.369 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 34 to 34 for partition exampletopic3-0
      17:47:45.370 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 337 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.11.109:9092 (id: 30 rack: us-west2-c), 10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 30, replicas = [30,20,10], isr = [30,20], offlineReplicas = [10])], controller = 10.20.11.109:9092 (id: 30 rack: us-west2-c))}

      17:47:45.376 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972865376, latencyMs=44, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3626), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:45.376 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:45.376 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 2147483617 rack: null) using address /10.20.11.109
      connect(id=2147483617, address=/10.20.11.109:9092...)
      17:47:45.396 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
      at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:661)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1341)
      at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1304)
      at org.apache.kafka.Main$1.run(Main.java:130)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.396 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 2147483617 disconnected.
      17:47:45.396 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 2147483617 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.396 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Cancelled request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=6, clientId=consumer-1, correlationId=3628) due to node 2147483617 being disconnected
      17:47:45.396 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:45.396 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.403 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.454 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.498 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.498 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972865498, latencyMs=102, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3629), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.11.109:9092 (id: 30 rack: null)))
      17:47:45.498 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null)
      17:47:45.498 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Group coordinator 10.20.11.109:9092 (id: 2147483617 rack: null) is unavailable or invalid, will attempt rediscovery
      17:47:45.498 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending FindCoordinator request to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:45.506 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.521 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 34 to 35 for partition exampletopic3-0
      17:47:45.521 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 338 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b), 10.20.11.109:9092 (id: 30 rack: us-west2-c)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.11.109:9092 (id: 30 rack: us-west2-c))}

      17:47:45.522 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received FindCoordinator response ClientResponse(receivedTimeMs=1550972865522, latencyMs=24, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=3631), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=10.20.2.110:9092 (id: 20 rack: null)))
      17:47:45.522 [Thread-0] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Discovered group coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      17:47:45.522 [Thread-0] INFO org.apache.kafka.clients.ClusterConnectionStates - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Hostname for node 2147483627 changed from 10.20.2.109 to 10.20.2.110.
      17:47:45.522 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.2.110:9092 (id: 2147483627 rack: null) using address /10.20.2.110
      connect(id=2147483627, address=/10.20.2.110:9092...)
      17:47:45.546 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:45.546 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Created socket with SO_RCVBUF = 65905, SO_SNDBUF = 131810, SO_TIMEOUT = 0 to node 2147483627
      17:47:45.547 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Completed connection to node 2147483627. Fetching API versions.
      17:47:45.547 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating API versions fetch from node 2147483627.
      17:47:45.568 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.568 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:45.568 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:45.568 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.568 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Recorded API versions for node 2147483627: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 4 [usable: 4], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 3 [usable: 3], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 2 [usable: 2], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 2 [usable: 2], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1], ElectPreferredLeaders(43): UNSUPPORTED)
      17:47:45.595 [Thread-0] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Offset commit failed on partition exampletopic3-0 at offset 30059: The coordinator is loading and hence can't process requests.
      17:47:45.598 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      17:47:45.620 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.670 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.700 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      17:47:45.720 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.726 [Thread-0] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Offset commit failed on partition exampletopic3-0 at offset 30059: The coordinator is loading and hence can't process requests.
      17:47:45.773 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.827 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.880 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:45.880 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30059 for partition exampletopic3-0
      17:47:45.932 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:45.932 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:45.950 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:45.950 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:45.950 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:45.950 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.002 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.052 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 1373, value = bb094f23-f7c7-47b7-b62f-c467bc42e462
      STILL UNRECEIVED, DELAY = : 1449, value = dcdc5599-7b1a-43de-8d9d-5bbdf05db5c5
      STILL UNRECEIVED, DELAY = : 1300, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:46.082 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30060 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:46.082 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built full fetch (sessionId=1651867364, epoch=INITIAL) for node 20 with 1 partition(s).
      17:47:46.082 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED FullFetchRequest(exampletopic3-0) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      poll emptiness = false
      RECEIVING TOOK LONG TIME: 1449
      17:47:46.103 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.106 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30060 for partition exampletopic3-0
      17:47:46.154 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.204 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.214 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent a full fetch response that created a new incremental fetch session 1250380980 with 1 response partition(s)
      17:47:46.214 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Fetch READ_UNCOMMITTED at offset 30060 for partition exampletopic3-0 returned fetch data (error=NONE, highWaterMark=30061, lastStableOffset = 30061, logStartOffset = 0, abortedTransactions = null, recordsSizeInBytes=141)
      17:47:46.254 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.305 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 1601, value = bb094f23-f7c7-47b7-b62f-c467bc42e462
      STILL UNRECEIVED, DELAY = : 1528, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:46.310 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:46.310 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=1) for node 20. Added 0 partition(s), altered 1 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:46.310 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(exampletopic3-0), toForget=(), implied=()) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      poll emptiness = false
      RECEIVING TOOK LONG TIME: 1602
      17:47:46.335 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:46.355 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.381 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:46.399 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:46.400 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:46.400 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:46.400 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.454 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.505 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 1753, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:46.535 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initialize connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) for sending metadata request
      17:47:46.535 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:46.553 [Thread-0] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
      at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
      at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1256)
      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1200)
      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1176)
      at org.apache.kafka.Main$1.run(Main.java:113)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:46.553 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 30 disconnected.
      17:47:46.553 [Thread-0] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:46.554 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:46.556 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.608 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:46.656 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:46.660 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.715 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.765 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.816 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 2076, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 339 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=2) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:46.859 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:46.869 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.924 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:46.977 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:46.980 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:47.031 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.085 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:47.085 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:47.103 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:47.104 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:47.104 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:47.104 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.157 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 2400, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:47.211 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.264 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:47.305 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:47.315 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.368 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.389 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:47.398 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:47.418 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:47.418 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:47.418 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:47.418 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.470 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 2727, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:47.509 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:47.509 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=3) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:47.509 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:47.523 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.576 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:47.627 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.632 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:47.681 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.731 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.782 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 3053, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:47.835 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.864 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:47.890 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:47.944 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:47.957 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:47.998 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:47.998 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:48.016 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:48.017 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:48.017 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:48.017 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.037 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:48.070 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.121 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.137 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:48.138 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 340 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      STILL UNRECEIVED, DELAY = : 3378, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:48.160 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:48.160 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=4) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:48.160 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:48.173 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.218 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      17:47:48.224 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.236 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:48.236 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Completed asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      poll emptiness = true
      17:47:48.277 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.285 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:48.328 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.381 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.431 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.483 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 3708, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:48.533 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.584 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.589 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      poll emptiness = true
      17:47:48.607 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:48.607 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:48.607 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:48.607 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.616 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:48.659 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.699 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:48.699 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      17:47:48.710 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.760 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.804 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      17:47:48.810 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 4035, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:48.817 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:48.817 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=5) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:48.817 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:48.865 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:48.918 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:48.937 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:48.971 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:48.971 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:48.995 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:48.995 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:48.995 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:48.995 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.048 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.102 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 4358, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:49.140 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:49.155 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.209 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:49.264 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.288 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:49.315 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.348 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:49.370 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.422 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.449 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:49.449 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 341 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:49.474 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 4710, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:49.492 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:49.492 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=6) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:49.492 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:49.526 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.578 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.588 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      poll emptiness = true
      17:47:49.606 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:49.606 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:49.606 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:49.606 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.615 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:49.660 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.715 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.767 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.818 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 5036, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:49.869 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:49.924 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:49.956 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:49.976 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.022 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:50.026 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.077 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.127 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:50.127 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      STILL UNRECEIVED, DELAY = : 5378, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:50.160 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:50.160 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=7) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:50.160 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:50.171 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:50.171 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:50.171 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:50.171 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.225 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:50.275 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.285 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:50.330 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.384 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.434 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.467 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      STILL UNRECEIVED, DELAY = : 5705, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:50.487 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.539 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:50.593 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.607 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:50.644 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.693 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:50.698 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.715 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:50.731 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:50.731 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:50.731 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:50.731 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.784 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.796 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:50.796 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 342 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      STILL UNRECEIVED, DELAY = : 6028, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:50.810 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:50.810 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=8) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:50.810 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:50.835 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.885 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:50.939 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:50.940 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:50.993 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.047 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.099 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 6361, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:51.149 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.200 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:51.255 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.263 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:51.308 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:51.308 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:51.326 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:51.326 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:51.326 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:51.326 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.344 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:51.381 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.434 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 6684, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:51.466 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:51.466 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=9) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:51.466 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:51.487 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.542 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:51.590 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:51.596 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.649 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.703 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.756 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.778 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      STILL UNRECEIVED, DELAY = : 7010, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:51.797 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:51.797 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      17:47:51.810 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.861 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:51.913 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:51.915 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.925 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:51.943 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:51.943 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:51.943 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:51.943 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.994 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:51.995 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:52.048 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.099 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.099 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:52.099 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 343 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      STILL UNRECEIVED, DELAY = : 7334, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:52.116 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:52.116 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=10) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:52.116 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:52.150 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.200 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:52.236 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:52.250 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.303 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.357 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.410 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 7656, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:52.463 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:52.463 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:52.484 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:52.484 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:52.484 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:52.484 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.536 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:52.561 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:52.587 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.637 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.643 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:52.689 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.740 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 7980, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:52.762 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:52.762 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=11) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:52.762 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:52.790 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.818 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:52.835 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:52.835 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:52.835 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:52.835 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:52.884 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:52.886 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.936 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:52.989 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.043 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 8303, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:53.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.100 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:53.143 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:53.194 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.210 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:53.244 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.293 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:53.294 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.349 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.393 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:53.393 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 344 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:53.404 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 8629, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:53.411 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      17:47:53.411 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:53.411 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=12) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:53.411 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:53.431 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:53.431 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Completed asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      17:47:53.458 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.510 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:53.535 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:53.563 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.616 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:53.616 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:53.634 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:53.634 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:53.634 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:53.634 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.687 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.705 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:53.724 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:53.725 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:53.725 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:53.725 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 8954, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:53.776 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.830 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:53.859 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:53.882 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.936 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:53.939 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:53.989 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.039 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 9277, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:54.060 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:54.060 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=13) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:54.060 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:54.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.144 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:54.184 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:54.196 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.250 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.300 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.351 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 9603, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:54.393 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:54.402 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.452 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:54.506 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.529 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:54.560 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.594 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:54.613 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.664 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.695 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:54.695 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 345 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:54.703 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      STILL UNRECEIVED, DELAY = : 9951, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:54.733 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:54.733 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=14) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:54.733 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:54.747 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:54.747 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:54.747 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:54.747 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.780 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      17:47:54.798 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      17:47:54.799 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:54.850 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:54.850 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:54.852 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:54.867 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:54.867 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:54.867 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:54.867 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.917 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:54.970 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.020 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 10270, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:55.071 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.121 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:55.174 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.191 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:55.226 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.257 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:55.278 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.333 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.383 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 10611, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:55.393 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:55.393 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=15) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:55.393 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:55.434 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.484 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:55.516 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:55.535 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.588 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.643 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.661 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:55.682 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:55.682 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:55.682 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:55.682 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.695 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      STILL UNRECEIVED, DELAY = : 10934, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:55.733 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.783 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:55.783 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:55.802 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:55.803 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:55.803 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:55.803 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:55.842 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:55.854 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.909 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:55.918 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:55.963 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.015 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.020 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:56.020 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 346 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      STILL UNRECEIVED, DELAY = : 11261, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:56.043 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:56.043 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=16) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:56.043 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:56.068 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.120 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:56.167 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:56.174 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.226 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.277 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.330 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 11588, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:56.383 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.435 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:56.486 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.494 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:56.537 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.579 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:56.589 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.640 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.690 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 11913, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:56.695 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:56.695 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=17) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:56.695 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:56.743 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:56.743 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:56.760 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:56.760 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:56.760 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:56.761 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:56.814 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.820 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:56.850 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:56.866 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:56.866 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:56.866 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:56.866 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.917 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:56.969 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 12238, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:57.020 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.021 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:57.071 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:57.125 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.142 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:57.180 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.227 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:57.235 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.286 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.328 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:57.328 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 347 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:57.337 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 12562, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:57.344 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:57.344 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=18) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:57.344 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:57.389 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.440 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:57.466 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:57.492 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.544 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.595 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.650 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 12885, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:57.700 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.719 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:57.736 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:57.736 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:57.736 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:57.736 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:57.736 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:57.754 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:57.755 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:57.755 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:57.755 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:57.789 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:57.806 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.860 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.867 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:57.867 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      17:47:57.911 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.965 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:57.968 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      STILL UNRECEIVED, DELAY = : 13211, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:57.993 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:57.993 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=19) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:57.993 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:58.017 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.072 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:58.114 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:58.126 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.177 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.227 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.281 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 13535, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:58.330 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:58.333 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.383 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.412 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      poll emptiness = true
      17:47:58.430 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:58.434 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.437 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:58.437 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Completed asynchronous auto-commit of offsets {exampletopic3-0=OffsetAndMetadata{offset=30061, leaderEpoch=null, metadata=''}}
      17:47:58.484 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.534 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.585 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:58.585 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:58.603 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:58.603 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:58.603 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:58.603 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.612 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      STILL UNRECEIVED, DELAY = : 13855, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:58.638 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:58.638 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 348 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:58.638 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:58.638 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=20) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:58.638 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:58.654 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.707 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:58.760 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.761 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:58.810 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.861 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.912 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:58.914 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:58.931 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:58.931 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:58.931 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:58.931 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 14181, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:58.984 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.039 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:59.086 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:59.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.143 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.166 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:59.197 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.249 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 14505, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:59.287 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:59.287 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=21) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:59.287 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:59.300 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.354 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:59.404 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.405 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:59.454 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:47:59.454 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:47:59.472 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:59.472 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:47:59.472 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:47:59.472 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.526 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.576 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 14824, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:59.628 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.642 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:59.681 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:47:59.728 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:47:59.733 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.783 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.836 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.844 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:47:59.861 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:47:59.861 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:47:59.862 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:47:59.862 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:47:59.910 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:47:59.912 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 15150, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:47:59.932 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:47:59.932 [Thread-0] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 349 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      17:47:59.932 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:59.932 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=22) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:47:59.933 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:47:59.966 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.017 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:00.053 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:00.070 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.121 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.175 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.225 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 15473, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:48:00.276 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.327 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:00.378 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.379 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:00.430 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.461 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:48:00.481 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:48:00.481 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:48:00.500 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:48:00.500 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:48:00.500 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:48:00.500 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.552 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 15800, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:48:00.582 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:48:00.582 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=23) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:48:00.582 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:48:00.607 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.659 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:00.703 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:00.711 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.762 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.814 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.859 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:48:00.877 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:48:00.877 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 30 disconnected.
      17:48:00.877 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 30 (/10.20.11.109:9092) could not be established. Broker may not be available.
      17:48:00.877 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.888 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending Heartbeat request to coordinator 10.20.2.110:9092 (id: 2147483627 rack: null)
      STILL UNRECEIVED, DELAY = : 16125, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:48:00.907 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Received successful Heartbeat response
      17:48:00.931 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:00.934 [Thread-0] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending metadata request (type=MetadataRequest, topics=exampletopic3) to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:48:00.983 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:01.029 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:01.034 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.085 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.114 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:48:01.140 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.193 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.217 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updating last seen epoch from 35 to 35 for partition exampletopic3-0
      17:48:01.217 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 350 to MetadataCache

      {cluster=Cluster(id = kd3GKmi_RHmTmyJja7HU0g, nodes = [10.20.2.110:9092 (id: 20 rack: us-west2-b)], partitions = [Partition(topic = exampletopic3, partition = 0, leader = 20, replicas = [30,20,10], isr = [20], offlineReplicas = [30,10])], controller = 10.20.2.110:9092 (id: 20 rack: us-west2-b))}

      STILL UNRECEIVED, DELAY = : 16452, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:48:01.234 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Added READ_UNCOMMITTED fetch request for partition exampletopic3-0 at offset 30061 to node 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:48:01.234 [Thread-0] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Built incremental fetch (sessionId=1250380980, epoch=24) for node 20. Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of 1 partition(s)
      17:48:01.234 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), implied=(exampletopic3-0)) to broker 10.20.2.110:9092 (id: 20 rack: us-west2-b)
      17:48:01.243 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.293 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:01.344 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.357 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:01.394 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) for sending metadata request
      17:48:01.394 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.2.109:9092 (id: 20 rack: us-west2-b) using address /10.20.2.109
      connect(id=20, address=/10.20.2.109:9092...)
      17:48:01.417 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.2.109 disconnected
      java.net.ConnectException: Socket is not connected
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:48:01.418 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 20 disconnected.
      17:48:01.418 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 20 (/10.20.2.109:9092) could not be established. Broker may not be available.
      17:48:01.418 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.468 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.522 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      STILL UNRECEIVED, DELAY = : 16778, value = 7aa1dd6e-1fb3-4c41-a59f-ef82fd75647a
      17:48:01.572 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.626 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      poll emptiness = true
      17:48:01.677 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.682 [Thread-0] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Committed offset 30061 for partition exampletopic3-0
      17:48:01.727 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
      17:48:01.756 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node 10.20.11.109:9092 (id: 30 rack: us-west2-c) using address /10.20.11.109
      connect(id=30, address=/10.20.11.109:9092...)
      17:48:01.761 [kafka-coordinator-heartbeat-thread | KafkaExampleConsume4] DEBUG org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-1, groupId=KafkaExampleConsume4] Node 20 sent an incremental fetch response for session 1250380980 with 0 response partition(s), 1 implied partition(s)
      17:48:01.777 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with /10.20.11.109 disconnected
      java.net.ConnectException: Connection refused
      at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
      at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
      at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
      at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:521)
      at org.apache.kafka.common.network.Selector.poll(Selector.java:473)
      at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
      at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
      at java.base/java.lang.Thread.run(Thread.java:834)
      17:48:01.777 [kafka-producer-network-thread | producer-1] DEBUG org

      ```

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              BrianAttwell Brian
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: