Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.9.0.1
-
Fix Version/s: 0.10.1.0
-
Component/s: controller
-
Labels:None
Description
We recently upgraded from 0.8.2.1 to 0.9.0.1. Since then, several times per day, the controllers in our clusters have their connection to all brokers disconnected, and then successfully reconnected a few hundred ms later. Each time this occurs we see a brief spike in our 99th percentile produce and consume times, reaching several hundred ms.
Here is an example of what we're seeing in the controller.log:
[2016-06-28 14:15:35,416] WARN [Controller-151-to-broker-160-send-thread], Controller 151 epoch 106 fails to send request {…} to broker Node(160, broker.160.hostname, 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 160 was disconnected before the response was read
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
at scala.Option.foreach(Option.scala:236)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
at kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:129)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:180)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
... one each for all brokers (including the controller) ...
[2016-06-28 14:15:35,721] INFO [Controller-151-to-broker-160-send-thread], Controller 151 connected to Node(160, broker.160.hostname, 9092) for sending state change requests (kafka.controller.RequestSendThread)
… one each for all brokers (including the controller) ...
Issue Links
- links to
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
We saw this issue yesterday, I don't know if this helps, but it may be useful while debugging.
- Kafka 0.9.0.1 and 0.9.0.1 clients.
- There is ISR shrink and immediate ISR expand visible in graphs based on JMX of Kafka brokers.
- Consumers were unable to commit offsets at that time.
2016-08-02 14:25:29.589 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-7]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.560 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-2]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.562 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-0]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.563 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-5]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.570 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-6]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.570 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-3]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.570 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-4]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.572 INFO o.a.k.c.c.internals.ConsumerCoordinator [Consumer-1]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will find new coordinator and retry 2016-08-02 14:25:52.572 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-7]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.573 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-2]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.573 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-0]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.574 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-5]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.575 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-6]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.575 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-3]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.575 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-4]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.576 INFO o.a.k.c.c.internals.AbstractCoordinator [Consumer-1]: Marking the coordinator 2147483646 dead. 2016-08-02 14:25:52.576 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-7]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.577 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-2]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.577 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-0]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.577 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-5]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.578 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-6]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.578 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-3]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.578 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-4]: Auto offset commit failed: The request timed out. 2016-08-02 14:25:52.579 WARN o.a.k.c.c.internals.ConsumerCoordinator [Consumer-1]: Auto offset commit failed: The request timed out.
Production of messages was failing for few seconds this time. I found no issue in logs of consumers. Exceptions in controller.log are the same as in description. Multiple ISR shrinks and expands happened during 6:30 pm to 9:30 pm based on our monitoring and graphs, then the issue disappeared.
2016-08-10 20:01:40.611 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired 2016-08-10 20:01:40.683 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired ... 2016-08-10 20:01:43.714 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired 2016-08-10 20:01:43.714 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired 2016-08-10 20:01:43.838 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-3, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.840 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-0, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.840 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-0, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.840 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-8, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.841 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-4, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.841 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-0, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.841 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-10, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.842 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-0, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.842 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-6, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.842 WARN o.a.kafka.clients.producer.internals.Sender [kafka-producer-network-thread | (consumer group)]: Got error produce response with correlation id 113676841 on topic-partition (topic)-6, retrying (0 attempts left). Error: NETWORK_EXCEPTION 2016-08-10 20:01:43.919 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired 2016-08-10 20:01:43.920 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired ... 2016-08-10 20:01:43.971 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired 2016-08-10 20:01:43.972 WARN (application class) [kafka-producer-network-thread | (consumer group)]: Write to Kafka failed: (topic), org.apache.kafka.common.errors.TimeoutException: Batch Expired
Could we be hitting the idle connection timeout? The default setting for connections.max.idle.ms is 10 minutes. When the cluster is stable, I'm not sure there's much traffic from the controller, so we may proactively disconnect the connection. Perhaps someone can try increasing it?
Hi Jason and others,
I have finally found the root cause for our case. I was grep-ing the controller logs for a while to hide disturbing stack traces and noticed that all warnings from last month always mention the same topic, which is quite strange because we have about 50 of them total. This one is the largest one, with the highest throughput, but not strongly larger than others. Time stamps of the warnings are also from the afternoon or early evening when Kafka receives the highest traffic, so I believe the issue is somehow related to performance. There is also always leader=2, ISR changes between [1,2] and [1,2,3], related partitions are always 2, 7, 11.
While working on a completely different task yesterday, I also by coincidence found that consumption speed of events buffered in Kafka for this topic reached its limits, other topics were ok. Producers were only little slower than consumers. So we are trying to add few more partitions to increase parallelism both in consumers and internally in Kafka (replication). I guess it will help.
# egrep 'WARN|IOException' controller.log | less -S
[2016-08-10 19:00:14,502] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2,3],zk_version=145,replicas=[2,3,1]},{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2,3],zk_version=120,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(1, (host1), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
[2016-08-10 19:00:14,506] WARN [Controller-1-to-broker-4-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2,3],zk_version=145,replicas=[2,3,1]},{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2,3],zk_version=120,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(4, (host4), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 4 was disconnected before the response was read
[2016-08-10 19:00:14,506] WARN [Controller-1-to-broker-2-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2,3],zk_version=145,replicas=[2,3,1]},{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2,3],zk_version=120,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(2, (host2), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
[2016-08-10 19:00:14,502] WARN [Controller-1-to-broker-3-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2,3],zk_version=145,replicas=[2,3,1]},{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2,3],zk_version=120,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(3, (host3), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 3 was disconnected before the response was read
[2016-08-10 19:50:41,999] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2],zk_version=129,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(1, (host1), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
[2016-08-10 19:50:42,002] WARN [Controller-1-to-broker-3-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2],zk_version=129,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(3, (host3), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 3 was disconnected before the response was read
[2016-08-10 19:50:42,002] WARN [Controller-1-to-broker-4-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2],zk_version=129,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(4, (host4), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 4 was disconnected before the response was read
[2016-08-10 19:50:42,002] WARN [Controller-1-to-broker-2-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=11,controller_epoch=18,leader=2,leader_epoch=23,isr=[1,2],zk_version=129,replicas=[2,1,3]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(2, (host2), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
[2016-08-10 20:10:12,044] WARN [Controller-1-to-broker-2-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=7,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=136,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(2, (host2), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
[2016-08-10 20:10:12,044] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=7,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=136,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(1, (host1), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
[2016-08-10 20:10:12,044] WARN [Controller-1-to-broker-3-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=7,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=136,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(3, (host3), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 3 was disconnected before the response was read
[2016-08-10 20:10:12,044] WARN [Controller-1-to-broker-4-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=7,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=136,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(4, (host4), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 4 was disconnected before the response was read
[2016-08-10 21:04:11,998] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=168,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(1, (host1), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
[2016-08-10 21:04:12,001] WARN [Controller-1-to-broker-4-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=168,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(4, (host4), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 4 was disconnected before the response was read
[2016-08-10 21:04:11,998] WARN [Controller-1-to-broker-2-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=168,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(2, (host2), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
[2016-08-10 21:04:11,998] WARN [Controller-1-to-broker-3-send-thread], Controller 1 epoch 20 fails to send request {controller_id=1,controller_epoch=20,partition_states=[{topic=(always same topic),partition=2,controller_epoch=18,leader=2,leader_epoch=26,isr=[1,2],zk_version=168,replicas=[2,3,1]}],live_brokers=[{id=1,end_points=[{port=9092,host=(host1),security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=(host2),security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=(host3),security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=(host4),security_protocol_type=0}]}]} to broker Node(3, (host3), 9092). Reconnecting to broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 3 was disconnected before the response was read
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/1734
KAFKA-3916: Check for disconnects properly before sending from the controller
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-3916
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1734.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1734
commit 1cc38623c047128d9c83309c66f37491726c8082
Author: Jason Gustafson <jason@confluent.io>
Date: 2016-08-12T00:40:32Z
KAFKA-3916: Check for disconnects properly before sending from the controller
Issue resolved by pull request 1734
https://github.com/apache/kafka/pull/1734
Github user asfgit closed the pull request at:
Hi,
I am using Kafka version 0.10.0.2 in the production environment. I am facing issues in my kafka broker and consumer machines which is very similar to issue described here.
Controller logs are very similar to the one described above:
|WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]} to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]} to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]} to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]} to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]} to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]} to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]} to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,010] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]} to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]} to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]} to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]} to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]} to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]} to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]} to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]} to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]} to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]} to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,060] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]} to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread - [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]} to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
Consumer error log:
|2017-08-26 23:47:54,738 WARN [kafka-producer-network-thread | producer-8] o.a.k.c.p.i.Sender [kafka-producer-network-thread | (consumer group)] Got error produce response with correlation id 10322 on topic-partition topic-0, retrying (9 attempts left). Error: NETWORK_EXCEPTION
I am having 5000 topics right now with a retention period of 1 hour. The maximum size of data during peak load is 3-4 GB in a machine and I am having 6 kafka broker machines of 6 core and 16 GB RAM.
Can someone please point out if there's something wrong in my approach? Do I need to update to latest version?
I have this problem with 0.10 vers.