[KAFKA-4091] Unable to produce or consume on any topic

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.0.0
• Component/s: core
• Environment: Amazon Linux, t2.micro
• Assignee: Unassigned
• Reporter: Avi Chopra
• Created: 26/Aug/16 18:12
• Updated: 18/May/18 09:51
• Resolved: 18/May/18 09:51

Description

While trying to set kafka on 2 slave and 1 master box, got a weird condition where I was not able to consume or produce to a topic.

Using Mirror Maker to sync data between slave <--> Master. Getting following logs unending :

[2016-08-26 14:28:33,897] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:43,515] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:45,118] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:46,721] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:48,324] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:49,927] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:53,029] WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)

Only way I could recover was by restarting Kafka which produced this kind of logs :

[2016-08-26 14:30:54,856] WARN Found a corrupted index file, /tmp/kafka-logs/_consumer_offsets-43/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,856] INFO Recovering unflushed segment 0 in log __consumer_offsets-43. (kafka.log.Log) [2016-08-26 14:30:54,857] INFO Completed load of log __consumer_offsets-43 with log end offset 0 (kafka.log.Log) [2016-08-26 14:30:54,860] WARN Found a corrupted index file, /tmp/kafka-logs/consumer_offsets-26/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,860] INFO Recovering unflushed segment 0 in log __consumer_offsets-26. (kafka.log.Log) [2016-08-26 14:30:54,861] INFO Completed load of log __consumer_offsets-26 with log end offset 0 (kafka.log.Log) [2016-08-26 14:30:54,864] WARN Found a corrupted index file, /tmp/kafka-logs/_consumer_offsets-35/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)

ERROR Error when sending message to topic dr_ubr_analytics_limits with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

The consumer group command was showing a major lag.

This is my test phase so I was able to restart and recover from the master box but I want know what caused this issue and how can it be avoided. Is there a way to debug this issue?