We are running a 10 node Kafka 0.8.1 cluster and experienced a failure as following.
At some time, broker A stopped acting as controller any more. We see this by kafka.controller - KafkaController - ActiveControllerCount in JMX metrics jumped from 1 to 0.
In the meanwhile, broker A was still running and registering itself in the zookeeper /kafka/controller node. So no other brokers could be elected as new controller.
Since that the cluster was running without controller. Producers and consumers still worked. But functions requiring a controller such as new topic leader election and topic leader failover were not working any more.
A force restart of broker A could lead to a controller election and bring the cluster back to a correct state.
Here is our brief observations. I can provide more necessary informations if needed.