[KAFKA-5195] Endless NotLeaderForPartitionException for ReplicaFetcherThread

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.1.1
• Component/s: replication

Environment

3 Kafka brokers on top of Kubernetes, using Docker image wurstmeister/kafka:0.10.1.1.

Environment variables:
      KAFKA_ADVERTISED_HOST_NAME: kafka-ypimp-2
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-ypimp-0:2181,zookeeper-ypimp-1:2181,zookeeper-ypimp-2:2181
      KAFKA_DELETE_TOPIC_ENABLE: true
      KAFKA_BROKER_ID: 2
      JMX_PORT: 1099
      KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka-ypimp-2.default.svc.cluster.local -Dcom.sun.management.jmxremote.rmi.port=1099
      KAFKA_LOG_RETENTION_HOURS: 96
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: false



Zookeeper version: 3.4.8.
Number of Zk nodes: 3.

• Assignee: Unassigned
• Reporter: Andrea Gardiman
• Created: 08/May/17 14:20
• Updated: 19/Jul/18 20:23

Description

One of the 3 brokers is suddenly in a bad state. It endlessly prints out the following message, for every partition:
[2017-05-08 13:51:16,748] ERROR [ReplicaFetcherThread-0-0], Error for partition [partition_name,5] to broker 0:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

In zookeeper, under /brokers/ids, I can't find the zkNode for broker 2. There are only the zkNodes 0 and 1.

What kind of error this can be?
Please, let me know if you need some more informaton, I don't know hot to properly debug it.

Many thanks.