[KAFKA-6715] Leader transition for all partitions lead by two brokers without visible reason

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.11.0.2
• Component/s: core, replication
• Environment: Kafka cluster on Amazon AWS EC2 r4.2xlarge instances with 5 nodes and a Zookeeper cluster on r4.2xlarge instances with 3 nodes. The cluster is distributed across 2 availability zones.
• Assignee: Unassigned
• Reporter: Uwe Eisele
• Created: 26/Mar/18 13:23
• Updated: 26/Mar/18 13:38

Description

In our cluster we experienced a situation, in which the leader of all partitions lead by two brokers has been moved mainly to one other broker (broker 1).

We don't know why this happend. At this time there was not broker outage, nor a broker shutdown has been initiated. The Zookeeper nodes of the affected brokers (/brokers/ids/3, /brokers/ids/4) has not been modified during this time.

In addition there are no logs that would indicate a leader transition for the affected brokers. We would expect to see a "sending become-leader LeaderAndIsr request" in the controller log for each partition, as well a "completed LeaderAndIsr request" in the state change log of the Kafka brokers that becomes the new leader and follower. Our log level for the kafka.controller and the state change log is set to TRACE.

Though all Brokers are running, the situation does not recover. It sticks in a highly imbalanced leader distribution, in which two brokers are no leader for any partition, and one broker is the leader for almost all partitions.

kafka-controller Log (Level TRACE):
[2018-03-19 17:03:54,042] TRACE [Controller 3]: Leader imbalance ratio for broker 5 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:03:54,042] TRACE [Controller 3]: Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:03:54,042] TRACE [Controller 3]: Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:03:54,043] TRACE [Controller 3]: Leader imbalance ratio for broker 3 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:03:54,043] TRACE [Controller 3]: Leader imbalance ratio for broker 4 is 0.0 (kafka.controller.KafkaController)
...
[2018-03-19 17:08:54,049] TRACE [Controller 3]: Leader imbalance ratio for broker 5 is 0.8054794520547945 (kafka.controller.KafkaController)
[2018-03-19 17:08:54,050] TRACE [Controller 3]: Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:08:54,050] TRACE [Controller 3]: Leader imbalance ratio for broker 2 is 0.4807692307692308 (kafka.controller.KafkaController)
[2018-03-19 17:08:54,051] TRACE [Controller 3]: Leader imbalance ratio for broker 3 is 1.0 (kafka.controller.KafkaController)
[2018-03-19 17:08:54,053] TRACE [Controller 3]: Leader imbalance ratio for broker 4 is 1.0 (kafka.controller.KafkaController)
...
[2018-03-19 17:23:54,080] TRACE [Controller 3]: Leader imbalance ratio for broker 5 is 0.8054794520547945 (kafka.controller.KafkaController)
[2018-03-19 17:23:54,081] TRACE [Controller 3]: Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController)
[2018-03-19 17:23:54,081] TRACE [Controller 3]: Leader imbalance ratio for broker 2 is 0.4807692307692308 (kafka.controller.KafkaController)
[2018-03-19 17:23:54,082] TRACE [Controller 3]: Leader imbalance ratio for broker 3 is 1.0 (kafka.controller.KafkaController)
[2018-03-19 17:23:54,084] TRACE [Controller 3]: Leader imbalance ratio for broker 4 is 1.0 (kafka.controller.KafkaController)

The imbalance was recognized by the controller, but nothing happend. The imbalance for broker 1 seems to be wrong, because this broker has taken the leader role for most of the partitions.

In addition it seems that the ReplicaFetcherThreads die without any log message (see attached stack trace), though we think this is not possible... However, we would expect log messages that state, that fetchers for partitions have been removed, as well that the ReplicaFetcherThreads are shutting down. The log level for kafka is set to INFO. In other situations, when a broker is shut down we see such entries in the log files.

Besides that, this caused underreplicated partitions. It seems that no broker fetches from the partitions with the newly assigned leaders. Like the situation with the highly imbalanced leader distribution the cluster sticks in this state and does not recover.

This is a recurring problem, however we cannot reproduce it.