[KAFKA-1908] Split brain

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.8.2.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Alexey Ozeritskiy
• Created: 29/Jan/15 12:23
• Updated: 23/Jun/19 03:01
• Resolved: 06/Jan/17 19:53

Description

In some cases, there may be two leaders for one partition.
Steps to reproduce:

1. We have 3 brokers, 1 partition with 3 replicas:

   TopicAndPartition: [partition,0]    Leader: 1       Replicas: [2,1,3]       ISR: [1,2,3]
   

2. controller works on broker 3
3. let the kafka port be 9092. We execute on broker 1:

   iptables -A INPUT -p tcp --dport 9092 -j REJECT
   

4. Initiate replica election
5. As a result:
   Broker 1:

   TopicAndPartition: [partition,0]    Leader: 1       Replicas: [2,1,3]       ISR: [1,2,3]
   

   Broker 2:

   TopicAndPartition: [partition,0]    Leader: 2       Replicas: [2,1,3]       ISR: [1,2,3]
   

6. Flush the iptables rules on broker 1

Now we can produce messages to

[partition,0]

. Replica-1 will not receive new data. A consumer can read data from replica-1 or replica-2. When it reads from replica-1 it resets the offsets and than can read duplicates from replica-2.

We saw this situation in our production cluster when it had network problems.