[KAFKA-851] Broken handling of leader and isr request leads to incorrect high watermark checkpoint file

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Component/s: replication
• Labels: kafka-0.8 p1
• Assignee: Neha Narkhede
• Reporter: Neha Narkhede
• Created: 04/Apr/13 22:35
• Updated: 05/Apr/13 04:28
• Resolved: 05/Apr/13 04:28

Description

The broker depends on receiving a list of all partitions from the controller on startup. It uses this information to create a list of partitions that will get check pointed to the high watermark checkpoint file. However, during a make follower operation, it adds a partition to the high watermark checkpoint list only if its leader is alive. Due to this, even if the controller sends a full list of partitions to the broker, replica manager filters it to keep only those partitions whose leader is alive. This leads to the high watermark value for the rest of those partitions to reset to 0. Which, in turn, leads to the follower to fetch from the beginning of leader's log instead of min(log end offset, high watermark). The effect of this is very long lag on the replica fetchers leading to high under replicated partition count