[KAFKA-1106] HighwaterMarkCheckpoint failure puting broker into a bad state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 0.8.0
• Component/s: core
• Labels: reliability
• Assignee: Unassigned
• Reporter: David Lao
• Created: 29/Oct/13 01:34
• Updated: 17/Mar/18 06:46
• Resolved: 17/Mar/18 06:46

Description

I'm encountering a case where broker get stuck due to HighwaterMarkCheckpoint failing to recover from reading what appear to be corrupted isr entries. Once in this state, leader election can never succeed and hence stalling the entire cluster.

Please see the detailed stack trace from the attached log. Perhaps failing fast when HighwaterMarkCheckpoint fails to read would force the broker to restart and recover.