[KAFKA-8739] rejoining broker fails to sanity check existing log segments

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: replication
• Assignee: Unassigned
• Reporter: sanjiv marathe
• Created: 31/Jul/19 08:48
• Updated: 31/Jul/19 08:48

Description

kafka claims it can be used as a storage. But following scenario proves other wise.

1. Consider a topic with single partition, repl-factor 2, with two brokers, say A and B.... with A is the leader.
2. Broker B fails due to sector errors. Sysadmin fixes the issues and brings it up again after a few minutes. A few log segments are lost/corrupted.
3. Broker B catches up with missed out msgs by fetching them from the leader A, but does not realize the issue with earlier log segments.
4. Broker A fails, B becomes the leader.
5. A new consumer requests msgs from the beginning. Broker B fails to deliver msgs belonging to corrupted log segments.

Suggested solution

A broker, immediately after coming up, should go through a sanity check, e.g. CRC check of its log segments. Any corrupted/lost, should be refetched from the leader.