[KAFKA-15695] Local log start offset is not updated on the follower after rebuilding remote log auxiliary state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.7.0
• Component/s: replication, Tiered-Storage
• Labels: KIP-405 tiered-storage
• Assignee: Nikhil Ramakrishnan
• Reporter: Nikhil Ramakrishnan
• Created: 26/Oct/23 16:21
• Updated: 26/Feb/24 20:23
• Resolved: 12/Dec/23 16:14

Description

In 3.6, the local log start offset is not updated when reconstructing the auxiliary state of the remote log on a follower.

The impact of this bug is significant because, if this follower becomes the leader before the local log start offset has had a chance to be updated, reads from any offset between [wrong log start offset; actual log start offset] will be routed on the local storage, which does not contain the corresponding data. Consumer reads will in this case never be satisfied.

 

Reproduction case 1:

1. Create cluster with 2 brokers, broker 0 and broker 1.
2. Create a topic topicA with RF=1, 1 partition (topicA-0) on broker 0, and 1 batch per segment.
3. Produce 3 records to topicA, such that segment 1 and segment 2 with the first two records are copied to remote and deleted from local storage.
4. Reassign replica to add broker 1 to the replica set for topicA-0, and elect broker 1 as the leader.
5. Try to consume from the beginning of topicA-0.

 

Reproduction case 2:

1. Create a cluster with 2 brokers, broker 0 and broker 1.
2. Create a topic topicA with RF=2, 1 partition (topicA-0) and 2 batches per segment, with broker 0 as the leader.
3. Stop broker 1, and produce 3 records to topicA, such that segment 1 with the first two records are copied to remote and deleted from local storage.
4. Start broker 1, let it catch up with broker 0.
5. Stop broker 0 such that broker 1 is elected as the leader, and try to consume from the beginning of topicA-0.

Consumer read will not be satisfied in these cases because the local log start offset is not updated on broker 1 when it builds the auxiliary state of the remote log segments.

Issue Links

• is a child of: KAFKA-15420 Kafka Tiered Storage V1
• links to: GitHub Pull Request #14649