[KAFKA-16073] Kafka Tiered Storage: Consumer Fetch Error Due to Delayed localLogStartOffset Update During Segment Deletion

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.1
• Fix Version/s: 3.8.0, 3.7.1, 3.6.3
• Component/s: core, Tiered-Storage
• Labels: KIP-405 kip-405 tiered-storage
• Flags: Important
• Assignee: hzh0425
• Reporter: hzh0425
• Created: 01/Jan/24 14:30
• Updated: 24/Apr/24 09:53
• Resolved: 24/Apr/24 09:53

Description

The identified bug in Apache Kafka's tiered storage feature involves a delayed update of localLogStartOffset in the UnifiedLog.deleteSegments method, impacting consumer fetch operations. When segments are deleted from the log's memory state, the localLogStartOffset isn't promptly updated. Concurrently, ReplicaManager.handleOffsetOutOfRangeError checks if a consumer's fetch offset is less than the localLogStartOffset. If it's greater, Kafka erroneously sends an OffsetOutOfRangeException to the consumer.

In a specific concurrent scenario, imagine sequential offsets: offset1 < offset2 < offset3. A client requests data at offset2. While a background deletion process removes segments from memory, it hasn't yet updated the LocalLogStartOffset from offset1 to offset3. Consequently, when the fetch offset (offset2) is evaluated against the stale offset1 in ReplicaManager.handleOffsetOutOfRangeError, it incorrectly triggers an OffsetOutOfRangeException. This issue arises from the out-of-sync update of localLogStartOffset, leading to incorrect handling of consumer fetch requests and potential data access errors.

Issue Links

• is a child of: KAFKA-15420 Kafka Tiered Storage V1
• links to: GitHub Pull Request #15141
• links to: GitHub Pull Request #15631
• links to: GitHub Pull Request #15747
• links to: GitHub Pull Request #15748