[KAFKA-15168] Handle overlapping remote log segments in RemoteLogMetadata cache

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Assignee: Kamal Chandraprakash
• Reporter: Kamal Chandraprakash
• Created: 08/Jul/23 19:25
• Updated: 24/Jul/23 18:31
• Resolved: 24/Jul/23 17:31

Description

For a partition p0 and a given leader epoch, the remote log manager can upload duplicate segments due to leader change. RemoteLogMetadata cache should handle the duplicate segments which may affect the follower and consumer.

(eg)

L0 uploaded the segment PZyYVdsJQWeBAdBDPqkcVA at t0 for offset range 10 - 90
L1 uploads the segment L5Ufv71IToiZYKgsluzcyA at t1 for offset range 5 - 100

In the RemoteLogLeaderEpochState class, the offsetToId is a navigable map. It sorts the entries by start-offset which keeps the state as:

(5 - 100) -> L5Ufv71IToiZYKgsluzcyA (T1)
(10 - 90) -> PZyYVdsJQWeBAdBDPqkcVA (T0)

For a fetch request with fetch-offset as 92, the RemoteLogLeaderEpochState will return the segment PZyYVdsJQWeBAdBDPqkcVA instead of L5Ufv71IToiZYKgsluzcyA, which doesn't have the respective offset and throws error back to the caller.

Issue Links

• links to: GitHub Pull Request #14004