[KAFKA-17505] New consumer seekToBeginning/End should run in background thread

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848-client-support
• Assignee: TaiJuWu
• Reporter: Lianet Magrans
• Created: 09/Sep/24 20:31
• Updated: 08/Oct/24 18:40
• Resolved: 08/Oct/24 18:40

Description

seekToBeginning and seekToEnd perform 2 operations:

1. retrieve assigned partitions from the subscription state if no partitions provided by the user
2. request offsets reset for those partitions (to EARLIEST or LATEST offset) (update partition state in the subscription state)

In the new asyn consumer, these operation are performed sequentially in the app thread, which could lead to errors if retrieving the assigned partitions first, and then requesting the reset: the assignment could change in the background thread in-between 1) and 2) and the reset would fail with
"No current assignment for partition"
 

We should move the implementation to the background thread, where all modifications to the subscription state should occur. We could introduce a new event (something like OffsetResetEvent or similar) that would take as input the set of partitions received in the api calls and the OffsetResetStrategy (EARLIEST or LATEST). When processing it in the background, it would safely perform the same 2 actions that are currently in the app thread seekToBeginning/End.

Issue Links

• relates to: KAFKA-17448 New consumer seek should update positions in background thread
• links to: GitHub Pull Request #17230