[FLINK-24398] KafkaSourceFetcherManager should re-use an existing SplitFetcher to commit offset if possible

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Labels: auto-deprioritized-minor pull-request-available
• Assignee: Unassigned
• Reporter: Dong Lin
• Created: 28/Sep/21 12:39
• Updated: 14/Apr/22 22:38

Description

Currently KafkaSourceFetcherManager::commitOffset() will create a new SplitFetcher if the fetchers.get(0) == null. As a result, if the first fetcher has already been closed and removed, N fetchers will be created for N commitOffset() invocations.

A more efficient approach is to re-use an existing fetcher to commit the offset if there is any running fetchers in this KafkaSourceFetcherManager.

Issue Links

• links to: GitHub Pull Request #17375