[KAFKA-15173] Consumer event queues should be bounded

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor events
• Assignee: Kirk True
• Reporter: Philip Nee
• Created: 10/Jul/23 19:30
• Updated: 30/Jan/24 19:14

Description

The async consumer uses ApplicationEventQueue and BackgroundEventQueue to facilitate message passing between the application thread and the background thread.  The current implementation is boundless, which can potentially cause OOM and other performance-related issues.

I think the queues need a finite bound, and we need to decide how to handle the situation when the bound is reached.  In particular, I would like to answer these questions:

 

1. What should the upper limit be for both queues: Can this be a configurable, memory-based bound? Or just an arbitrary number of events as the bound.
2. What should happen when the application event queue is filled up?  It seems like we should introduce a new exception type and notify the user that the consumer is full.
3. What should happen when the background event queue is filled up? This seems less likely to happen, but I imagine it could happen when the user stops polling the consumer, causing the queue to be filled.
4. Is it necessary to introduce a public configuration for the queue? I think initially we would select an arbitrary constant number and see the community feedback to make a forward plan accordingly.

Issue Links

• is required by: KAFKA-14246 Update threading model for Consumer