[KAFKA-18018] Consumption failing with DefaultStatePersister for 1 million records

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Abhinav Dixit
• Created: 14/Nov/24 14:33
• Updated: 14/Nov/24 14:33

Description

I was running some performance tests and I couldn't consume 1,000,000 records using 10 share consumers. I've narrowed down the issue to DefaultStatePersister . When I run the same tests using NoOpShareStatePersister, then I don't see the reliability issue.

Steps to reproduce (could be maybe reproduced by other ways as well, but this is how I noticed it consistently) - # Create a topic with a single partition

1. Produce 1000 records into the topic and consume it with the help of 10 share consumers.
2. Again produce 1000 records into the topic and consume it with the help of 10 share consumers.
3. Produce 1,000,000 records into the topic and consume it with the help of 10 share consumers.

You'll see the records not getting consumed even within 5 minutes (It shouldn't take more than 5-6 seconds to consume 1,000,000 records with 10 share consumers, based on my past experience)

PS - I haven't noticed any issues using console share consumers, so probably an issue with scale