[KAFKA-13322] Java client produces a large amount of garbage during a poll

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: clients
• Labels: new-consumer-threading-should-fix
• Assignee: Unassigned
• Reporter: Michail
• Created: 24/Sep/21 10:38
• Updated: 11/Feb/22 07:24
• Resolved: 04/Feb/22 20:55

Description

The java kafka consumer creates multiple collections during a single poll command: in my test system i have a consumer that polls a topic with 100 partitions and even though no messages are coming through, the code allocates around 100M per 5 minutes.
 
I've investigated the allocations and the biggest ones can be easily avoided by moving them to the instance level, something that can be done as KafkaConsumer is not thread safe. Purpose of this Jira is to get rid of most of them applying either this or a similar approach.

Issue Links

• is related to: SPARK-36837 Upgrade Kafka to 3.1.0
• links to: GitHub Pull Request #11353