[KAFKA-1895] Investigate moving deserialization and decompression out of KafkaConsumer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer
• Assignee: Jason Gustafson
• Reporter: Jay Kreps
• Created: 23/Jan/15 19:08
• Updated: 06/Nov/17 17:41

Description

The consumer implementation in KAFKA-1760 decompresses fetch responses and deserializes them into ConsumerRecords which are then handed back as the result of poll().

There are several downsides to this:
1. It is impossible to scale serialization and decompression work beyond the single thread running the KafkaConsumer.
2. The results can come back during the processing of other calls such as commit() etc which can result in caching these records a little longer.

An alternative would be to have ConsumerRecords wrap the actual compressed serialized MemoryRecords chunks and do the deserialization during iteration. This way you could scale this over a thread pool if needed.

Issue Links

• relates to: KAFKA-2045 Memory Management on the consumer