[KAFKA-14878] Reduce assignment data size to improve Kafka Consumer scalability

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Armand
• Created: 04/Apr/23 13:40
• Updated: 04/Apr/23 13:40

Description

This issue is linked to https://issues.apache.org/jira/browse/KAFKA-7149
which describes scalability problems with Kafka streams and was resolved by this PR.

We have the same problem with Kafka Consumer. If we scale our solution with 50 instances (each instance has 5 consumers, so a consumer group can contain 50 * 5 = 250 consumers) with 5000 partitions, the leader of the group will send SyncGroup requests exceeding 100 MB to the group coordinator.

The only solution we found is to increase the max.message.bytes of the __consumer_offsets topic. Is there any plan to compress requests from the consumer in order to avoid sending too big requests over the network?