[KAFKA-8799] Support ability to pass global user data to consumers during Assignment

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer, streams
• Assignee: Unassigned
• Reporter: Vinoth Chandar
• Created: 13/Aug/19 21:26
• Updated: 14/Aug/19 18:41

Description

This is a follow up from KAFKA-7149 

Background : 

Although we reduced the size of the AssignmentInfo object sent during each rebalance from leader to all followers in KAFKA-7149, we still repeat the same partitionsByHost map for each host (all this when interactive queries are enabled) and thus still end up sending redundant bytes to the broker and also logging a large kafka message.

With 100s of streams instances, this overhead can grow into tens of megabytes easily.  

Proposal :

Extend the group assignment protocol to be able to support passing of an additional byte[], which can now contain the HostInfo -> partitions/partitionsByHost data just one time.

final class GroupAssignment {
        private final Map<String, Assignment> assignments;

        // bytes sent to each consumer from leader
        private final byte[] globalUserData
        ...
    }

 
This can generally be handy to any other application like Streams, that does some stateful processing or lightweight cluster management
 

Issue Links

• is related to: KAFKA-7149 Reduce assignment data size to improve kafka streams scalability