[KAFKA-12675] Improve sticky general assignor scalability and performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Assignee: Luke Chen
• Reporter: Luke Chen
• Created: 16/Apr/21 03:25
• Updated: 04/Jun/21 08:05
• Resolved: 02/Jun/21 22:02

Description

Currently, we have "general assignor" for non-equal subscription case and "constrained assignor" for all equal subscription case. There's a performance test for constrained assignor with:

topicCount = 500;
partitionCount = 2000;
consumerCount = 2000;

in testLargeAssignmentAndGroupWithUniformSubscription, total 1 million partitions and we can complete the assignment within 2 second in my machine.

However, if we let 1 of the consumer subscribe to only 1 topic, it'll use "general assignor", and the result with the same setting as above is: OutOfMemory,
Even we down the count to:

topicCount = 50;
partitionCount = 1000;
consumerCount = 1000;

We still got OutOfMemory.

With this setting:

topicCount = 50;
partitionCount = 800;
consumerCount = 800;

We can complete in 10 seconds in my machine, which is still slow.

 

Since we are going to set default assignment strategy to "CooperativeStickyAssignor" soon,  we should improve the scalability and performance for sticky general assignor.

Issue Links

• links to: GitHub Pull Request #10552