[KAFKA-6039] Improve TaskAssignor to be more load balanced

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: streams
• Labels: optimization user-experience
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 10/Oct/17 00:56
• Updated: 15/May/20 07:37
• Resolved: 15/May/20 07:36

Description

Today our task placement may still generate sub-optimal assignment regarding load balance. One reason is that it does not account for sub-topologies. For example say you have an aggregation following from a repartition topic, then you will end up with two sub-topologies where the first one is very light and the second one is computational heavy with state stores, however when we consider their tasks we treat them equally so in the worst case one client can get X number of tasks from first sub-topology and be very idle while the other getting X number of tasks from the second sub-topology and busy to death.

One strawman approach to make this better is try to achieve balance across sub-topologies: i.e. each client trying to get similar amount of tasks within a sub-topology. However there are some more considerations to include (as mentioned in the sub-taks).

Issue Links

• is a parent of: KAFKA-5578 Streams Task Assignor should consider the staleness of state directories when allocating tasks
• Is contained by: KAFKA-6145 Warm up new KS instances before migrating tasks - potentially a two phase rebalance
• links to: GitHub Pull Request #4624