[KAFKA-4696] Streams standby task assignment should be state-store aware

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.2.0, 0.11.0.0
• Fix Version/s: 2.6.0
• Component/s: streams
• Assignee: A. Sophie Blee-Goldman
• Reporter: Damian Guy
• Created: 25/Jan/17 11:08
• Updated: 05/May/20 17:57
• Resolved: 05/May/20 17:57

Description

Task Assignment is currently not aware of which tasks have State Stores. This can result in uneven balance of standby task assignment as all tasks are assigned, but only those tasks with state-stores are ever created by StreamThread. So what seems like an optimal strategy during assignment time could be sub-optimal post-assignment.

For example, lets say we have 4 tasks (2 with state-stores), 2 clients, numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks. One of the clients may end up with both state-store tasks, while the other has none.

Further to this, standby task configuration is currently "all or nothing". It might make sense to allow more fine grained configuration, i.e., the ability to specify the number of standby replicas individually for each stateful operator.

Issue Links

• contains: KAFKA-6583 Metadata should include number of state stores for task
• is blocked by: KAFKA-6583 Metadata should include number of state stores for task
• links to: GitHub Pull Request #4615
• links to: GitHub Pull Request #5012
• mentioned in: Page Loading...