[KAFKA-5578] Streams Task Assignor should consider the staleness of state directories when allocating tasks

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: streams
• Assignee: Unassigned
• Reporter: Damian Guy
• Created: 10/Jul/17 17:49
• Updated: 15/May/20 07:37
• Resolved: 15/May/20 07:36

Description

During task assignment we use the presence of a state directory to assign precedence to which instances should be assigned the task. We first chose previous active tasks, but then fall back to the existence of a state dir. Unfortunately we don't take into account the recency of the data from the available state dirs. So in the case where a task has run on many instances, it may be that we chose an instance that has relatively old data.

When doing task assignment we should take into consideration the age of the data in the state dirs. We could use the data from the checkpoint files to determine which instance is most up-to-date and attempt to assign accordingly (obviously making sure that tasks are still balanced across available instances)

Issue Links

• is a child of: KAFKA-6039 Improve TaskAssignor to be more load balanced
• Is contained by: KAFKA-6145 Warm up new KS instances before migrating tasks - potentially a two phase rebalance