[KAFKA-12959] Prioritize assigning standby tasks to threads without any active tasks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.2.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Ravi Bhardwaj
• Created: 17/Jun/21 06:11
• Updated: 05/Mar/22 08:12
• Resolved: 05/Mar/22 08:12

Description

Currently while distributing the standby tasks streams does not check if there are threads without any tasks or with less number of tasks. This can lead to few threads getting assigned both active and standby tasks when are threads within the same instance without any tasks assigned.

Example:

App 1:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
 New active tasks: [0_1]
 New standby tasks: [1_0]
 Existing active tasks: []
 Existing standby tasks: [1_0]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
 New active tasks: [1_1]
 New standby tasks: []
 Existing active tasks: [1_1]
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []

 

App2:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
 New active tasks: [1_0]
 New standby tasks: [1_1]
 Existing active tasks: []
 Existing standby tasks: [1_0, 1_1]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
 New active tasks: [0_0]
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []

 The standby tasks in both apps is assigned to Thread-1 even though it has an active tasks when Thread-3 and Thread-4 didn't have any tasks assigned.

 

Issue Links

• links to: GitHub Pull Request #11493