[FLINK-11815] Support execution with static resources (Flip-6): E.g. spreading tasks across available set of TMs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.4, 1.7.2, 1.8.0
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Piotr Nowojski
• Created: 04/Mar/19 15:45
• Updated: 02/Nov/20 15:11
• Resolved: 02/Nov/20 15:11

Description

FLIP-6 code path changed how tasks are distributed across the cluster which is causing problems in some scenarios. This issue intends to improve this.

Known issues:
1. key groups are currently assigned randomly, ignoring their size. Previously larger key groups were evenly spread around the available task managers
2. if task slots are registered before slot request, the code have a tendency to group requests together on the same machine because we are using a LinkedHashMap
3. ???

CC [e-mail]

Issue Links

• is related to: FLINK-12921 Flink Job Scheduling