[FLINK-20835] Implement FineGrainedSlotManager

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yangze Guo
• Reporter: Yangze Guo
• Created: 04/Jan/21 06:44
• Updated: 13/Feb/24 11:11
• Resolved: 04/Feb/21 02:11

Description

Introduce SlotManager plugin for fine-grained resource management.

• Request slot from TaskExecutor with the actual resource profiles.
• Use ResourceTracker to bookkeep the resource requirements
• Introduce TaskExecutorTracker, which bookkeep the total / available resource and slot assignment for registered task executor.
  • Bookkeep task manager total and available resources.
  • Bookkeep slot allocations and assignments.
  • Intorduce PendingTaskManager.
  • Map registered task executors to matching PendingTaskManager.

Issue Links

• causes: FLINK-34434 DefaultSlotStatusSyncer doesn't complete the returned future
• is blocked by: FLINK-20836 Register TaskManager with total and default slot resource profile in SlotManager
• links to: GitHub Pull Request #14647