[FLINK-12761] Fine grained resource management

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.8.0, 1.9.0
• Component/s: Runtime / Coordination
• Labels: Umbrella pull-request-available
• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 06/Jun/19 07:01
• Updated: 04/Dec/19 16:18
• Resolved: 04/Dec/19 16:18

Description

This is an umbrella issue for enabling fine grained resource management in Flink.

Fine grained resource management is a big topic that requires long term efforts. There are many issues to be addressed and designing decisions to be made, some of which may not be resolved in short time. Here we propose our design and implementation plan for the upcoming release 1.9, as well as our thoughts and ideas on the long term road map on this topic.

A practical short term target is to enable fine grained resource management for batch sql jobs only in the upcoming Flink 1.9. This is necessary for batch operators added from blink to achieve good performance.

Please find detailed design and implementation plan in attached docs. Any comment and feedback are welcomed and appreciated.

Issue Links

• links to: Implementation Steps
• links to: Memory Management Design
• links to: Resource Scheduling Design

Sub-Tasks

• Set resource profiles for task slots: Closed
• Fail job immediately if tasks’ resource needs can not be satisfied.: Closed
• Bookkeeping of available resources of allocated slots in SlotPool.: Closed
• Dynamically allocate TaskExecutor’s managed memory to slots.: Closed
• Memory configuration implementation: Closed
• Launch customized TaskExecutor for tasks whose resource needs cannot be satisfied by existing TaskExecutor-s in active container mode.: Closed