[FLINK-16430] FLIP-119 Pipelined Region Scheduling

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Runtime / Coordination

Release Note

Beginning from Flink 1.12, jobs will be scheduled in the unit of pipelined regions. A pipelined region is a set of pipelined connected tasks. This means that, for streaming jobs which consist of multiple regions, it no longer waits for all tasks to acquire slots before starting to deploy tasks. Instead, any region can be deployed once it has acquired enough slots for within tasks. For batch jobs, tasks will not be assigned slots and get deployed individually. Instead, a task will be deployed together with all other tasks in the same region, once the region has acquired enough slots.

• Assignee: Zhu Zhu
• Reporter: Zhu Zhu
• Created: 05/Mar/20 01:48
• Updated: 18/Mar/21 10:07
• Resolved: 08/Oct/20 17:39

Description

Pipelined region scheduling is targeting to allow batch jobs with PIPELINED data exchanges to run without the risk to encounter a resource deadlock.

More details see FLIP-119

Issue Links

• is a parent of: FLINK-18689 Deterministic Slot Sharing
• is blocked by: FLINK-14162 Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies
• is related to: FLINK-19142 Local recovery can be broken if slot hijacking happened during a full restart
• is related to: FLINK-19923 Remove unused BulkSlotProvider, its implementation and tests
• relates to: FLINK-10643 Bubble execution: Resource aware job execution
• links to: FLIP-119
• mentioned in: Page Loading...
• mentioned in: Page Loading...