[FLINK-12069] Add proper lifecycle management for intermediate result partitions

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.8.0, 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination, Runtime / Network
• Assignee: Chesnay Schepler
• Reporter: Till Rohrmann
• Created: 29/Mar/19 16:09
• Updated: 02/Jul/19 15:09
• Resolved: 02/Jul/19 15:09

Description

In order to properly execute batch jobs, we should make the lifecycle management of intermediate result partitions the responsibility of the JobMaster/Scheduler component. The Scheduler knows best when an intermediate result partition is no longer needed and, thus, can be freed. So for example, a blocking intermediate result should only be released after all subsequent blocking intermediate results have been completed in order to speed up potential failovers.

Moreover, having explicit control over intermediate result partitions, could also enable use cases like result partition sharing between jobs and even across clusters (by simply not releasing the result partitions).

Issue Links

• depends upon: FLINK-12890 Add partition lifecycle related Shuffle API
• is related to: FLINK-10941 Slots prematurely released which still contain unconsumed data
• is related to: FLINK-12070 Make blocking result partitions consumable multiple times
• is related to: FLINK-4256 Fine-grained recovery
• links to: Design document