[FLINK-12069] Add proper lifecycle management for intermediate result partitions

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.8.0, 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination, Runtime / Network
• Assignee: Chesnay Schepler
• Reporter: Till Rohrmann
• Created: 29/Mar/19 16:09
• Updated: 02/Jul/19 15:09
• Resolved: 02/Jul/19 15:09

Description

In order to properly execute batch jobs, we should make the lifecycle management of intermediate result partitions the responsibility of the JobMaster/Scheduler component. The Scheduler knows best when an intermediate result partition is no longer needed and, thus, can be freed. So for example, a blocking intermediate result should only be released after all subsequent blocking intermediate results have been completed in order to speed up potential failovers.

Moreover, having explicit control over intermediate result partitions, could also enable use cases like result partition sharing between jobs and even across clusters (by simply not releasing the result partitions).

Issue Links

• depends upon: FLINK-12890 Add partition lifecycle related Shuffle API
• is related to: FLINK-10941 Slots prematurely released which still contain unconsumed data
• is related to: FLINK-12070 Make blocking result partitions consumable multiple times
• is related to: FLINK-4256 Fine-grained recovery
• links to: Design document

Sub-Tasks

• Track stored partitions on the TM: Closed
• Track allocated partitions on the JM: Closed
• Maintain connection between JM<->TM until all partitions have been released: Closed
• Release all partitions on job termination: Closed
• Make partition release on consumption optional: Closed
• Update RM idleness check for TMs to take partitions into account: Closed
• Add JobID argument to TaskExecutorGateway#releasePartitions: Closed
• Fix invalid check released state during ResultPartition#createSubpartitionView: Closed
• Refactor the pin logic in ReleaseOnConsumptionResultPartition: Closed
• Add elaborate partition release logic: Closed
• Release partitions for reset vertices: Closed
• Rework forceReleaseOnConsumptionFlag to be JM internal concept: Closed