[FLINK-986] Add intermediate results to distributed runtime

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Runtime / Coordination
• Assignee: Ufuk Celebi
• Reporter: Ufuk Celebi
• Created: 26/Jun/14 10:20
• Updated: 13/Feb/17 13:25
• Resolved: 12/Jan/15 08:18

Description

Support for intermediate results in the runtime is currently blocking different efforts like fault tolerance or result collection at the client.

Issue Links

• is blocked by: FLINK-967 Make intermediate results a first-class citizen in the JobGraph
• is related to: FLINK-1350 Add blocking intermediate result partitions
• is related to: FLINK-1359 Make IntermediateResultPartitionManager aware of consumed partitions
• is related to: FLINK-1373 Add documentation for intermediate results and network stack to internals
• relates to: FLINK-1730 Add a FlinkTools.persist style method to the Data Set.
• relates to: FLINK-2250 Backtracking of intermediate results
• supercedes: FLINK-25 [GitHub] Offer a buffer-oriented abstraction from Nephele to Pact
• supercedes: FLINK-478 Deep support for reading outputs outside a plan
• supercedes: FLINK-982 Convert DataSet back into Collection
• supercedes: FLINK-1093 Add non-multiplexing NetworkConnectionManager
• supercedes: FLINK-1222 Let tasks finish as soon as their data is sent