[FLINK-2250] Backtracking of intermediate results

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Component/s: API / DataSet, Runtime / Coordination
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 19/Jun/15 12:31
• Updated: 26/Feb/19 15:56
• Resolved: 26/Feb/19 15:56

Description

With intermediate results available in the distributed runtime as of FLINK-986, we could now incrementally resume failed jobs if we cached the results FLINK-1404. Moreover, Flink users could build incremental Flink jobs using count/collect/print and, ultimately, also continue from old job results in an interactive shell environment like the scala-shell.

The following tasks need to be completed for that to happen:

• Cache the results
• Keep the ExecutionGraph in the JobManager
• Change the scheduling mechanism to track back the results from the sinks
• Implement a session management to eventually discard old results and ExecutionGraphs from the TaskManagers/JobManager

Issue Links

• is duplicated by: FLINK-6110 Flink unnecessarily repeats shared work triggered by different blocking sinks, leading to massive inefficiency
• is related to: FLINK-986 Add intermediate results to distributed runtime
• is superceded by: FLINK-10429 Redesign Flink Scheduling, introducing dedicated Scheduler component
• relates to: FLINK-2245 Programs that contain collect() reported as multiple jobs in the Web frontend
• relates to: FLINK-1730 Add a FlinkTools.persist style method to the Data Set.