[FLINK-1846] Sinks inside of iterations

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Won't Do
• Component/s: API / DataSet, Runtime / Coordination, Runtime / Task
• Assignee: Markus Holzemer
• Reporter: Markus Holzemer
• Created: 08/Apr/15 12:43
• Updated: 28/Feb/19 14:30
• Resolved: 26/Feb/19 15:49

Description

I startet work on adding support for file sinks inside of iterations to flink.
My current plan includes two different modes: one where only the output file of the last iteration is kept and one where the output file of every single iteration is kept (the corresponding superstep number is added as a prefix).

I am also thinking about adding a mode where the output of every iteration is appended to one file or where a "sliding window" of n files is kept for all iterations.

Issue Links

• requires: FLINK-951 Reworking of Iteration Synchronization, Accumulators and Aggregators