[FLINK-2314] Make Streaming File Sources Persistent

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Assignee: Kostas Kloudas
• Reporter: Stephan Ewen
• Created: 02/Jul/15 14:15
• Updated: 28/Feb/19 11:15
• Resolved: 14/Jun/16 17:48

Description

Streaming File sources should participate in the checkpointing. They should track the bytes they read from the file and checkpoint it.

One can look at the sequence generating source function for an example of a checkpointed source.

Issue Links

• is duplicated by: FLINK-3515 Make the "file monitoring source" exactly-once
• relates to: FLINK-3677 FileInputFormat: Allow to specify include/exclude file name patterns
• requires: FLINK-3896 Allow a StreamTask to be Externally Cancelled.
• supercedes: FLINK-3796 FileSourceFunction doesn't respect InputFormat's life cycle methods
• links to: GitHub Pull Request #997
• links to: GitHub Pull Request #2020