[FLINK-2314] Make Streaming File Sources Persistent

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Assignee: Kostas Kloudas
• Reporter: Stephan Ewen
• Created: 02/Jul/15 14:15
• Updated: 28/Feb/19 11:15
• Resolved: 14/Jun/16 17:48

Description

Streaming File sources should participate in the checkpointing. They should track the bytes they read from the file and checkpoint it.

One can look at the sequence generating source function for an example of a checkpointed source.

Issue Links

• is duplicated by: FLINK-3515 Make the "file monitoring source" exactly-once
• relates to: FLINK-3677 FileInputFormat: Allow to specify include/exclude file name patterns
• requires: FLINK-3896 Allow a StreamTask to be Externally Cancelled.
• supercedes: FLINK-3796 FileSourceFunction doesn't respect InputFormat's life cycle methods
• links to: GitHub Pull Request #997
• links to: GitHub Pull Request #2020

Sub-Tasks

• Add functionality to be a able to restore from specific point in a FileInputFormat: Closed
• Refactor the whole file monitoring source to take a fileInputFormat as an argument.: Closed
• Make File Monitoring Function checkpointable.: Closed