[FLINK-9940] File source continuous monitoring mode: S3 files sometimes missed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Component/s: API / DataStream
• Labels: EMRFS Flink S3 pull-request-available

Environment

Flink 1.5, EMRFS

• Assignee: Huyen Levan
• Reporter: Huyen Levan
• Created: 25/Jul/18 04:20
• Updated: 14/Feb/21 10:17
• Resolved: 04/Feb/21 11:08

Description

When using StreamExecutionEnvironment.readFile() with FileProcessingMode.PROCESS_CONTINUOUSLY mode to monitor an S3 prefix, if there is a high amount of new/modified files at the same time, the directory monitoring process might miss some files. The number of missing files depends on the monitoring interval.

Cause: Flink tracks which files it has read by remembering the modification time of the file that was added (or modified) last. So when there are multiple files having a same last-modified timestamp.

Suggested solution (thanks to [Fabian Hueske): a hybrid approach that keeps the names of all files that have a mod timestamp that is larger than the max mod time minus an offset. org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction

Issue Links

• is duplicated by: FLINK-8046 ContinuousFileMonitoringFunction wrongly ignores files with exact same timestamp
• links to: GitHub Pull Request #6613