[FLINK-8046] ContinuousFileMonitoringFunction wrongly ignores files with exact same timestamp

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.2
• Component/s: API / DataStream
• Labels: pull-request-available stream
• Assignee: Unassigned
• Reporter: Dr. Juan Miguel Cejuela
• Created: 10/Nov/17 17:35
• Updated: 24/Aug/20 08:49
• Resolved: 24/Aug/20 08:49

Description

The current monitoring of files sets the internal variable `globalModificationTime` to filter out files that are "older". However, the current test (to check "older") does

`boolean shouldIgnore = modificationTime <= globalModificationTime;` (rom `shouldIgnore`)

The comparison should strictly be SMALLER (NOT smaller or equal). The method documentation also states "This happens if the modification time of the file is smaller than...".

The equality acceptance for "older", makes some files with same exact timestamp to be ignored. The behavior is also non-deterministic, as the first file to be accepted ("first" being pretty much random) makes the rest of files with same exact timestamp to be ignored.

Issue Links

• duplicates: FLINK-9940 File source continuous monitoring mode: S3 files sometimes missed
• links to: GitHub Pull Request #4997
• links to: GitHub Pull Request #4997