[FLINK-4329] Fix Streaming File Source Timestamps/Watermarks Handling

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.3, 1.2.0
• Component/s: Connectors / Common
• Assignee: Kostas Kloudas
• Reporter: Aljoscha Krettek
• Created: 08/Aug/16 12:00
• Updated: 05/Oct/16 22:24
• Resolved: 05/Oct/16 22:24

Description

The ContinuousFileReaderOperator does not correctly deal with watermarks, i.e. they are just passed through. This means that when the ContinuousFileMonitoringFunction closes and emits a Long.MAX_VALUE that watermark can "overtake" the records that are to be emitted in the ContinuousFileReaderOperator. Together with the new "allowed lateness" setting in window operator this can lead to elements being dropped as late.

Also, ContinuousFileReaderOperator does not correctly assign ingestion timestamps since it is not technically a source but looks like one to the user.

Issue Links

• links to: GitHub Pull Request #2350
• links to: GitHub Pull Request #2546
• links to: GitHub Pull Request #2593