[FLINK-3435] Change interplay of Ingestion Time and Event Time

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10.2
• Fix Version/s: 1.0.0
• Component/s: API / DataStream
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 17/Feb/16 15:55
• Updated: 28/Feb/19 11:15
• Resolved: 24/Feb/16 21:02

Description

Currently, "EventTime" and "IngestionTime" are completely the same.

For both happens the following:

• Sources generate ingestion time timestamps and watermarks
• If a user adds a manual timestamp extractor / watermark generator, then those override the ingestion time timestamps and watermarks
• That implies that event time on a certain input falls back to ingestion time, if one forgets (or incorrectly uses) the timestamp extractors
• Also, Ingestion Time and Event Time simply mix if some inputs have timestamp assigners, and others have not.

This behavior is quite tricky to understand. After some discussions with aljoscha and rmetzger, we suggest to change it the following way.

1. On Ingestion Time, the timestamps and watermarks are generated as they are now.

2. On event time, no default timestamps and watermarks are generated. If a user does not implement a timestamp extractor / watermark generator, then the event time operations will fail fast.

3. If one wants to use ingestion time on event time settings (mix), one can use an explicit "WallClockTimetampsAndWatermark" generator.

4. Later, the "Ingestion Time" settings should automatically disable and user-defined timestamp extractors / assigners.

Sub-Tasks

• Change Timestamp Extractors to not support negative timestamps: Resolved
• EventTime WindowAssigners should error on negative timestamps: Resolved