[FLINK-2674] Rework windowing logic

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 1.0.0
• Component/s: API / DataStream
• Assignee: Unassigned
• Reporter: Stephan Ewen
• Created: 15/Sep/15 12:08
• Updated: 28/Feb/19 11:15
• Resolved: 04/Apr/16 14:53

Description

The windowing logic needs a major overhaul. This follows the design documents:

• https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=60624830

Specifically, the following shortcomings need to be addressed:

• Global parallel windows should be dropped
  -> for time, local windows are aligned and serve the same purpose
  -> there is currently no known robust and efficient parallel implementation of custom strategies

• Event time and out of order arrival needs to be supported

• Eviction of not accessed keys does not work. Non-accessed keys linger infinitely

• Performance is currently bad for time windows, due to a overly general implementation

• Resources are leaking, threads are not shut down

• Elements are stored multiple times (discretizers, window buffers)

• Finally, many implementations are buggy, produce wrong results

Sub-Tasks

• Add utilities for scheduled triggers: Closed
• Add abstraction for keyed window state: Closed
• Add a general-purpose keyed-window operator: Resolved
• Create a dedicated aligned-event time window operator: Closed
• Add a dedicated processing time aligned keyed window operator: Closed
• Add utilities for heap-backed keyed state in time panes: Closed
• Add new window API to streaming API: Resolved
• Add API for non-parallel non-keyed Windows: Closed
• Update documentation to reflect new Stream/Window API: Closed
• Remove Old Windowing Logic and API: Closed
• Add javadocs/comments to new windowing mechanics: Closed
• Add Windowed Join/CoGroup Operator Based on Tagged Union: Closed
• Rework DataStream API: Closed
• Add Fold Window Operation for new Windowing API: Closed
• Make State of General-Purpose Window Operators Fault-Tolerant: Closed
• Add support for accumulating/discarding for Event-Time Windows: Closed
• Key for Keyed State is not set upon Window Evaluation: Closed