[FLINK-1618] Add parallel time discretisation for time-window transformations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Component/s: API / DataStream
• Assignee: Gyula Fora
• Reporter: Gyula Fora
• Created: 02/Mar/15 09:00
• Updated: 28/Feb/19 11:15
• Resolved: 20/Mar/15 12:45

Description

Currently discretizers for all windowing policies including time are executed with parallelism 1 when they define global windows. (for instance: sum of the last 10 minutes)

While this is necessary for arbitrary policies like delta based or user-defined policies. Some discretizers such as Time can be implemented in a distributed fashion.

Distributed time discretisers (and other types) can be implemented in the following way:

-The discretisers should create StreamWindow s with incrementally increasing ID-s starting from the same value so that it is possible to merge them after the transformation

• The partitioner for each discretizer should send the number of partitions created to the merger (the merger should be aware of the number of partitioners present to wait for all the information)
• Based on all the partitioning info the merger can merge the windows properly afterwards