[FLINK-12665] Introduce MiniBatchIntervalInferRule and watermark assigner operators

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.9.0
• Component/s: Table SQL / Planner, Table SQL / Runtime
• Labels: pull-request-available
• Assignee: godfrey he
• Reporter: godfrey he
• Created: 29/May/19 06:35
• Updated: 04/Jul/19 13:03
• Resolved: 04/Jul/19 13:03

Description

This issue aims to introduce MiniBatchIntervalInferRule to infer the mini-batch interval of watermark assigner, This rule could handle the following two kinds of operator:
1. supports operators which supports mini-batch and does not require watermark, e.g. group aggregate. In this case, StreamExecWatermarkAssigner with Protime mode will be created if not exist, and the interval value will be set as SQL_EXEC_MINIBATCH_ALLOW_LATENCY.
2. supports operators which requires watermark, e.g. window join, window aggregate. In this case, StreamExecWatermarkAssigner already exists, and its MiniBatchIntervalTrait will be updated as the merged intervals from its outputs.
Currently, mini-batched window aggregate is not supported, and will be introduced later.

this issue also introduces watermark assigner operators, including:
1. WatermarkAssignerOperator, that extracts timestamps from stream elements and generates periodic watermarks.
2. MiniBatchedWatermarkAssignerOperator, that extracts timestamps from stream elements and generates watermarks with specified emit latency.
3. MiniBatchAssignerOperator, that emits mini-batch marker in a given period.

Issue Links

• is a child of: FLINK-11488 Add a basic Blink planner framework
• links to: GitHub Pull Request #8562