[FLINK-20611] Support 2 phase local aggregate for window aggregate operation

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Table SQL / API, Table SQL / Planner
• Labels: stale-major
• Assignee: Unassigned
• Reporter: Suhan Mao
• Created: 15/Dec/20 09:10
• Updated: 29/Apr/21 14:52
• Resolved: 29/Apr/21 14:52

Description

In our company, many jobs run with window aggregate and suffer from large amounts of intermediate shuffle. If we can enable 2 phase local agg on window aggregate, the performance will be improved.

This issue aims to support 2 phase local agg in window aggregate operator both in SQL and Table API. Simple tumble window will be supported in the first place.

 

We still use below configurations, but they take effect only inside each single window.

table.exec.mini-batch.enabled

table.exec.mini-batch.allow-latency

table.exec.mini-batch.size

 

Issue Links

• is related to: FLINK-19604 FLIP-145: Support SQL windowing table-valued function