[FLINK-5564] Add User Defined Aggregates in Table API

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Implemented
• Fix Version/s: 1.3.0, 1.4.0
• Component/s: Table SQL / API
• Assignee: Shaoxuan Wang
• Reporter: Shaoxuan Wang
• Created: 18/Jan/17 19:47
• Updated: 12/Oct/17 12:31
• Resolved: 12/Oct/17 12:31

Description

User-defined aggregates would be a great addition to the Table API / SQL.
The current aggregate interface is not well suited for the external users. This issue proposes to redesign the aggregate such that we can expose an better external UDAGG interface to the users. The detailed design proposal can be found here: https://docs.google.com/document/d/19JXK8jLIi8IqV9yf7hOs_Oz67yXOypY7Uh5gIOK2r-U/edit

Motivation:
1. The current aggregate interface is not very concise to the users. One needs to know the design details of the intermediate Row buffer before implements an Aggregate. Seven functions are needed even for a simple Count aggregate.
2. Another limitation of current aggregate function is that it can only be applied on one single column. There are many scenarios which require the aggregate function taking multiple columns as the inputs.
3. “Retraction” is not considered and covered in the current Aggregate.
4. It might be very good to have a local/global aggregate query plan optimization, which is very promising to optimize UDAGG performance in some scenarios.

Proposed Changes:
1. Implement an aggregate dataStream API (Done by FLINK-5582)
2. Update all the existing aggregates to use the new aggregate dataStream API
3. Provide a better User-Defined Aggregate interface
4. Add retraction support
5. Add local/global aggregate

Issue Links

• requires: FLINK-5582 Add a general distributive aggregate function

Sub-Tasks

• New aggregate function interface and built-in aggregate functions: Closed
• Apply new aggregation functions for datastream and dataset tables: Closed
• Apply new aggregation functions for dataset tables: Closed
• code generation for user-defined aggregate functions: Resolved
• Add user-defined aggregation functions to documentation.: Closed
• Add support to register UDAGG in Table and SQL API: Closed
• Add support for the aggregate on multi fields: Resolved
• Remove old Aggregate interface and built-in functions: Closed
• Add retract method into the aggregateFunction: Closed
• Remove `getAccumulatorType` method from build-in `AggregateFunction`: Closed
• Remove preparation mapper of DataSetAggregate: Resolved
• codeGen dataStream aggregates that use AggregateAggFunction: Closed
• codeGen dataStream aggregates that use ProcessFunction: Closed
• codeGen DataSet Goupingwindow Aggregates: Closed
• Finalize the AggregateFunction interface and refactoring built-in aggregates: Closed