[FLINK-19169] Support Pandas UDAF in PyFlink (FLIP-137)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.12.0
• Component/s: API / Python
• Assignee: Huang Xingbo
• Reporter: Huang Xingbo
• Created: 09/Sep/20 07:10
• Updated: 14/Jan/21 09:20
• Resolved: 03/Nov/20 11:29

Description

Pandas UDF has been supported in FLINK 1.11 (FLIP-97). It solves the high serialization/deserialization overhead in Python UDF and makes it convenient to leverage the popular Python libraries such as Pandas, Numpy, etc. Since Pandas UDF has so many advantages, we want to support Pandas UDAF to extend usage of Pandas UDF.

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• Add Pandas Batch Group Aggregation Function Operator: Closed
• Add Batch Physical Pandas Group Aggregate Rule and RelNode: Closed
• Add Python building blocks to make sure the basic functionality of Pandas Batch Group Aggregation could work: Closed
• Introduce BatchArrowPythonGroupWindowAggregateFunctionOperator: Closed
• Introduce BatchArrowPythonOverWindowAggregateFunctionOperator: Closed
• Add Batch Physical Pandas Group Window Aggregate Rule and RelNode: Closed
• Support Pandas Batch Over Window Aggregation: Closed
• Support Pandas Stream Group Window Aggregation: Closed
• Support Pandas Stream Over Window Aggregation: Closed
• Add Pandas UDAF Doc: Closed