[FLINK-25718] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.15.0
• Component/s: API / Python
• Assignee: Huang Xingbo
• Reporter: Huang Xingbo
• Created: 20/Jan/22 09:03
• Updated: 08/Aug/22 08:09

Description

We have provided PyFlink Runtime framework to support Python user-defined functions since Flink 1.10. The PyFlink Runtime framework is called Process Mode, which depends on an inter-process communication architecture based on the Apache Beam Portability framework. Although starting a dedicated process to execute Python user-defined functions could have better resource isolation, it will bring greater resource and performance overhead.

In order to overcome the resource and performance problems on Process Mode, we will propose a new execution mode which executes Python user-defined functions in the same thread instead of a separate process.

Sub-Tasks

• Support Pandas UDF in Thread Mode: Open
• Support Python UDAF in Thread Mode: Open
• Support Pandas UDAF in Thread Mode: Open