[FLINK-22605] Python UDF should only be created once regardless of how many times it is invoked

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: API / Python
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: Yik San Chan
• Created: 08/May/21 06:40
• Updated: 11/Mar/24 12:44

Description

Follow up to http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/I-call-Pandas-UDF-N-times-do-I-have-to-initiate-the-UDF-N-times-tt43576.html.

Currently, when we call Python UDF N times, the Python UDF is constructed N times. This may become a concern to performance, when we want to load large resources in the open() method of the UDF, which is quite common in machine learning use cases.

I propose we optimize in PyFlink framework level s.t. no matter how many times a UDF is called in the execution environment, it is only initiated once.