[FLINK-21222] Support loopback mode to allow Python UDF worker and client reuse the same Python VM

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.14.0
• Component/s: API / Python
• Labels: auto-unassigned pull-request-available
• Release Note: Instead of launching a separate Python process, it will reuse the Python process of the client side to execute Python UDFs if you are running jobs locally. This makes it easy for Python UDFs debugging.
• Assignee: Huang Xingbo
• Reporter: Dian Fu
• Created: 01/Feb/21 07:08
• Updated: 13/Sep/21 11:47
• Resolved: 16/Aug/21 11:15

Description

Currently the Python process for the client and the Python UDF worker runs in different Python processes. It would bring the following benefits if we could support loopback mode to allow Python UDF worker and client reuse the same Python VM:
1) It could greatly reduce the job initialization overhead in mini-cluster mode as there is no need to start the Python process any more.
2) It makes the debug easier.

Issue Links

• links to: GitHub Pull Request #16732