[FLINK-22517] Fix pickle compatibility problem in different Python versions

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.13.0, 1.12.3, 1.14.4, 1.15.0
• Component/s: API / Python
• Assignee: Unassigned
• Reporter: Huang Xingbo
• Created: 29/Apr/21 02:10
• Updated: 07/Apr/22 07:26

Description

Since release-1.12, PyFlink has supported Python3 8. Starting from Python 3.8, the default protocol version used by pickle is pickle5(https://www.python.org/dev/peps/pep-0574/), which will raising the following exception if the client uses python 3.8 to compile program and the cluster node uses python 3.7 or python 3.6 to run python udf:

ValueError: unsupported pickle protocol: 5

The workaround is to first let the python version used by the client be 3.6 or 3.7. For how to specify the client-side python execution environment, please refer to the doc(https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/python_config.html#python-client-executable).