[FLINK-32034] Python's DistUtils is deprecated as of 3.10

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.18.0, 1.17.2
• Component/s: API / Python
• Labels: pull-request-available

Environment

Kubernetes

Java 11

Python 3.10.9

• Flags: Patch

Language

Java

• Assignee: Colten Pilgreen
• Reporter: Colten Pilgreen
• Created: 08/May/23 14:23
• Updated: 15/Jun/23 12:16
• Resolved: 15/Jun/23 12:16

Description

I have recent just went through an upgrade from 1.13 to 1.17, along with that I upgraded the python version on our Flink Session server. Most everything that is part of our workflow works, except for Python Dependency Management.

After doing some digging, I found the reason is due to the DeprecationWarning that is printed when trying to get the site packages path. The script is 

GET_SITE_PACKAGES_PATH_SCRIPT and it is executed in the getSitePackagesPath method in the PythonEnvironmentManagerUtils class. The issue is that the DeprecationWarning is included into the PYTHONPATH environment variable which is passed to the beam runner. The deprecation warning breaks Python's ability to find the site packages due to characters that are not allowed in filesystem paths.

 

Example of the PYTHONPATH environment variable:
PYTHONPATH == <string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives:/tmp/python-dist-c63e1464-925c-4289-bb71-c6f50e83186f/python-requirements/lib/python3.10/site-packages
HADOOP_CONF_DIR == /opt/flink/conf

Issue Links

• links to: GitHub Pull Request #22782