[FLINK-32136] Pyflink gateway server launch fails when purelib != platlib

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.3
• Fix Version/s: 1.18.0, 1.16.3, 1.17.2
• Component/s: API / Python
• Labels: pull-request-available
• Assignee: Dian Fu
• Reporter: William Ashley
• Created: 19/May/23 17:34
• Updated: 17/Jun/23 10:03
• Resolved: 17/Jun/23 10:03

Description

On distros where python's purelib is different than platlib (e.g. Amazon Linux 2, but from my research it's all of the Redhat-based ones), you wind up with components of packages being installed across two different locations (e.g. /usr/local/lib/python3.7/site-packages/pyflink and /usr/local/lib64/python3.7/site-packages/pyflink).

_find_flink_home handles this, and in flink releases <= 1.13.2 its setting of the FLINK_LIB_DIR environment variable was the one being used. However, from 1.13.3, a refactoring of launch_gateway_server_process (1.13.2, 1.13.3) re-ordered some method calls. prepare_environment_variable's non-awareness of multiple homes and setting of FLINK_LIB_DIR now is the one that matters, and it is the incorrect location.

I've confirmed this problem on Amazon Linux 2 and 2023. The problem does not exist on, for example, Ubuntu 20 and 22 (for which platlib == purelib).

Repro steps on Amazon Linux 2

yum -y install python3 java-11
pip3 install apache-flink==1.13.3
python3 -c 'from pyflink.table import EnvironmentSettings ; EnvironmentSettings.new_instance()'

The resulting error is

The flink-python jar is not found in the opt folder of the FLINK_HOME: /usr/local/lib64/python3.7/site-packages/pyflink
Error: Could not find or load main class org.apache.flink.client.python.PythonGatewayServer
Caused by: java.lang.ClassNotFoundException: org.apache.flink.client.python.PythonGatewayServer
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py", line 214, in new_instance
    return EnvironmentSettings.Builder()
  File "/usr/local/lib64/python3.7/site-packages/pyflink/table/environment_settings.py", line 48, in {}init{}
    gateway = get_gateway()
  File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py", line 62, in get_gateway
    _gateway = launch_gateway()
  File "/usr/local/lib64/python3.7/site-packages/pyflink/java_gateway.py", line 112, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

The flink home under /lib64/ does not contain the jar, but it is in the /lib/ location

bash-4.2# find /usr/local/lib64/python3.7/site-packages/pyflink -name "flink-python*.jar"
bash-4.2# find /usr/local/lib/python3.7/site-packages/pyflink -name "flink-python*.jar"
/usr/local/lib/python3.7/site-packages/pyflink/opt/flink-python_2.11-1.13.3.jar

 

Issue Links

• links to: GitHub Pull Request #22802