[FLINK-34892] Nightly AWS connectors build fails on running python tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: aws-connector-4.2.0
• Component/s: Connectors / AWS
• Labels: pull-request-available
• Assignee: Aleksandr Pilipenko
• Reporter: Aleksandr Pilipenko
• Created: 20/Mar/24 10:04
• Updated: 21/Mar/24 08:55
• Resolved: 21/Mar/24 08:55

Description

Build for externalized python connector code fails: https://github.com/apache/flink-connector-aws/actions/runs/8351768294/job/22860710449

2024-03-20T00:14:35.5215863Z __________________ FlinkKinesisTest.test_kinesis_streams_sink __________________
2024-03-20T00:14:35.5216781Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/testing/test_case_utils.py:149: in setUp
2024-03-20T00:14:35.5217584Z     self.env = StreamExecutionEnvironment.get_execution_environment()
2024-03-20T00:14:35.5218901Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/datastream/stream_execution_environment.py:876: in get_execution_environment
2024-03-20T00:14:35.5219751Z     gateway = get_gateway()
2024-03-20T00:14:35.5220635Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:64: in get_gateway
2024-03-20T00:14:35.5221378Z     _gateway = launch_gateway()
2024-03-20T00:14:35.5222111Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/java_gateway.py:110: in launch_gateway
2024-03-20T00:14:35.5222956Z     p = launch_gateway_server_process(env, args)
2024-03-20T00:14:35.5223854Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:262: in launch_gateway_server_process
2024-03-20T00:14:35.5224649Z     java_executable = find_java_executable()
2024-03-20T00:14:35.5225583Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:75: in find_java_executable
2024-03-20T00:14:35.5226449Z     java_home = read_from_config(KEY_ENV_JAVA_HOME, None, flink_conf_file)
2024-03-20T00:14:35.5227099Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-03-20T00:14:35.5227450Z 
2024-03-20T00:14:35.5227774Z key = 'env.java.home', default_value = None
2024-03-20T00:14:35.5228925Z flink_conf_file = '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
2024-03-20T00:14:35.5229778Z 
2024-03-20T00:14:35.5230010Z     def read_from_config(key, default_value, flink_conf_file):
2024-03-20T00:14:35.5230581Z         value = default_value
2024-03-20T00:14:35.5231236Z         # get the realpath of tainted path value to avoid CWE22 problem that constructs a path or URI
2024-03-20T00:14:35.5232195Z         # using the tainted value and might allow an attacker to access, modify, or test the existence
2024-03-20T00:14:35.5232940Z         # of critical or sensitive files.
2024-03-20T00:14:35.5233417Z >       with open(os.path.realpath(flink_conf_file), "r") as f:
2024-03-20T00:14:35.5234874Z E       FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/flink-connector-aws/flink-connector-aws/flink-python/.tox/py310-cython/lib/python3.10/site-packages/pyflink/conf/flink-conf.yaml'
2024-03-20T00:14:35.5235954Z 
2024-03-20T00:14:35.5236484Z .tox/py310-cython/lib/python3.10/site-packages/pyflink/pyflink_gateway_server.py:58: FileNotFoundError 

Failure started after the release of apache-flink python package for 1.19.0 due to change of default config file provided within artifact.

 

 

Issue comes from outdated copy of pyflink_gateway_server.py created as part of https://github.com/apache/flink-connector-kafka/pull/69 (same change is duplicated in AWS connectors repository).

Issue Links

• links to: GitHub Pull Request #133