[FLINK-35208] Respect pipeline.cached-files during processing Python dependencies

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.20.0
• Component/s: API / Python
• Labels: pull-request-available
• Assignee: Dian Fu
• Reporter: Dian Fu
• Created: 22/Apr/24 09:22
• Updated: 28/Apr/24 07:40
• Resolved: 28/Apr/24 07:40

Description

Currently, PyFlink will make use of distributed cache (StreamExecutionEnvironment#cachedFiles) during handling the Python dependencies(See https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/python/util/PythonDependencyUtils.java#L339 for more details). 

However, if pipeline.cached-files is configured, it will clear StreamExecutionEnvironment#cachedFiles(see https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L1132 for more details) which may break the above functionalities.

Issue Links

• links to: GitHub Pull Request #24716