[FLINK-34641] Possibility to add Python Files from http/https sources in PythonDriver

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.18.0, 1.18.1, 1.18.2
• Component/s: API / Python

Language

Java

• Assignee: Unassigned
• Reporter: Aleksei Perminov
• Created: 11/Mar/24 11:43
• Updated: 11/Mar/24 15:21

Description

Problem: Now it is impossible to add python files from the external http sources, only fs sources are available. It is not very convenient to add Python packages to the Flink Docker image or external fs every time the new version of the Python job is built, especially when using the Session mode. 

Proposal: Add possibility to handle the http/https protocols in the PythonDriver so that it is possible to add packages from the external Python repositories (e.g. PyPi). 

Example of the solution in the PythonEnvUtils.java

private static void addToPythonPath(PythonEnvironment env, List<Path> pythonFiles) {
...
 if (isValidURL(pythonFile.toString()) && new URL(pythonFile.toString()).getProtocol().startsWith("http")) {
    getFileFromHTTP(pythonFile, targetPath);
} else if (!pythonFile.getFileSystem().isDistributedFS()) {
...
}