[FLINK-36464] Local artifact fetcher should not require base directory

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0, 1.20.0
• Component/s: Deployment / Kubernetes
• Release Note: Duplicate of https://issues.apache.org/jira/browse/FLINK-35833
• Assignee: Unassigned
• Reporter: Fabian Paul
• Created: 10/Oct/24 07:52
• Updated: 10/Oct/24 08:02
• Resolved: 10/Oct/24 08:02

Description

Currently Flink tries to fetch all artifacts referenced by the job into a central base directory. https://github.com/apache/flink/blob/6e7bf9b6f939cd1d183e416c0187ad24270e0bc1/flink-clients/src/main/java/org/apache/flink/client/program/artifact/ArtifactFetchManager.java#L76 

In case, users only rely on artifacts that are resolvable locally this base directory is ignored. https://github.com/apache/flink/blob/6e7bf9b6f939cd1d183e416c0187ad24270e0bc1/flink-clients/src/main/java/org/apache/flink/client/program/artifact/LocalArtifactFetcher.java#L35 

 

Unfortunately, Flink always tries to provision the base directory leading to strange errors in case Flink has no write privileges on the base directory path. https://github.com/apache/flink/blob/6e7bf9b6f939cd1d183e416c0187ad24270e0bc1/flink-clients/src/main/java/org/apache/flink/client/program/artifact/ArtifactFetchManager.java#L123 

Since the base directory is not needed for local execution/local artifact fetcher, we should allow the creation of the base directory to be disabled.

Issue Links

• is duplicated by: FLINK-35833 ArtifactFetchManager always creates artifact dir