[FLINK-32775] Support yarn.provided.lib.dirs to add parent directory to classpath

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.19.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Archit Goyal
• Reporter: Archit Goyal
• Created: 07/Aug/23 22:52
• Updated: 14/Sep/23 01:52
• Resolved: 14/Sep/23 01:52

Description

Currently with yarn.provided.lib.dirs, Flink libs can be copied to HDFS location in each cluster and when set Flink tries to reuse the same jars avoiding uploading it every time and YARN also caches it in the nodes.

 

This works fine with jars but if we try to add the xml file parent directory to path, Flink job submission fails. If I add the parent directory of the xml to the 

yarn.ship-files

 Flink job is submitted successfully.

Issue Links

• links to: https://github.com/apache/flink/pull/23164
• mentioned in: Page Loading...