[FLINK-20681] Support specifying the hdfs path when ship archives or files

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.19.0
• Component/s: Deployment / YARN
• Labels: auto-deprioritized-major auto-unassigned pull-request-available pull-requests-available stale-assigned
• Assignee: junzhong qin
• Reporter: Ruguo Yu
• Created: 19/Dec/20 15:58
• Updated: 30/Aug/23 14:54
• Resolved: 30/Aug/23 14:54

Description

Currently, our team try to submit flink job that depends extra resource with yarn-application target, and use two options: "yarn.ship-archives" and "yarn.ship-files".

But above options only support specifying local resource and shiping them to hdfs, besides if it can support remote resource on distributed filesystem (such as hdfs), then get the following benefits：

• client will exclude the local resource uploading to accelerate the job submission process
• yarn will cache them on the nodes so that they doesn't need to be downloaded for application

Issue Links

• relates to: FLINK-20811 Support HTTP paths for yarn ship files/archives
• links to: GitHub Pull Request #14538
• links to: GitHub Pull Request #23219
• mentioned in: Page Loading...