[FLINK-18362] Add support for shipping archives

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Implemented
• Affects Version/s: 1.10.1
• Fix Version/s: 1.12.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: gramo lee
• Reporter: Noah
• Created: 18/Jun/20 09:43
• Updated: 13/Apr/21 20:30
• Resolved: 30/Jul/20 10:15

Description

Hello,
Our team are running flink cluster on YARN, and it works so well 👍

Functional requirements

Is there any option to ship archive to YARN applications?

Backgrounds

Recently, one of job has been shut down with jdk8 version related issues.
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699

It's easy problem if we could set latest jdk on `containerized.taskmanager.env.JAVA_HOME`.
However, cluster administrator said it's difficult to install the latest jdk on all cluster machines.

So, we planned to run a job on latest jdk that is shipped via shared resources.
There's an option `yarn.ship-directories` but it's quite slow because jdk has large number of files.

If Flink supports to ship archive such as `yarn.ship-archive`, we can ship jdk archive to remote machines and use shipped jdk location as JAVA_HOME (using `yarn.container-start-command-template` )

Issue Links

• is related to: FLINK-13838 Support shipping archives for Yarn deployment
• links to: GitHub Pull Request #12791