Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Implemented
-
1.10.1
Description
Hello,
Our team are running flink cluster on YARN, and it works so well 👍
Functional requirements
Is there any option to ship archive to YARN applications?
Backgrounds
Recently, one of job has been shut down with jdk8 version related issues.
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699
It's easy problem if we could set latest jdk on `containerized.taskmanager.env.JAVA_HOME`.
However, cluster administrator said it's difficult to install the latest jdk on all cluster machines.
So, we planned to run a job on latest jdk that is shipped via shared resources.
There's an option `yarn.ship-directories` but it's quite slow because jdk has large number of files.
If Flink supports to ship archive such as `yarn.ship-archive`, we can ship jdk archive to remote machines and use shipped jdk location as JAVA_HOME (using `yarn.container-start-command-template` )
Attachments
Attachments
Issue Links
- is related to
-
FLINK-13838 Support shipping archives for Yarn deployment
- Closed
- links to