[FLINK-24897] Make usrlib work for YARN application and per-job

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Biao Geng
• Reporter: Biao Geng
• Created: 15/Nov/21 09:15
• Updated: 11/Feb/22 06:01
• Resolved: 11/Feb/22 06:01

Description

Hi there, 
I am working to utilize application mode to submit flink jobs to YARN cluster but I find that currently there is no easy way to ship my user-defined jars(e.g. some custom connectors or udf jars that would be shared by some jobs) and ask the FlinkUserCodeClassLoader to load classes in these jars. 

I checked some relevant jiras, like  FLINK-21289. In k8s mode, there is a solution that users can use `usrlib` directory to store their user-defined jars and these jars would be loaded by FlinkUserCodeClassLoader when the job is executed on JM/TM.

But on YARN mode, `usrlib` does not work as that:

In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if I want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my local machine) to remote cluster, I must not set  UserJarInclusion to DISABLED due to the checkArgument(). However, if I do not set that option to DISABLED, the user jars to be shipped will be added into systemClassPaths. As a result, classes in those user jars will be loaded by AppClassLoader. 

But if I do not ship these jars, there is no convenient way to utilize these jars in my flink run command. Currently, all I can do seems to use `-C` option, which means I have to upload my jars to some shared store first and then use these remote paths. It is not so perfect as we have already make it possible to ship jars or files directly and we also introduce `usrlib` in application mode on YARN. It would be more user-friendly if we can allow shipping `usrlib` from local to remote cluster while using FlinkUserCodeClassLoader to load classes in the jars in `usrlib`.

 

Issue Links

• blocks: FLINK-26000 Drop Per-Job Mode
• relates to: FLINK-26047 Support usrlib in HDFS for YARN application mode
• links to: GitHub Pull Request #18531