[FLINK-24682] Unify the -C option behavior in both yarn application and per-job mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 1.12.3
• Component/s: Deployment / YARN

Environment

flink 1.12.3

yarn 2.8.5

• Assignee: Unassigned
• Reporter: Biao Geng
• Created: 28/Oct/21 11:52
• Updated: 10/Feb/22 05:07
• Resolved: 10/Feb/22 05:07

Description

Recently, when switching the job submission mode from per-job mode to application mode on yarn, we found the behavior of '-C' ('–-classpath') is somehow misleading:
In per-job mode, the `main()` method of the program is executed in the local machine and '-C' option works well when we use it to specify some local user jars like -C file://xxxxxx.jar.
But in application mode, this option works differently: as the `main()` method will be executed on the job manager in the cluster, it is unclear where the url like `file://xxxxxx.jar` points. It seems that `file://xxxxxx.jar` is located on the job manager machine in the cluster due to the code. If that is true, it may mislead users as in per-job mode, it refers to the the jars in the client machine. 
In summary, if we can unify the -C option behavior in both yarn application and per-job mode, it would help users to switch to application mode more smoothly and more importantly, it makes it much easier to specify some local jars, that should be loaded by UserClassLoader, on the client machine.