[OOZIE-3472] Improve Spark Action compatibility with Oozie launcher

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.1.0
• Component/s: action
• Assignee: Junfan Zhang
• Reporter: Junfan Zhang
• Created: 15/Apr/19 10:17
• Updated: 28/Apr/19 02:38

Description

In the production environment, when using the spark action, our users often encounter conflicts between the user jar and the launcher, causing the launcher to fail to start.
To do this we have a maven plugin to guide the user to remove Hadoop related dependencies from the user jar. But the user jar is more complicated and sometimes not easy to remove. Therefore, it is appropriate to solve this problem from the oozie side.

We research code found that the spark action is inherited to the Java action. The reason for the conflict is because the Java action will put the user jar into the cache before the mr starts (related link). If there is a Hadoop dependency in the user jar and the version is incompatible, a conflict will occur.

From the root cause analysis, the spark action just uses the map node in mr as a spark submit client, and does not need to add the user jar to the mr distributed cache. We solved this conflict by using spark submit sdk to load the user jar from HDFS directly. It currently works well in our production environment.