[SPARK-12650] No means to specify Xmx settings for spark-submit in cluster deploy mode for Spark on YARN

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.5.2
• Component/s: Spark Submit
• Labels: bulk-closed

Environment

Hadoop 2.6.0

• Assignee: Unassigned
• Reporter: John Vines
• Created: 05/Jan/16 16:03
• Updated: 21/May/19 04:14
• Resolved: 21/May/19 04:14

Description

Background-
I have an app master designed to do some work and then launch a spark job.

Issue-
If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all, leading to the jvm taking a default heap which is relatively large. This causes a large amount of vmem to be taken, so that it is killed by yarn. This can be worked around by disabling Yarn's vmem check, but that is a hack.

If I run it in yarn-client mode, it's fine as long as my container has enough space for the driver, which is manageable. But I feel that the utter lack of Xmx settings for what I believe is a very small jvm is a problem.

I believe this was introduced with the fix for SPARK-3884

Issue Links

• relates to: SPARK-14453 Remove SPARK_JAVA_OPTS environment variable