[SPARK-38960] Spark should fail fast if initial memory too large(set by "spark.executor.extraJavaOptions") for executor to start

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.4.0
• Component/s: Spark Core, Spark Submit, YARN
• Assignee: Unassigned
• Reporter: BingKun Pan
• Created: 20/Apr/22 01:43
• Updated: 14/Jun/22 00:33
• Resolved: 14/Jun/22 00:33

Description

if you set initial memory(set by "spark.executor.extraJavaOptions=-Xms{XXX}G" ) larger than maximum memory(set by "spark.executor.memory")

Eg.

     spark.executor.memory=1G

     spark.executor.extraJavaOptions=-Xms2G

 

from the driver process you just see executor failures with no warning, since the more meaningful errors are buried in the executor logs. 

Eg., on Yarn, you see:

Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

Instead we should just fail fast with a clear error message in the driver logs.

Issue Links

• links to: [Github] Pull Request #36273 (panbingkun)