[SPARK-25073] Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error request to adjust yarn.scheduler.maximum-allocation-mb

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.3.1
• Fix Version/s: 2.4.0
• Component/s: Spark Submit
• Assignee: Sujith Chacko
• Reporter: vivek kumar
• Created: 09/Aug/18 12:17
• Updated: 24/Aug/18 13:58
• Resolved: 24/Aug/18 13:58

Description

When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error request to adjust Yarn.scheduler.maximum-allocation-mb. Expecting the error request to be  more around yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

 

Scenario 1. yarn.scheduler.maximum-allocation-mb =4g and yarn.nodemanager.resource.memory-mb =8G

1. Launch shell on Yarn with am.memory less than nodemanager.resource memory but greater than yarn.scheduler.maximum-allocation-mb

eg; spark-shell --master yarn --conf spark.yarn.am.memory 5g

 Error: java.lang.IllegalArgumentException: Required AM memory (5120+512 MB) is above the max threshold (4096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.

at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)

 

Scenario 2. yarn.scheduler.maximum-allocation-mb =15g and yarn.nodemanager.resource.memory-mb =8g

a. Launch shell on Yarn with am.memory greater than nodemanager.resource memory but less than yarn.scheduler.maximum-allocation-mb

eg; spark-shell --master yarn --conf spark.yarn.am.memory=10g

 Error :

java.lang.IllegalArgumentException: Required AM memory (10240+1024 MB) is above the max threshold (8096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.

at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)

 

b. Launch shell on Yarn with am.memory greater than nodemanager.resource memory and yarn.scheduler.maximum-allocation-mb

eg; spark-shell --master yarn --conf spark.yarn.am.memory=17g

 Error:

java.lang.IllegalArgumentException: Required AM memory (17408+1740 MB) is above the max threshold (8096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.

at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:325)

 

Expected : Error request for scenario2 should be more around yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

Issue Links

• links to: [Github] Pull Request #22199 (sujith71955)