[SPARK-7504] NullPointerException when initializing SparkContext in YARN-cluster mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Deploy, YARN
• Labels: deployment yarn yarn-client
• Target Version/s: 1.4.0
• Assignee: Zoltán Zvara
• Reporter: Zoltán Zvara
• Created: 09/May/15 14:31
• Updated: 16/May/15 11:07
• Resolved: 15/May/15 19:17

Description

It is not clear for most users that, while running Spark on YARN a SparkContext with a given execution plan can be run locally as yarn-client, but can not deploy itself to the cluster. This is currently performed using org.apache.spark.deploy.yarn.Client. I think we should support deployment through SparkContext, but this is not the point I wish to make here.

Configuring a SparkContext to deploy itself currently will yield an ERROR while accessing spark.yarn.app.id in YarnClusterSchedulerBackend, and after that a NullPointerException while referencing the ApplicationMaster instance.

Spark should clearly inform the user that it might be running in yarn-cluster mode without a proper submission using Client and that deploying is not supported directly from SparkContext.

Issue Links

• links to: [Github] Pull Request #6029 (ehnalis)
• links to: [Github] Pull Request #6083 (ehnalis)