[SPARK-22574] Wrong request causing Spark Dispatcher going inactive

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.2, 2.3.0
• Component/s: Mesos, Spark Submit
• Assignee: German Schiavon Matteo
• Reporter: German Schiavon Matteo
• Created: 21/Nov/17 15:51
• Updated: 01/Oct/24 17:49
• Resolved: 13/Dec/17 21:37

Description

When submitting a wrong CreateSubmissionRequest to Spark Dispatcher is causing a bad state of Dispatcher and making it inactive as a mesos framework.

The class CreateSubmissionRequest initialise its arguments to null as follows:

CreateSubmissionRequest.scala

  var appResource: String = null
  var mainClass: String = null
  var appArgs: Array[String] = null
  var sparkProperties: Map[String, String] = null
  var environmentVariables: Map[String, String] = null

There are some checks of these variables but not in all of them, for example in appArgs and environmentVariables.

If you don't set appArgs it will cause the following error:

error

17/11/21 14:37:24 INFO MesosClusterScheduler: Reviving Offers.
Exception in thread "Thread-22" java.lang.NullPointerException
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555)
	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621)

Because it's trying to access to it without checking whether is null or not.

Issue Links

• is related to: SPARK-49845 Make `appArgs` and `environmentVariables` optional in REST API
• links to: [Github] Pull Request #19793 (Gschiavon)
• links to: [Github] Pull Request #19966 (Gschiavon)