Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22574

Wrong request causing Spark Dispatcher going inactive

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.2, 2.3.0
    • Component/s: Mesos, Spark Submit
    • Labels:
      None

      Description

      When submitting a wrong CreateSubmissionRequest to Spark Dispatcher is causing a bad state of Dispatcher and making it inactive as a mesos framework.

      The class CreateSubmissionRequest initialise its arguments to null as follows:

      CreateSubmissionRequest.scala
        var appResource: String = null
        var mainClass: String = null
        var appArgs: Array[String] = null
        var sparkProperties: Map[String, String] = null
        var environmentVariables: Map[String, String] = null
      

      There are some checks of these variables but not in all of them, for example in appArgs and environmentVariables.

      If you don't set appArgs it will cause the following error:

      error
      17/11/21 14:37:24 INFO MesosClusterScheduler: Reviving Offers.
      Exception in thread "Thread-22" java.lang.NullPointerException
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.getDriverCommandValue(MesosClusterScheduler.scala:444)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.buildDriverCommand(MesosClusterScheduler.scala:451)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.org$apache$spark$scheduler$cluster$mesos$MesosClusterScheduler$$createTaskInfo(MesosClusterScheduler.scala:538)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:570)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler$$anonfun$scheduleTasks$1.apply(MesosClusterScheduler.scala:555)
      	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.scheduleTasks(MesosClusterScheduler.scala:555)
      	at org.apache.spark.scheduler.cluster.mesos.MesosClusterScheduler.resourceOffers(MesosClusterScheduler.scala:621)
      

      Because it's trying to access to it without checking whether is null or not.

        Attachments

          Activity

            People

            • Assignee:
              gschiavon German Schiavon Matteo
              Reporter:
              gschiavon German Schiavon Matteo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: