Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4158

Spark throws exception when Mesos resources are missing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.1, 1.2.0
    • Component/s: Mesos
    • Labels:
      None

      Description

      Spark throws an exception when trying to check resources which haven't been offered by Mesos. This is an error in Spark, and should be corrected as such. Here's a sample:

      val data Exception in thread "Thread-41" java.lang.IllegalArgumentException: No resource called cpus in [name: "mem"
      type: SCALAR
      scalar {
        value: 2067.0
      }
      role: "*"
      , name: "disk"
      type: SCALAR
      scalar {
        value: 900.0
      }
      role: "*"
      , name: "ports"
      type: RANGES
      ranges {
        range {
          begin: 31000
          end: 32000
        }
      }
      role: "*"
      ]
              at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.org$apache$spark$scheduler$cluster$mesos$CoarseMesosSchedulerBackend$$getResource(CoarseMesosSchedulerBackend.scala:236)
              at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend$$anonfun$resourceOffers$1.apply(CoarseMesosSchedulerBackend.scala:200)
              at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend$$anonfun$resourceOffers$1.apply(CoarseMesosSchedulerBackend.scala:197)
              at scala.collection.Iterator$class.foreach(Iterator.scala:727)
              at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
              at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
              at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
              at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.resourceOffers(CoarseMesosSchedulerBackend.scala:197)
      

        Attachments

          Activity

            People

            • Assignee:
              brenden Brenden Matthews
              Reporter:
              brenden Brenden Matthews
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: