[SPARK-4158] Spark throws exception when Mesos resources are missing

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Mesos
• Target Version/s: 1.1.1, 1.2.0
• Assignee: Brenden Matthews
• Reporter: Brenden Matthews
• Created: 30/Oct/14 20:34
• Updated: 06/Nov/14 00:03
• Resolved: 06/Nov/14 00:03

Description

Spark throws an exception when trying to check resources which haven't been offered by Mesos. This is an error in Spark, and should be corrected as such. Here's a sample:

val data Exception in thread "Thread-41" java.lang.IllegalArgumentException: No resource called cpus in [name: "mem"
type: SCALAR
scalar {
  value: 2067.0
}
role: "*"
, name: "disk"
type: SCALAR
scalar {
  value: 900.0
}
role: "*"
, name: "ports"
type: RANGES
ranges {
  range {
    begin: 31000
    end: 32000
  }
}
role: "*"
]
        at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.org$apache$spark$scheduler$cluster$mesos$CoarseMesosSchedulerBackend$$getResource(CoarseMesosSchedulerBackend.scala:236)
        at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend$$anonfun$resourceOffers$1.apply(CoarseMesosSchedulerBackend.scala:200)
        at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend$$anonfun$resourceOffers$1.apply(CoarseMesosSchedulerBackend.scala:197)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.resourceOffers(CoarseMesosSchedulerBackend.scala:197)

Issue Links

• links to: [SPARK-4158] Fix for missing resources. #3024