[SPARK-18935] Use Mesos "Dynamic Reservation" resource for Spark

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.0.1, 2.0.2
• Fix Version/s: 2.3.0
• Component/s: Mesos
• Assignee: Stavros Kontopoulos
• Reporter: jackyoh
• Created: 20/Dec/16 04:05
• Updated: 29/Nov/17 22:16
• Resolved: 29/Nov/17 22:16

Description

I'm running spark on Apache Mesos

Please follow these steps to reproduce the issue:

1. First, run Mesos resource reserve:

curl -i -d slaveId=c24d1cfb-79f3-4b07-9f8b-c7b19543a333-S0 -d resources='[{"name":"cpus","type":"SCALAR","scalar":

{"value":20}

,"role":"spark","reservation":{"principal":""}},{"name":"mem","type":"SCALAR","scalar":

{"value":4096}

,"role":"spark","reservation":{"principal":""}}]' -X POST http://192.168.1.118:5050/master/reserve

2. Then run spark-submit command:

./spark-submit --class org.apache.spark.examples.SparkPi --master mesos://192.168.1.118:5050 --conf spark.mesos.role=spark ../examples/jars/spark-examples_2.11-2.0.2.jar 10000

And the console will keep loging same warning message as shown below:

16/12/19 22:33:28 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Issue Links

• links to: [Github] Pull Request #19390 (skonto)