[SPARK-823] spark.default.parallelism's default is inconsistent across scheduler backends

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.7.3, 0.8.0, 0.9.1
• Component/s: Documentation, PySpark, Scheduler, Spark Core
• Assignee: Ilya Ganelin
• Reporter: Josh Rosen
• Created: 19/Jul/13 09:24
• Updated: 17/May/20 17:48
• Resolved: 09/Feb/15 16:31

Description

The 0.7.3 configuration guide says that spark.default.parallelism's default is 8, but the default is actually max(totalCoreCount, 2) for the standalone scheduler backend, 8 for the Mesos scheduler, and threads for the local scheduler:

https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala#L157
https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/mesos/MesosSchedulerBackend.scala#L317
https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/local/LocalScheduler.scala#L150

Should this be clarified in the documentation? Should the Mesos scheduler backend's default be revised?