Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-823

spark.default.parallelism's default is inconsistent across scheduler backends

    XMLWordPrintableJSON

    Details

      Description

      The 0.7.3 configuration guide says that spark.default.parallelism's default is 8, but the default is actually max(totalCoreCount, 2) for the standalone scheduler backend, 8 for the Mesos scheduler, and threads for the local scheduler:

      https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala#L157
      https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/mesos/MesosSchedulerBackend.scala#L317
      https://github.com/mesos/spark/blob/v0.7.3/core/src/main/scala/spark/scheduler/local/LocalScheduler.scala#L150

      Should this be clarified in the documentation? Should the Mesos scheduler backend's default be revised?

        Attachments

          Activity

            People

            • Assignee:
              ilganeli Ilya Ganelin
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: