[SPARK-17522] [MESOS] More even distribution of executors on Mesos cluster

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: Mesos
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Sun Rui
• Created: 13/Sep/16 09:19
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

The MesosCoarseGrainedSchedulerBackend launch executors in a round-robin way among accepted offers that are received at once, but it is observed that typically executors are launched on a small number of slaves.

It is found that MesosCoarseGrainedSchedulerBackend mostly is receiving only one offer once on a cluster composed of many nodes, so that the round-robin assignment of executors among offers do not have expected result, which leads to the fact that executors are located on a smaller number of slave nodes than expected, which suffers bad data locality.

An experimental slight change to MesosCoarseGrainedSchedulerBackend::buildMesosTasks() shows better executor distribution among nodes:

    while (launchTasks) {
      launchTasks = false

      for (offer <- offers) {
        ...
       }

+      if (conf.getBoolean("spark.deploy.spreadOut", true)) {
+        launchTasks = false
+      }
    }
    tasks.toMap

One of my spark programs can run 30% faster due to this change because of better data locality.