Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5095

Support launching multiple mesos executors in coarse grained mesos mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Mesos
    • Labels:
      None
    • Target Version/s:

      Description

      Currently in coarse grained mesos mode, it's expected that we only launch one Mesos executor that launches one JVM process to launch multiple spark executors.

      However, this become a problem when the JVM process launched is larger than an ideal size (30gb is recommended value from databricks), which causes GC problems reported on the mailing list.

      We should support launching mulitple executors when large enough resources are available for spark to use, and these resources are still under the configured limit.

      This is also applicable when users want to specifiy number of executors to be launched on each node

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tnachen Timothy Chen
                Reporter:
                tnachen Timothy Chen
              • Votes:
                8 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: