Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22567

spark.mesos.executor.memoryOverhead equivalent for the Driver when running on Mesos

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: Mesos
    • Labels:

      Description

      spark.mesos.executor.memoryOverhead is:
      "The amount of additional memory, specified in MB, to be allocated per executor. By default, the overhead will be larger of either 384 or 10% of spark.executor.memory"

      It is important for every JVM process to have memory available to it, beyond its heap (Xmx) for native allocations.

      When using the MesosClusterDispatcher and running the Driver on Mesos (https://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode), it appears that the Driver's mesos sandbox is allocated with the same amount of memory (configured with spark.driver.memory) as the heap (Xmx) itself. This increases the prevalence of OOM exceptions.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              michaelmoss Michael Moss
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: