Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21157

Report Total Memory Used by Spark Executors

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.1.1
    • None
    • Input/Output
    • None

    Description

      Building on some of the core ideas of SPARK-9103, this JIRA proposes tracking total memory used by Spark executors, and a means of broadcasting, aggregating, and reporting memory usage data in the Spark UI.

      Here, "total memory used" refers to memory usage that is visible outside of Spark, to an external observer such as YARN, Mesos, or the operating system. The goal of this enhancement is to give Spark users more information about how Spark clusters are using memory. Total memory will include non-Spark JVM memory and all off-heap memory.

      Please consult the attached design document for further details.

      Attachments

        1. TotalMemoryReportingDesignDoc.pdf
          317 kB
          Jose Soltren

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue SPARK-24958 Add executors' process tree total memory information to heartbeat signals

          • Major - Major loss of function.
          • Resolved
          relates to

          Umbrella - An overarching type made of sub-tasks SPARK-23206 Additional Memory Tuning Metrics

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              jsoltren Jose Soltren
              Votes:
              5 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: