Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9103

Tracking spark's memory usage

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • Spark Core, Web UI

    Description

      Currently spark only provides little memory usage information (RDD cache on webUI) for the executors. User have no idea on what is the memory consumption when they are running spark applications with a lot of memory used in spark executors. Especially when they encounter the OOM, it’s really hard to know what is the cause of the problem. So it would be helpful to give out the detail memory consumption information for each part of spark, so that user can clearly have a picture of where the memory is exactly used.

      The memory usage info to expose should include but not limited to shuffle, cache, network, serializer, etc.

      User can optionally choose to open this functionality since this is mainly for debugging and tuning.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              liyezhang556520 Zhang, Liye
              Votes:
              4 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: