Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23670

Memory leak of SparkPlanGraphWrapper in sparkUI

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • SQL
    • None

    Description

      Memory leak on driver for a long time running application. We have application using structured streaming and running 48 hours. But driver fails with out of memory after 25 hours. After investigating heap dump we found that most of the memory was occupied with a lot of SparkPlanGraphWrapper objects inside InMemoryStore.

       Application was run with option: 
      --driver-memory 4G

      Attachments

        1. heap.png
          99 kB
          Myroslav Lisniak
        2. heapdump_OOM.png
          994 kB
          Michael Spector
        3. Screen Shot 2018-09-06 at 23.19.56.png
          318 kB
          Michael Spector

        Activity

          People

            myroslav.lisniak Myroslav Lisniak
            myroslav.lisniak Myroslav Lisniak
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: