Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23670

Memory leak of SparkPlanGraphWrapper in sparkUI

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • SQL
    • None

    Description

      Memory leak on driver for a long time running application. We have application using structured streaming and running 48 hours. But driver fails with out of memory after 25 hours. After investigating heap dump we found that most of the memory was occupied with a lot of SparkPlanGraphWrapper objects inside InMemoryStore.

       Application was run with option: 
      --driver-memory 4G

      Attachments

        1. Screen Shot 2018-09-06 at 23.19.56.png
          318 kB
          Michael Spector
        2. heapdump_OOM.png
          994 kB
          Michael Spector
        3. heap.png
          99 kB
          Myroslav Lisniak

        Activity

          People

            myroslav.lisniak Myroslav Lisniak
            myroslav.lisniak Myroslav Lisniak
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: