Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25380

Generated plans occupy over 50% of Spark driver memory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.3.1
    • None
    • Spark Core

    • Spark 2.3.1 (AWS emr-5.16.0)

       

    Description

      When debugging an OOM exception during long run of a Spark application (many iterations of the same code) I've found that generated plans occupy most of the driver memory. I'm not sure whether this is a memory leak or not, but it would be helpful if old plans could be purged from memory anyways.

      Attached are screenshots of OOM heap dump opened in JVisualVM.

       

      Attachments

        1. Screen Shot 2018-09-12 at 8.20.05.png
          385 kB
          Michael Spector
        2. Screen Shot 2018-09-06 at 23.19.56.png
          318 kB
          Michael Spector
        3. image-2018-09-16-14-21-38-939.png
          715 kB
          Nir Hedvat
        4. heapdump_OOM.png
          994 kB
          Michael Spector

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26103 OutOfMemory error with large query plans

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              spektom Michael Spector
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: