Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24437

Memory leak in UnsafeHashedRelation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.2.0
    • None
    • SQL

    Description

      There seems to memory leak with org.apache.spark.sql.execution.joins.UnsafeHashedRelation

      We have a long running instance of STS.

      With each query execution requiring Broadcast Join, UnsafeHashedRelation is getting added for cleanup in ContextCleaner. This reference of UnsafeHashedRelation is being held at some other Collection and not becoming eligible for GC and because of this ContextCleaner is not able to clean it.

      Attachments

        1. Screen Shot 2018-11-01 at 10.38.30 AM.png
          258 kB
          David Vogelbacher
        2. Screen Shot 2018-05-30 at 2.07.22 PM.png
          491 kB
          gagan taneja
        3. Screen Shot 2018-05-30 at 2.05.40 PM.png
          269 kB
          gagan taneja

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-22575 Making Spark Thrift Server clean up its cache

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              tanejagagan gagan taneja
              Votes:
              2 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: