Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21492

Memory leak in SortMergeJoin

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0, 2.3.0, 2.3.1, 3.0.0
    • 2.4.5, 3.0.0
    • SQL
    • None

    Description

      In SortMergeJoin, if the iterator is not exhausted, there will be memory leak caused by the Sort. The memory is not released until the task end, and cannot be used by other operators causing performance drop or OOM.

      Attachments

        Issue Links

          Activity

            People

              XuanYuan Yuanjian Li
              zhzhan Zhan Zhang
              Votes:
              3 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: