Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13958

Executor OOM due to unbounded growth of pointer array in Sorter

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.1
    • Fix Version/s: 1.6.2, 2.0.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      While running a job we saw that the executors are OOMing because in UnsafeExternalSorter's growPointerArrayIfNecessary function, we are just growing the pointer array indefinitely.

      https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L292

      This is a regression introduced in PR- https://github.com/apache/spark/pull/11095

        Attachments

          Activity

            People

            • Assignee:
              sitalkedia@gmail.com Sital Kedia
              Reporter:
              sitalkedia@gmail.com Sital Kedia
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: