Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26525

Fast release memory of ShuffleBlockFetcherIterator

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.2
    • Fix Version/s: 3.0.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      Currently, spark would not release ShuffleBlockFetcherIterator until the whole task finished.

      In some conditions, it incurs memory leak.

      An example is Shuffle -> map -> Coalesce(shuffle = false). Each ShuffleBlockFetcherIterator contains  some metas about MapStatus(blocksByAddress) and each ShuffleMapTask will keep n(max to shuffle partitions) shuffleBlockFetcherIterator for they are refered by onCompleteCallbacks of TaskContext, in some case, it may take huge memory and the memory will not released until the task finished.

      Actually, We can release ShuffleBlockFetcherIterator as soon as it's consumed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liupengcheng liupengcheng
                Reporter:
                liupengcheng liupengcheng
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: