Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2468 Netty-based block server / client module
  3. SPARK-3018

Release all ManagedBuffers upon task completion/failure

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      BlockFetcherIterator retains ManagedBuffers returned by BlockClient.fetchBlocks. Those buffers are released when the iterators are traversed fully. In the case of task failures or completion without exhausting the iterator, this could lead to memory leak.

        Attachments

          Activity

            People

            • Assignee:
              rxin Reynold Xin
              Reporter:
              rxin Reynold Xin

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment