Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2468 Netty-based block server / client module
  3. SPARK-3018

Release all ManagedBuffers upon task completion/failure

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      BlockFetcherIterator retains ManagedBuffers returned by BlockClient.fetchBlocks. Those buffers are released when the iterators are traversed fully. In the case of task failures or completion without exhausting the iterator, this could lead to memory leak.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rxin Reynold Xin
                Reporter:
                rxin Reynold Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: