Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26418

Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      In current code path,  OneForOneStreamManager holds StreamState in a Map named streams. 

      A StreamState is initialized and put into streams when OpenBlocks request received.

      One specific StreamState is removed from streams in two scenarios below:

      1. The last chunk of a stream is fetched
      2. The connection of ChunkFetch is closed

      StreamState will never be clean up, if OpenBlocks request is received without and following  ChunkFetch request. This will cause memory leak in server side, which is harmful for long running service such as ExternalShuffleService.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wang-shuo Wang Shuo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: