Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31689

ShuffleBlockFetchIterator keeps localBlocks in its memory even though it never uses it

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.4.5
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      The localBlocks is created and used in the initialize method of ShuffleBlockFetchIterator but is never used after that. 
      It can be local to the initialize method instead of being a field in the ShuffleBlockFetchIterator instance. It hold son to memory until iterator instance is alive which is unnecessary.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              csingh Chandni Singh
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: