Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5493

In-memory map outputs can be leaked after shuffle completes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 0.23.9
    • Fix Version/s: 2.1.1-beta
    • Component/s: mrv2
    • Labels:
      None

      Description

      MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and inMemoryMapOutputs to a list of map outputs that is subsequently processed, but it does not clear those sets. This prevents some of the map outputs from being garbage collected and significantly reduces the memory available for the subsequent reduce phase.

        Attachments

        1. MAPREDUCE-5493.patch
          3 kB
          Jason Darrell Lowe
        2. MAPREDUCE-5493-branch-0.23.patch
          4 kB
          Jason Darrell Lowe

          Issue Links

            Activity

              People

              • Assignee:
                jlowe Jason Darrell Lowe
                Reporter:
                jlowe Jason Darrell Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: