Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5543

In-memory map outputs can be leaked after shuffle completes in 0.23

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.9
    • Fix Version/s: 0.23.10
    • Component/s: mrv2
    • Labels:
      None

      Description

      MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and inMemoryMapOutputs to a list of map outputs that is subsequently processed, but it does not clear those sets. This prevents some of the map outputs from being garbage collected and significantly reduces the memory available for the subsequent reduce phase.

      This was fixed for trunk and branch-2 by MAPREDUCE-5493, but that has since been closed after 2.1.1 released. This JIRA tracks backporting the fix to branch-0.23 as well.

        Attachments

        1. MAPREDUCE-5493-branch-0.23.patch
          4 kB
          Jason Darrell Lowe

        Issue Links

          Activity

            People

            • Assignee:
              jlowe Jason Darrell Lowe
              Reporter:
              jlowe Jason Darrell Lowe

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment