Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5043

Fetch failure processing can cause AM event queue to backup and eventually OOM

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.7, 2.1.0-beta
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: mr-am
    • Labels:
      None

      Description

      Saw an MRAppMaster with a 3G heap OOM. Upon investigating another instance of it running, we saw the UI in a weird state where the task table and task attempt tables in the job overview page weren't consistent. The AM log showed the AsyncDispatcher had hundreds of thousands of events in the event queue, and jstacks showed it spending a lot of time in fetch failure processing. It turns out fetch failure processing is currently very expensive, with a triple for loop where the inner loop is calling the quite-expensive TaskAttempt.getReport. That function ends up type-converting the entire task report, counters and all, and performing locale conversions among other things. It does this for every reduce task in the job, for every map task that failed. And when it's done building up the large task report, it pulls out one field, the phase, then throws the report away.

      While the AM is busy processing fetch failures, tasks attempts are continuing to send events to the AM including memory-expensive events like status updates which include the counters. These back up in the AsyncDispatcher event queue and eventually even an AM with a large heap size will run out of memory and crash or expire because it thrashes in garbage collect.

        Attachments

        1. MAPREDUCE-5043.patch
          8 kB
          Jason Darrell Lowe

        Issue Links

          Activity

            People

            • Assignee:
              jlowe Jason Darrell Lowe
              Reporter:
              jlowe Jason Darrell Lowe

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment