[MAPREDUCE-5124] AM lacks flow control for task events

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.3-alpha, 0.23.5
• Fix Version/s: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
• Component/s: mr-am
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Jason Darrell Lowe
• Created: 02/Apr/13 14:30
• Updated: 14/Feb/18 17:31
• Resolved: 01/Dec/17 20:36

Description

The AM does not have any flow control to limit the incoming rate of events from tasks. If the AM is unable to keep pace with the rate of incoming events for a sufficient period of time then it will eventually exhaust the heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event processing, but the AM could still get behind if it's starved for CPU and/or handling a very large job with tens of thousands of active tasks.

Issue Links

• breaks: MAPREDUCE-7028 Concurrent task progress updates causing NPE in Application Master
• breaks: MAPREDUCE-7020 Task timeout in uber mode can crash AM
• breaks: MAPREDUCE-7053 Timed out tasks can fail to produce thread dump
• is related to: MAPREDUCE-6242 Progress report log is incredibly excessive in application master
• relates to: YARN-270 RM scheduler event handler thread gets behind
• relates to: MAPREDUCE-5043 Fetch failure processing can cause AM event queue to backup and eventually OOM
• relates to: YARN-3630 YARN should suggest a heartbeat interval for applications