[MAPREDUCE-7249] Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
• Component/s: applicationmaster, mrv2
• Labels: Reviewed
• Assignee: Wilfred Spiegelenburg
• Reporter: Wilfred Spiegelenburg
• Created: 27/Nov/19 10:14
• Updated: 17/Aug/20 21:44
• Resolved: 28/Nov/19 09:26

Description

Same issue as in MAPREDUCE-7240 but this one has a different state in which the Exception TA_TOO_MANY_FETCH_FAILURE event is received:

2019-11-18 23:03:40,270 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this event at current state for attempt_1568654141590_630203_m_003108_1
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1183)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:148)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1388)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1380)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)

The stack trace is from a CDH release which is highly patched 2.6 release.

Issue Links

• is related to: MAPREDUCE-7240 Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER' cause job error
• relates to: MAPREDUCE-7264 overall reduction of ApplicationMaster exit because of unhandled TA_TOO_MANY_FETCH_FAILURE event