[TEZ-4561] Improve reported exception when DAGAppMaster is shutting down

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.5
• Assignee: Ayush Saxena
• Reporter: László Bodor
• Created: 06/May/24 13:28
• Updated: 03/Sep/24 10:46
• Resolved: 03/Sep/24 10:46

Description

https://github.com/apache/tez/blob/66a6ca64b5edde0d30bea0962cb132f3c4982469/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L1683

the AM can return this exception during a shutdown like below:

TezUncheckedException: Cannot get ApplicationACLs before all services have started
   at org.apache.tez.dag.app.DAGAppMaster$RunningAppContext.getApplicationACLs(DAGAppMaster.java:1733)
   at org.apache.tez.dag.app.rm.container.AMContainerImpl$LaunchRequestTransition.transition(AMContainerImpl.java:513)
   at org.apache.tez.dag.app.rm.container.AMContainerImpl$LaunchRequestTransition.transition(AMContainerImpl.java:470)
   at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
   at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
   at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
   at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:493)
   at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:64)
   at org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:441)
   at org.apache.tez.dag.app.rm.container.AMContainerImpl.handle(AMContainerImpl.java:78)
   at org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:68)
   at org.apache.tez.dag.app.rm.container.AMContainerMap.handle(AMContainerMap.java:40)
   at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:200)
   at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:118)
   at java.base/java.lang.Thread.run(Thread.java:829)\r

which is confusing, and doesn't make the log reader aware that getServiceState() != STATE.STARTED is not an initialization problem (especially confusing in case of an AM which is already running for a long time), instead STATE.STOPPED

we should check that and report (maybe even with a timestamp when the shutdownhook was started)

Issue Links

• links to: GitHub Pull Request #365