[TEZ-1885] Should check whether AM is shutting down first in submitDAGToAppMaster

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Target Version/s: 0.8.6
• Assignee: Jeff Zhang
• Reporter: Jeff Zhang
• Created: 22/Dec/14 05:52
• Updated: 14/Mar/17 03:40

Description

Here's logs which demonstrate the scenario: Shutdown AM -> submit new DAG
But the error message is not correct (" App master already running a DAG" ), it should be "AM unable to accept new DAG submissions, In the process of shutting down"

2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster: No current running DAG, shutting down the AM
2014-12-16 06:26:54,372 INFO [Timer-0] org.apache.tez.dag.app.DAGAppMaster: Handling DAGAppMaster shutdown
2014-12-16 06:26:56,465 INFO [IPC Server handler 0 on 37746] org.apache.hadoop.ipc.Server: IPC Server handler 0 on 37746, call org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.submitDAG from 10.66.7.123:35917 Call#7994 Retry#0: error: org.apache.tez.dag.api.TezException: App master already running a DAG
org.apache.tez.dag.api.TezException: App master already running a DAG
                at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:931)
                at org.apache.tez.dag.app.DAGAppMaster$DAGClientHandler.submitDAG(DAGAppMaster.java:1065)
                at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:131)
                at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:8292)
                at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)
                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
                at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
                at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:415)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
2014-12-16 06:26:56,491 INFO [IPC Server handler 0 on 37746] org.apache.tez.dag.app.DAGAppMaster: Received message to shutdown AM

Issue Links

• is related to: TEZ-1273 Refactor DAGAppMaster to state machine based