[TEZ-2663] SessionNotRunning exceptions are wrapped in a ServiceException from a dying AM

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0-alpha, 0.7.1
• Assignee: Hitesh Shah
• Reporter: Gopal Vijayaraghavan
• Created: 30/Jul/15 18:23
• Updated: 16/Sep/15 01:14
• Resolved: 06/Aug/15 18:08

Description

The scenario in TEZ-2548 throws a SessionNotRunning from the AM right now, which gets wrapped in a ServiceException by the protobuf layer, so that the exception thrown by TezClient cannot be caught and handled cleanly.

2015-07-30 01:32:50,997 ERROR [HiveServer2-Background-Pool: Thread-494()]: exec.Task (TezTask.java:execute(191)) - Failed to execute tez graph.
org.apache.tez.dag.api.TezException: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.SessionNotRunning): AM unable to accept new DAG submissions. In the process of shutting down
        at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1265)
        at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:120)
        at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
        at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)

Issue Links

• is related to: TEZ-2548 TezClient submitDAG can hang if the AM is in the process of shutting down