[HIVE-18227] Tez parallel execution fail

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Tez
• Target Version/s: 3.0.0
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 05/Dec/17 22:46
• Updated: 20/Jul/19 06:30

Description

Running tez Dag in parallel within a session fail. Here is the test case:

set hive.exec.parallel=true;
set hive.merge.tezfiles=true;
set tez.grouping.max-size=100000;
set tez.grouping.min-size=10000;

from student
insert overwrite table student4 select *
insert overwrite table student5 select *
insert overwrite table student6 select *;

The merge task run in parallel and result the exception:

org.apache.tez.dag.api.TezException: App master already running a DAG
        at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
        at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
        at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
        at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)