Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18227

Tez parallel execution fail

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Tez
    • Labels:
      None
    • Target Version/s:

      Description

      Running tez Dag in parallel within a session fail. Here is the test case:

      set hive.exec.parallel=true;
      set hive.merge.tezfiles=true;
      set tez.grouping.max-size=100000;
      set tez.grouping.min-size=10000;
      
      from student
      insert overwrite table student4 select *
      insert overwrite table student5 select *
      insert overwrite table student6 select *;
      

      The merge task run in parallel and result the exception:

      org.apache.tez.dag.api.TezException: App master already running a DAG
              at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
              at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
              at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
              at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
      

        Attachments

        1. HIVE-18227.4.patch
          3 kB
          László Bodor
        2. HIVE-18227.3.patch
          4 kB
          Daniel Dai
        3. HIVE-18227.2.patch
          4 kB
          Daniel Dai
        4. HIVE-18227.1.patch
          4 kB
          Daniel Dai

          Activity

            People

            • Assignee:
              daijy Daniel Dai
              Reporter:
              daijy Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: