[HIVE-7857] Hive query fails after Tez session times out

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Component/s: Tez
• Assignee: Vikram Dixit K
• Reporter: Vikram Dixit K
• Created: 22/Aug/14 22:03
• Updated: 13/Nov/14 19:40
• Resolved: 30/Sep/14 18:37

Description

Originally reported by deepesh
Steps to reproduce:

Open the Hive CLI, ensure that HIVE_AUX_JARS_PATH has hcatalog-core.jar in the path.
Keep it idle for more than 5 minutes (this is the default tez session timeout). Essentially Tez session should time out.
Run a Hive on Tez query, the query fails. Here is a sample CLI session:

    hive> select from_unixtime(unix_timestamp(), "dd-MMM-yyyy") from vectortab10korc limit 1;
    Query ID = hrt_qa_20140626002525_6e964079-4031-406b-85ed-cda9c65dca22
    Total jobs = 1
    Launching Job 1 out of 1
    Tez session was closed. Reopening...
    Session re-established.


    Status: Running (application id: application_1403688364015_1930)

    Map 1: -/-
    Map 1: 0/1
    Map 1: 0/1
    Map 1: 0/1
    Map 1: 0/1
    Map 1: 0/1
    Status: Failed
    Vertex failed, vertexName=Map 1, vertexId=vertex_1403688364015_1930_1_00, diagnostics=[Task failed, taskId=task_1403688364015_1930_1_00_000000, diagnostics=[AttemptID:attempt_1403688364015_1930_1_00_000000_0 Info:Container container_1403688364015_1930_01_000002 COMPLETED with diagnostics set to [Resource hdfs://ambari-sec-1403670773-others-2-1.cs1cloud.internal:8020/tmp/hive-hrt_qa/_tez_session_dir/3d3ef758-90f3-4bb3-86cb-902aeb3b8830/hive-hcatalog-core-0.13.0.2.1.3.0-554.jar changed on src filesystem (expected 1403741969169, was 1403742347351
    ], AttemptID:attempt_1403688364015_1930_1_00_000000_1 Info:Container container_1403688364015_1930_01_000003 COMPLETED with diagnostics set to [Resource hdfs://ambari-sec-1403670773-others-2-1.cs1cloud.internal:8020/tmp/hive-hrt_qa/_tez_session_dir/3d3ef758-90f3-4bb3-86cb-902aeb3b8830/hive-hcatalog-core-0.13.0.2.1.3.0-554.jar changed on src filesystem (expected 1403741969169, was 1403742347351
    ], AttemptID:attempt_1403688364015_1930_1_00_000000_2 Info:Container container_1403688364015_1930_01_000004 COMPLETED with diagnostics set to [Resource hdfs://ambari-sec-1403670773-others-2-1.cs1cloud.internal:8020/tmp/hive-hrt_qa/_tez_session_dir/3d3ef758-90f3-4bb3-86cb-902aeb3b8830/hive-hcatalog-core-0.13.0.2.1.3.0-554.jar changed on src filesystem (expected 1403741969169, was 1403742347351
    ], AttemptID:attempt_1403688364015_1930_1_00_000000_3 Info:Container container_1403688364015_1930_01_000005 COMPLETED with diagnostics set to [Resource hdfs://ambari-sec-1403670773-others-2-1.cs1cloud.internal:8020/tmp/hive-hrt_qa/_tez_session_dir/3d3ef758-90f3-4bb3-86cb-902aeb3b8830/hive-hcatalog-core-0.13.0.2.1.3.0-554.jar changed on src filesystem (expected 1403741969169, was 1403742347351
    ]], Vertex failed as one or more tasks failed. failedTasks:1]
    DAG failed due to vertex failure. failedVertices:1 killedVertices:0
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask

Issue Links

• links to: Review board