Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23748

tez task with File Merge operator generate tmp file with wrong suffix

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Tez
    • None

    Description

      background

      • SQL on TEZ 
      • it's a Occasional problem

      hiveserver2 log

      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Connecting to jdbc:hive2://xxx:10000/prod
      Connected to: Apache Hive (version 3.1.0.3.1.4.0-315)
      Driver: Hive JDBC (version 3.1.0.3.1.4.0-315)
      Transaction isolation: TRANSACTION_REPEATABLE_READ
      INFO : Compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
      INFO : Completed compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.887 seconds
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod
      INFO : Starting task [Stage-0:DDL] in serial mode
      INFO : Completed executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.197 seconds
      INFO : OK
      INFO : Concurrency mode is disabled, not creating a lock manager
      No rows affected (1.096 seconds)
      No rows affected (0.004 seconds)
      INFO : Compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
      INFO : Completed compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 1.324 seconds
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase
      INFO : Starting task [Stage-0:DDL] in serial mode
      INFO : Completed executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 12.895 seconds
      INFO : OK
      INFO : Concurrency mode is disabled, not creating a lock manager
      No rows affected (14.229 seconds)
      INFO : Compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): xxxxx
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, event
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, type:string, comment:null), FieldSchema(name:device_id, type:string, comment:null), FieldSchema(name:is_new, type:int, comment:null), FieldSchema(name:first_attribute, type:map<string,string>, comment:null), FieldSchema(name:first_app_version, type:string, comment:null), FieldSchema(name:first_platform_type, type:string, comment:null), FieldSchema(name:first_manufacturer, type:string, comment:null), FieldSchema(name:first_model, type:string, comment:null), FieldSchema(name:first_ipprovince, type:string, comment:null), FieldSchema(name:first_ipcity, type:string, comment:null), FieldSchema(name:last_attribute, type:map<string,string>, comment:null), FieldSchema(name:last_app_version, type:string, comment:null), FieldSchema(name:last_platform_type, type:string, comment:null), FieldSchema(name:last_manufacturer, type:string, comment:null), FieldSchema(name:last_model, type:string, comment:null), FieldSchema(name:last_ipprovince, type:string, comment:null), FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null)
      INFO : Completed compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); Time taken: 78.517 seconds
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Executing command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): xxxx
      INFO : Query ID = hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
      INFO : Total jobs = 3
      INFO : Launching Job 1 out of 3
      INFO : Starting task [Stage-1:MAPRED] in serial mode
      INFO : Subscribed to counters: [] for queryId: hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
      INFO : Tez session hasn't been created yet. Opening session
      INFO : Dag name: xxxx
      INFO : Status: Running (Executing on YARN cluster with App id application_1590738759875_100602)
      。。。。
      INFO : Starting task [Stage-8:CONDITIONAL] in serial mode
      INFO : Stage-5 is filtered out by condition resolver.
      INFO : Stage-4 is selected by condition resolver.
      INFO : Stage-6 is filtered out by condition resolver.
      INFO : Launching Job 3 out of 3
      INFO : Starting task [Stage-4:MAPRED] in serial mode
      INFO : Subscribed to counters: [] for queryId: hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
      INFO : Session is already open
      INFO : Dag name:xxxx
      INFO : Status: Running (Executing on YARN cluster with App id application_1590738759875_100602)
      ERROR : Status: Failed
      ERROR : Vertex failed, vertexName=File Merge, vertexId=vertex_1590738759875_100602_2_00, diagnostics=[Task failed, taskId=task_1590738759875_100602_2_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_0:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000002_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000002_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_1:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000009_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000009_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_2:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000004_1 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000004_1
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_3:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000006_2 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000006_2
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1590738759875_100602_2_00 [File Merge] killed/failed due to:OWN_TASK_FAILURE]
      ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
      INFO : org.apache.tez.common.counters.DAGCounter:
      INFO : NUM_FAILED_TASKS: 7
      INFO : NUM_SUCCEEDED_TASKS: 9
      INFO : TOTAL_LAUNCHED_TASKS: 16
      INFO : OTHER_LOCAL_TASKS: 1
      INFO : RACK_LOCAL_TASKS: 8
      INFO : AM_CPU_MILLISECONDS: 21600
      INFO : AM_GC_TIME_MILLIS: 40
      INFO : File System Counters:
      INFO : FILE_BYTES_READ: 19368
      INFO : OBS_BYTES_READ: 4338648131
      INFO : OBS_BYTES_WRITTEN: 4377004106
      INFO : OBS_READ_OPS: 4503
      INFO : OBS_WRITE_OPS: 59
      INFO : org.apache.tez.common.counters.TaskCounter:
      INFO : GC_TIME_MILLIS: 1334
      INFO : TASK_DURATION_MILLIS: 1030360
      INFO : CPU_MILLISECONDS: 162060
      INFO : PHYSICAL_MEMORY_BYTES: 61851303936
      INFO : VIRTUAL_MEMORY_BYTES: 82899070976
      INFO : COMMITTED_HEAP_BYTES: 61851303936
      INFO : INPUT_RECORDS_PROCESSED: 897
      INFO : INPUT_SPLIT_LENGTH_BYTES: 4310433482
      INFO : OUTPUT_RECORDS: 0
      INFO : HIVE:
      INFO : RECORDS_OUT_INTERMEDIATE_File_Merge: 0
      INFO : RECORDS_OUT_OPERATOR_OFM_57: 0
      INFO : INPUT_RECORDS_PROCESSED: 897
      INFO : INPUT_SPLIT_LENGTH_BYTES: 4310433482
      INFO : TaskCounter_File_Merge_OUTPUT_out_File_Merge:
      INFO : OUTPUT_RECORDS: 0
      ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=File Merge, vertexId=vertex_1590738759875_100602_2_00, diagnostics=[Task failed, taskId=task_1590738759875_100602_2_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_0:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000002_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000002_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_1:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000009_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000009_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_2:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000004_1 to xxxl/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000004_1
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_3:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000006_2 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000006_2
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1590738759875_100602_2_00 [File Merge] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
      INFO : Completed executing command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); Time taken: 4243.011 seconds

       

      INFO : Concurrency mode is disabled, not creating a lock manager
      Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=File Merge, vertexId=vertex_1590738759875_100602_2_00, diagnostics=[Task failed, taskId=task_1590738759875_100602_2_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_0:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000002_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000002_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_1:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000009_0 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000009_0
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_2:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000004_1 to xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000004_1
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : attempt_1590738759875_100602_2_00_000007_3:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
      at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
      at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
      at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
      ... 16 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
      at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
      at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
      at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
      ... 17 more
      Caused by: java.io.IOException: Unable to rename xxx/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_task_tmp.-ext-10002/_tmp.000006_2 to xxxl/warehouse/tablespace/hive/temp.db/.hive-staging_hive_2020-06-09_03-33-29_060_8741275828320666826-9431/_tmp.-ext-10002/000006_2
      at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
      ... 20 more
      ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1590738759875_100602_2_00 [File Merge] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)
      Closing: 0: jdbc:hive2://xxxxx:10000/prod

       

       

       

       

       

       

       

      Attachments

        Activity

          People

            wanguangping wanguangping
            wanguangping wanguangping
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: