[IMPALA-10247] Data loading of functional-query ORC fails with EOFException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-10
• Assignee: Zoltán Borók-Nagy
• Reporter: Quanlong Huang
• Created: 16/Oct/20 06:21
• Updated: 02/Feb/23 12:43
• Resolved: 02/Feb/23 12:43

Description

Data loading of functional-query on ORC tables occasionally fails with

16:41:21 Loading custom schemas (logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-custom-schemas.log)... 
16:41:24   Loading custom schemas OK (Took: 0 min 4 sec)
16:41:24 Started Loading functional-query data in background; pid 23644.
16:41:24 Started Loading TPC-H data in background; pid 23645.
16:41:24 Loading functional-query data (logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-functional-query.log)... 
16:41:24 Started Loading TPC-DS data in background; pid 23646.
16:41:24 Loading TPC-H data (logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-tpch.log)... 
16:41:24 Loading TPC-DS data (logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/data_loading/load-tpcds.log)... 
16:48:51   Loading workload 'tpch' using exploration strategy 'core' OK (Took: 7 min 27 sec)
16:50:53     FAILED (Took: 9 min 29 sec)
16:50:53     'load-data functional-query exhaustive' failed. Tail of log: 

This looks similar to IMPALA-9923 but have a different error stacktrace:

2020-10-13T16:50:50,369  INFO [HiveServer2-Background-Pool: Thread-23853] ql.Driver: Executing command(queryId=jenkins_20201013165050_5dc3d632-a5c3-4f85-b2d3-8c1dc6682322): INSERT OVERWRITE TABLE tpcds_orc_def.web_sales
SELECT * FROM tpcds.web_sales
......
2020-10-13T16:50:53,423  INFO [HiveServer2-Background-Pool: Thread-23832] FileOperations: Reading manifest hdfs://localhost:20500/test-warehouse/managed/jointbl_orc_def/_tmp.base_0000001_0/000000_0.manifest
2020-10-13T16:50:53,423  INFO [HiveServer2-Background-Pool: Thread-23832] FileOperations: Reading manifest hdfs://localhost:20500/test-warehouse/managed/jointbl_orc_def/_tmp.base_0000001_0/000000_1.manifest
2020-10-13T16:50:53,423  INFO [HiveServer2-Background-Pool: Thread-23832] FileOperations: Looking at manifest file: hdfs://localhost:20500/test-warehouse/managed/jointbl_orc_def/_tmp.base_0000001_0/000000_0.manifest
2020-10-13T16:50:53,424 ERROR [HiveServer2-Background-Pool: Thread-23832] exec.Task: Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(java.io.EOFException)'
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1468)
        at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:798)
        at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
        at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:627)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:342)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
        at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
        at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
        at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
        at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
        at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.hadoop.hive.ql.exec.Utilities.handleDirectInsertTableFinalPath(Utilities.java:4587)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1462)
        ... 29 more
 

The failed query is

INSERT OVERWRITE TABLE tpcds_orc_def.web_sales
SELECT * FROM tpcds.web_sales

 

Issue Links

• is caused by: HIVE-24322 In case of direct insert, the attempt ID has to be checked when reading the manifest files
• is duplicated by: IMPALA-9728 Data load failed with EOFException writing functional_orc_def.complextypestbl_medium