[IMPALA-9728] Data load failed with EOFException writing functional_orc_def.complextypestbl_medium

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Infrastructure
• Labels: flaky
• Epic Color: ghx-label-13
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 06/May/20 16:18
• Updated: 17/Nov/20 20:02
• Resolved: 17/Nov/20 20:02

Description

INFO  : Compiling command(queryId=ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707): INSERT OVERWRITE TABLE functional_orc_def.complextypestbl_medium SELECT c.* FROM functional_parquet.complextypestbl c join functional.alltypes sort by id
INFO  : Warning: Map Join MAPJOIN[9][bigTable=alltypes] in task 'Map 2' is a cross product
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:c.id, type:bigint, comment:null), FieldSchema(name:c.int_array, type:array<int>, comment:null), FieldSchema(name:c.int_array_array, type:array<array<int>>, comment:null), FieldSchema(name:c.int_map, type:map<string,int>, comment:null), FieldSchema(name:c.int_map_array, type:array<map<string,int>>, comment:null), FieldSchema(name:c.nested_struct, type:struct<a:int,b:array<int>,c:struct<d:array<array<struct<e:int,f:string>>>>,g:map<string,struct<h:struct<i:array<double>>>>>, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707); Time taken: 0.063 seconds
INFO  : Executing command(queryId=ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707): INSERT OVERWRITE TABLE functional_orc_def.complextypestbl_medium SELECT c.* FROM functional_parquet.complextypestbl c join functional.alltypes sort by id
INFO  : Query ID = ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Subscribed to counters: [] for queryId: ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707
INFO  : Session is already open
INFO  : Dag name: INSERT OVERWRITE TABLE functional_orc_d...id (Stage-1)
INFO  : Setting tez.task.scale.memory.reserve-fraction to 0.30000001192092896
INFO  : Status: Running (Executing on YARN cluster with App id application_1588725973781_0033)
...

Getting log thread is interrupted, since query is done!
ERROR : Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(java.io.EOFException)'
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1470)
	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:798)
	at org.apache.hadoop.hive.ql.exec.Operator.jobClose(Operator.java:803)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.close(TezTask.java:620)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:335)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.hadoop.hive.ql.exec.Utilities.handleDirectInsertTableFinalPath(Utilities.java:4448)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.jobCloseOp(FileSinkOperator.java:1464)
	... 28 more

ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
INFO  : Completed executing command(queryId=ubuntu_20200506012349_3c5cedc8-49d6-4e72-b4a5-e06cb82d1707); Time taken: 38.896 seconds
INFO  : OK
Error: Error while compiling statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=3)
java.sql.SQLException: Error while compiling statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
	at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
	at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007)
	at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
	at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
	at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
	at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
	at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127)
	at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
	at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
	at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Closing: 0: jdbc:hive2://localhost:11050/default;auth=none

Filing for future reference, I don't have immediate plans to investigate.

Issue Links

• duplicates: IMPALA-10247 Data loading of functional-query ORC fails with EOFException