Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.9.0, 0.12.0
-
None
-
Ubuntu LXC (hive-trunk), CDH 4 on Debian
-
Thread safety for JDBCStatsPublisher.init() for hive.exec.parallel=true job inits
Description
While running a hive query with multiple independent stages, hive.exec.parallel is a valid optimization to use.
The query tested has 3 MR jobs - the first job is the root dependency and the 2 further job depend on the first one.
When hive.exec.parallel is turned on, the job fails with the following exception
java.io.IOException: java.lang.InterruptedException at org.apache.hadoop.ipc.Client.call(Client.java:1214) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) at $Proxy12.mkdirs(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at $Proxy12.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916) at org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921) at org.apache.hadoop.ipc.Client.call(Client.java:1208)
The query plan is as follows
Stage-9 is a root stage Stage-8 depends on stages: Stage-9 Stage-3 depends on stages: Stage-8 Stage-0 depends on stages: Stage-3 Stage-4 depends on stages: Stage-0 Stage-5 depends on stages: Stage-8 Stage-1 depends on stages: Stage-5 Stage-6 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-9 Map Reduce Local Work Stage: Stage-8 Map Reduce Map Join Operator Stage: Stage-3 Map Reduce Stage: Stage-0 Move Operator Stage: Stage-4 Stats-Aggr Operator Stage: Stage-5 Map Reduce Stage: Stage-1 Move Operator Stage: Stage-6 Stats-Aggr Operator
I cannot conclude that this is purely a hive issue, will file a bug on HDFS if that does show up during triage.
Triaged - set hive.stats.autogather=false; removes the bug.
Attachments
Attachments
Issue Links
- is related to
-
HADOOP-6762 exception while doing RPC I/O closes channel
- Closed
- relates to
-
HADOOP-9612 HADOOP client IPC layer should differentiate between interrupted threads and interrupts occurring during operations
- Open