[HIVE-8440] Hiveserver block query

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.12.0
• Component/s: Query Processor
• Labels: Hive0.12

Environment

OS:Red Hat Enterprise Linux Server release 6.5 (Santiago)
CPU:64 cores
Memory:64GB
Hadoop:CDH4.2.1

• Tags: Hiveserver block query
• Assignee: Unassigned
• Reporter: zengyongping
• Created: 13/Oct/14 09:08
• Updated: 09/Dec/14 10:57
• Resolved: 09/Dec/14 10:57

Description

In our environment Hiveserver handle approximately 15000 jobs per day。Sometimes we found a large number of ETL tasks wait in our ETL schedule queue to run。In this situation only few Hadoop Job running in Hadoop cluster,many maps and reduces slots free.We view Hiveserver heap found Hiveserver block a large number of query:
"pool-1-thread-17903" prio=10 tid=0x00007f89589c8000 nid=0x13865 waiting for monitor entry [0x00007f893413b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)

• waiting to lock <0x00000006000ac458> (a java.lang.Object)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
  at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
  at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
  at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
  at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

It is a problem of HiveServer cannot handle concurrent requests from more than one client.?Why HiveServer cannot handle concurrent requests?
https://github.com/apache/hive/blob/branch-0.12/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L975
Under code in org.apache.hadoop.hive.ql cause Hiveserver cannot handle concurrent requests?
int ret;
synchronized (compileMonitor)

{ ret = compile(command); }

Issue Links

• duplicates: HIVE-4239 Remove lock on compilation stage
• relates to: HIVE-4239 Remove lock on compilation stage