[HIVE-3408] A race condition is caused within QueryPlan class

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.7.1
• Component/s: Query Processor
• Labels: Concurrency

Environment

• Java 1.6.0_21, Java HotSpot(TM) 64-Bit Server VM
• Ubuntu 10.04

• Assignee: Unassigned
• Reporter: Muga Nishizawa
• Created: 27/Aug/12 07:56
• Updated: 02/Apr/13 01:40
• Resolved: 02/Apr/13 01:40

Description

Hive's threads are stopped at HashMap.getEntry(..) that is used within QueryPlan#extractCounters() and QueryPlan#updateCountersInQueryPlan(). It seems that a race condition problem of a HashSet object is caused when extractCounters() and updateCountersInQueryPlan() are concurrently executed. I hit the problem with Hive 0.7.1 but, I think that it also is caused with 0.8.1.

The problem is reported by several persons on mailing list.
http://mail-archives.apache.org/mod_mbox/hive-dev/201201.mbox/%3CCAKTRiE+3x31FDy+3F0c+jZSXQrhxBgT4DOyfZddm7sdX+cu=[e-mail]%3E

http://mail-archives.apache.org/mod_mbox/hive-user/201202.mbox/[e-mail]%3E

The following is a part of my thread dump.

"Thread-1091" prio=10 tid=0x00007fd17112b000 nid=0x1100 runnable [0x00007fd175f60000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at java.util.HashSet.contains(HashSet.java:184)
at org.apache.hadoop.hive.ql.QueryPlan.extractCounters(QueryPlan.java:342)
at org.apache.hadoop.hive.ql.QueryPlan.getQueryPlan(QueryPlan.java:419)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:592)
at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:493)
at org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:395)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:686)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)

"Thread-1090" prio=10 tid=0x00007fd17012f000 nid=0x10ff runnable [0x00007fd175152000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at java.util.HashSet.contains(HashSet.java:184)
at org.apache.hadoop.hive.ql.QueryPlan.updateCountersInQueryPlan(QueryPlan.java:297)
at org.apache.hadoop.hive.ql.QueryPlan.getQueryPlan(QueryPlan.java:420)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:592)
at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:493)
at org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:395)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:686)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)

Issue Links

• duplicates: HIVE-4000 Hive client goes into infinite loop at 100% cpu