[HADOOP-969] deadlock in job tracker RetireJobs

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Assignee: Owen O'Malley
• Reporter: Owen O'Malley
• Created: 02/Feb/07 17:58
• Updated: 08/Jul/09 16:52
• Resolved: 02/Feb/07 20:08

Description

The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:

"IPC Server handler 5 on 50020":
at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)

• waiting to lock <0x74487a80> (a java.util.Vector)
• locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
  at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
• locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:585)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
  "retireJobs":
  at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
• waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
  at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
  at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
• locked <0x74487bb0> (a java.util.ArrayList)
• locked <0x74487a80> (a java.util.Vector)
• locked <0x74487a58> (a java.util.TreeMap)
  at java.lang.Thread.run(Thread.java:595)

Found 1 deadlock.