Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-969

deadlock in job tracker RetireJobs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:

      "IPC Server handler 5 on 50020":
      at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)

      • waiting to lock <0x74487a80> (a java.util.Vector)
      • locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
        at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
      • locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
        "retireJobs":
        at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
      • waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
        at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
        at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
      • locked <0x74487bb0> (a java.util.ArrayList)
      • locked <0x74487a80> (a java.util.Vector)
      • locked <0x74487a58> (a java.util.TreeMap)
        at java.lang.Thread.run(Thread.java:595)

      Found 1 deadlock.

        Attachments

        1. jt-dead.patch
          5 kB
          Owen O'Malley

          Activity

            People

            • Assignee:
              omalley Owen O'Malley
              Reporter:
              omalley Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: