[HADOOP-1970] tasktracker hang in reduce. Deadlock between main and comm thread

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.14.1
• Fix Version/s: 0.14.2
• Assignee: Vivek Ratan
• Reporter: Koji Noguchi
• Created: 29/Sep/07 07:07
• Updated: 08/Jul/09 16:52
• Resolved: 02/Oct/07 23:02

Description

Saw one reduce task stuck on copy.
jstack on the reduce task(task_200709272248_0001_r_000150_0) process showed

 
Found one Java-level deadlock:
=============================
"Comm thread for task_200709272248_0001_r_000150_0":
  waiting to lock monitor 0x08144020 (object 0xd4e30aa8, a org.apache.hadoop.util.Progress),
  which is held by "main"
"main":
  waiting to lock monitor 0x08144084 (object 0xd4e30958, a org.apache.hadoop.util.Progress),
  which is held by "Comm thread for task_200709272248_0001_r_000150_0"

Java stack information for the threads listed above:
===================================================
"Comm thread for task_200709272248_0001_r_000150_0":
        at org.apache.hadoop.util.Progress.toString(Progress.java:113)
        - waiting to lock <0xd4e30aa8> (a org.apache.hadoop.util.Progress)
        at org.apache.hadoop.util.Progress.toString(Progress.java:116)
        - locked <0xd4e30958> (a org.apache.hadoop.util.Progress)
        at org.apache.hadoop.util.Progress.toString(Progress.java:108)
        at org.apache.hadoop.mapred.Task$1.run(Task.java:268)
        at java.lang.Thread.run(Thread.java:619)
"main":
        at org.apache.hadoop.util.Progress.startNextPhase(Progress.java:58)
        - waiting to lock <0xd4e30958> (a org.apache.hadoop.util.Progress)
        at org.apache.hadoop.util.Progress.complete(Progress.java:70)
        - locked <0xd4e30aa8> (a org.apache.hadoop.util.Progress)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:253)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)