[MAPREDUCE-5388] There are two TaskLaunchers for map after the TaskTracker reinit

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.21.0, 0.22.0, 1.0.3
• Component/s: tasktracker
• Assignee: Unassigned
• Reporter: Hua xu
• Created: 15/Jul/13 06:33
• Updated: 15/Jul/13 06:49

Description

The TaskTracker was asked to reinit as the full gc,then the process contained of two TaskLauncher instance for map:

2013-06-22 08:32:59,457 INFO org.apache.hadoop.mapred.TaskTracker: Process Thread Dump: lost task
36 active threads

......

Thread 43 (TaskLauncher for REDUCE tasks):
  State: WAITING
  Blocked count: 73824
  Waited count: 72848
  Waiting on java.util.LinkedList@98a9557
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2157)
Thread 42 (TaskLauncher for MAP tasks):
  State: WAITING
  Blocked count: 358814
  Waited count: 356105
  Waiting on org.apache.hadoop.ipc.Client$Call@6f764bc1
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.ipc.Client.call(Client.java:891)
    org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
    $Proxy5.getFileInfo(Unknown Source)
    sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    $Proxy5.getFileInfo(Unknown Source)
    org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:850)
    org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:620)
    org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
    org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:167)
    org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1543)
    org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1524)
    org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:3995)
    org.apache.hadoop.mapred.TaskTracker.localizeJobFiles(TaskTracker.java:1036)
    org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:977)

......

Thread 1 (main):
  State: RUNNABLE
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:157)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:211)
    org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:1792)
    org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1468)
    org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2329)
    org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3461)