Hadoop Common
  1. Hadoop Common
  2. HADOOP-5130

TaskTracker seems to hold onto the assigned task for a long while before launching it

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I saw atleast a couple of instances where the task assigned to the TaskTracker is launched several minutes after the receipt of the LaunchTaskAction:

      2009-01-27 13:55:53,402 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_200901270818_0006_m_000602_0
      2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_200901270818_0006_m_000602_0
      2009-01-27 13:55:55,129 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_200901270818_0006_m_000602_0
      2009-01-27 14:04:17,744 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200901270818_0006_m_176495965 given task: attempt_200901270818_0006_m_000602_0
      2009-01-27 14:04:24,020 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0% 
      2009-01-27 14:04:27,023 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0% 
      2009-01-27 14:04:30,026 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0% 
      2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901270818_0006_m_000602_0 1.0% 
      2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200901270818_0006_m_000602_0 is done.
      2009-01-27 14:04:30,362 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_200901270818_0006_m_000602_0  was 0
      

        Issue Links

          Activity

          Arun C Murthy created issue -
          Hide
          Amareshwari Sriramadasu added a comment -

          I also saw this behavior many times.
          And there are cases that these tasks got killed for not reporting progress for 10 mins.

          Show
          Amareshwari Sriramadasu added a comment - I also saw this behavior many times. And there are cases that these tasks got killed for not reporting progress for 10 mins.
          Hide
          Arun C Murthy added a comment -

          To clarify - the task did eventually launch and succeed, just the latency is too high (order of several minutes).

          Show
          Arun C Murthy added a comment - To clarify - the task did eventually launch and succeed, just the latency is too high (order of several minutes).
          Hide
          Arun C Murthy added a comment -

          Promoting this to a blocker for 0.20.0.

          Show
          Arun C Murthy added a comment - Promoting this to a blocker for 0.20.0.
          Arun C Murthy made changes -
          Field Original Value New Value
          Fix Version/s 0.20.0 [ 12313438 ]
          Priority Critical [ 2 ] Blocker [ 1 ]
          Hide
          Devaraj Das added a comment -

          Arun confirmed that this problem was due to the IPV4/V6 issue. This is discussed here - http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200812.mbox/%3CC56ED656.C28C%25runping@yahoo-inc.com%3E
          Adding -Djava.net.preferIPv4Stack=true to mapred.child.java.opts fixes the problem.

          Show
          Devaraj Das added a comment - Arun confirmed that this problem was due to the IPV4/V6 issue. This is discussed here - http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200812.mbox/%3CC56ED656.C28C%25runping@yahoo-inc.com%3E Adding -Djava.net.preferIPv4Stack=true to mapred.child.java.opts fixes the problem.
          Devaraj Das made changes -
          Resolution Won't Fix [ 2 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Nigel Daley made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Amareshwari Sriramadasu made changes -
          Link This issue relates to MAPREDUCE-1617 [ MAPREDUCE-1617 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development