Hadoop Common
  1. Hadoop Common
  2. HADOOP-2119

JobTracker becomes non-responsive if the task trackers finish task too fast

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.
      Show
      This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.

      Description

      I ran a job with 0 reducer on a cluster with 390 nodes.
      The mappers ran very fast.
      The jobtracker lacks behind on committing completed mapper tasks.
      The number of running mappers displayed on web UI getting bigger and bigger.
      The jos tracker eventually stopped responding to web UI.

      No progress is reported afterwards.

      Job tracker is running on a separate node.
      The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).

      1. hadoop-2119.patch
        11 kB
        Srikanth Kakani
      2. HADOOP-2119-v4.1.patch
        52 kB
        Amar Kamat
      3. HADOOP-2119-v5.1.patch
        55 kB
        Amar Kamat
      4. HADOOP-2119-v5.1.patch
        52 kB
        Amar Kamat
      5. HADOOP-2119-v5.2.patch
        51 kB
        Amar Kamat
      6. hadoop-jobtracker-thread-dump.txt
        60 kB
        Christian Kunz

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development