Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-339

JobTracker should give preference to failed tasks over virgin tasks so as to terminate the job ASAP if it is eventually going to fail.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Case in point... I have 1585 maps and 160 slots (40 nodes). The job is such that all maps fail within 2-3 minutes. The job takes forever to realise that the job is bad. It took 2526 failures for it to reach 4 failed attempts for a task.

      As I understand, currently the JT prefers a failed task if and only if a task tracker with a split replica for that map came asking for a task. In fact there may not be a single TT at all in the mapred cluster which has a replica for the splits used in this job (pre-0.20). This delays the job failure by a lot and hence degrades cluster utilization as a whole. If i'm on a shared cluster with many jobs waiting on it to fail, it's bad.

      The JT should prefer a failed task a lot earlier than waiting for a data local TT to come around asking.

      1. M339-0y20s.patch
        16 kB
        Chris Douglas
      2. MAPREDUCE-339-v1.2.patch
        10 kB
        Amar Kamat

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Gautam Kowshik
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development