Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-339

JobTracker should give preference to failed tasks over virgin tasks so as to terminate the job ASAP if it is eventually going to fail.

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Case in point... I have 1585 maps and 160 slots (40 nodes). The job is such that all maps fail within 2-3 minutes. The job takes forever to realise that the job is bad. It took 2526 failures for it to reach 4 failed attempts for a task.

      As I understand, currently the JT prefers a failed task if and only if a task tracker with a split replica for that map came asking for a task. In fact there may not be a single TT at all in the mapred cluster which has a replica for the splits used in this job (pre-0.20). This delays the job failure by a lot and hence degrades cluster utilization as a whole. If i'm on a shared cluster with many jobs waiting on it to fail, it's bad.

      The JT should prefer a failed task a lot earlier than waiting for a data local TT to come around asking.

        Attachments

        1. MAPREDUCE-339-v1.2.patch
          10 kB
          Amar Kamat
        2. M339-0y20s.patch
          16 kB
          Chris Douglas

          Activity

            People

            • Assignee:
              devaraj Devaraj Das
              Reporter:
              gautamk Gautam Kowshik
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: