Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-339

JobTracker should give preference to failed tasks over virgin tasks so as to terminate the job ASAP if it is eventually going to fail.

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Case in point... I have 1585 maps and 160 slots (40 nodes). The job is such that all maps fail within 2-3 minutes. The job takes forever to realise that the job is bad. It took 2526 failures for it to reach 4 failed attempts for a task.

      As I understand, currently the JT prefers a failed task if and only if a task tracker with a split replica for that map came asking for a task. In fact there may not be a single TT at all in the mapred cluster which has a replica for the splits used in this job (pre-0.20). This delays the job failure by a lot and hence degrades cluster utilization as a whole. If i'm on a shared cluster with many jobs waiting on it to fail, it's bad.

      The JT should prefer a failed task a lot earlier than waiting for a data local TT to come around asking.

      Attachments

        1. MAPREDUCE-339-v1.2.patch
          10 kB
          Amar Kamat
        2. M339-0y20s.patch
          16 kB
          Christopher Douglas

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ddas Devaraj Das
            gautamk Gautam Kowshik

            Dates

              Created:
              Updated:

              Slack

                Issue deployment