Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1158

JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.12.2
    • 0.15.0
    • None
    • None

    Description

      The JobTracker should keep a track (with feedback from Reducers) of how many times a fetch for a particular map output failed. If this exceeds a certain threshold, then that map should be declared as lost, and should be reexecuted elsewhere. Based on the number of such complaints from Reducers, the JobTracker can blacklist the TaskTracker. This will make the framework reliable - it will take care of (faulty) TaskTrackers that sometimes always fail to serve up map outputs (for which exceptions are not properly raised/handled, for e.g., if the exception/problem happens in the Jetty server).

      Attachments

        1. HADOOP-1158_2_20070808.patch
          50 kB
          Arun Murthy
        2. HADOOP-1158_20070702_1.patch
          48 kB
          Arun Murthy
        3. HADOOP-1158_3_20070809.patch
          51 kB
          Arun Murthy
        4. HADOOP-1158_4_20070817.patch
          51 kB
          Arun Murthy
        5. HADOOP-1158_5_20070823.patch
          51 kB
          Arun Murthy

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            acmurthy Arun Murthy
            ddas Devaraj Das
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment