Hadoop Common
  1. Hadoop Common
  2. HADOOP-1158

JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.2
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      The JobTracker should keep a track (with feedback from Reducers) of how many times a fetch for a particular map output failed. If this exceeds a certain threshold, then that map should be declared as lost, and should be reexecuted elsewhere. Based on the number of such complaints from Reducers, the JobTracker can blacklist the TaskTracker. This will make the framework reliable - it will take care of (faulty) TaskTrackers that sometimes always fail to serve up map outputs (for which exceptions are not properly raised/handled, for e.g., if the exception/problem happens in the Jetty server).

      1. HADOOP-1158_2_20070808.patch
        50 kB
        Arun C Murthy
      2. HADOOP-1158_20070702_1.patch
        48 kB
        Arun C Murthy
      3. HADOOP-1158_3_20070809.patch
        51 kB
        Arun C Murthy
      4. HADOOP-1158_4_20070817.patch
        51 kB
        Arun C Murthy
      5. HADOOP-1158_5_20070823.patch
        51 kB
        Arun C Murthy

        Issue Links

          Activity

          Devaraj Das created issue -
          Arun C Murthy made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-1246 [ HADOOP-1246 ]
          Devaraj Das made changes -
          Link This issue duplicates HADOOP-1454 [ HADOOP-1454 ]
          Arun C Murthy made changes -
          Assignee Arun C Murthy [ acmurthy ]
          Arun C Murthy made changes -
          Attachment HADOOP-1158_20070702_1.patch [ 12360935 ]
          Arun C Murthy made changes -
          Attachment HADOOP-1158_2_20070808.patch [ 12363431 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.15.0 [ 12312565 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment HADOOP-1158_3_20070809.patch [ 12363458 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment HADOOP-1158_4_20070817.patch [ 12363993 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment HADOOP-1158_5_20070823.patch [ 12364381 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development