Hadoop Common
  1. Hadoop Common
  2. HADOOP-2141

speculative execution start up condition based on completion time

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We had one job with speculative execution hang.

      4 reduce tasks were stuck with 95% completion because of a bad disk.

      Devaraj pointed out
      bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.

      It would be nice if speculative execution also starts up when tasks stop making progress.

      Devaraj suggested

      Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check.

      1. hadoop-2141-yahoo-v1.4.8.patch
        25 kB
        Amar Kamat
      2. hadoop-2141-yahoo-v1.4.1.patch
        22 kB
        Amar Kamat
      3. 2141.8.3.patch
        80 kB
        Devaraj Das
      4. 2141.8.2.patch
        80 kB
        Devaraj Das
      5. 2141.7.patch
        76 kB
        Devaraj Das
      6. 2141.6.patch
        76 kB
        Devaraj Das
      7. 2141.5.patch
        75 kB
        Devaraj Das
      8. 2141.4.patch
        69 kB
        Devaraj Das
      9. HADOOP-2141.v8.patch
        35 kB
        Andy Konwinski
      10. HADOOP-2141.v7.patch
        34 kB
        Andy Konwinski
      11. HADOOP-2141-v6.patch
        34 kB
        Andy Konwinski
      12. HADOOP-2141-v5.patch
        31 kB
        Andy Konwinski
      13. 2141.patch
        32 kB
        Devaraj Das
      14. HADOOP-2141-v4.patch
        30 kB
        Andy Konwinski
      15. HADOOP-2141-v3.patch
        30 kB
        Andy Konwinski
      16. HADOOP-2141-v2.patch
        30 kB
        Andy Konwinski
      17. HADOOP-2141.patch
        28 kB
        Andy Konwinski

        Issue Links

          Activity

          Jeff Hammerbacher made changes -
          Link This issue relates to MAPREDUCE-2039 [ MAPREDUCE-2039 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Amar Kamat made changes -
          Attachment hadoop-2141-yahoo-v1.4.8.patch [ 12435253 ]
          Amar Kamat made changes -
          Attachment hadoop-2141-yahoo-v1.4.1.patch [ 12431756 ]
          Robert Chansler made changes -
          Release Note Improves the speculative execution heuristic. The heuristic is currently based on the progress-rates of tasks and the expected time of completion. Also, statistics about trackers are collected, and speculative tasks are not given to the ones deduced to be slow.
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note Updated speculative execution scheduler Improves the speculative execution heuristic. The heuristic is currently based on the progress-rates of tasks and the expected time of completion. Also, statistics about trackers are collected, and speculative tasks are not given to the ones deduced to be slow.
          Resolution Fixed [ 1 ]
          Devaraj Das made changes -
          Attachment 2141.8.3.patch [ 12410655 ]
          Devaraj Das made changes -
          Attachment 2141.8.2.patch [ 12410621 ]
          Devaraj Das made changes -
          Attachment 2141.7.patch [ 12410480 ]
          Devaraj Das made changes -
          Attachment 2141.6.patch [ 12410463 ]
          Devaraj Das made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Devaraj Das made changes -
          Attachment 2141.5.patch [ 12410208 ]
          Devaraj Das made changes -
          Attachment 2141.4.patch [ 12409166 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141.v8.patch [ 12408228 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141.v7.patch [ 12407448 ]
          Andy Konwinski made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.21.0 [ 12313563 ]
          Affects Version/s 0.19.0 [ 12313211 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141-v6.patch [ 12403769 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141-v5.patch [ 12400597 ]
          Devaraj Das made changes -
          Attachment 2141.patch [ 12400261 ]
          Nigel Daley made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141-v4.patch [ 12395077 ]
          Arun C Murthy made changes -
          Affects Version/s 0.20.0 [ 12313438 ]
          Affects Version/s 0.19.0 [ 12313211 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Fix Version/s 0.19.0 [ 12313211 ]
          Andy Konwinski made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.20.0 [ 12313438 ]
          Fix Version/s 0.19.0 [ 12313211 ]
          Release Note Updated speculative execution scheduler
          Affects Version/s 0.19.0 [ 12313211 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141-v3.patch [ 12392157 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141-v2.patch [ 12389273 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Andy Konwinski made changes -
          Affects Version/s 0.19.0 [ 12313211 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.15.0 [ 12312565 ]
          Andy Konwinski made changes -
          Attachment HADOOP-2141.patch [ 12388788 ]
          Andy Konwinski made changes -
          Assignee Arun C Murthy [ acmurthy ] Andy Konwinski [ andyk ]
          Runping Qi made changes -
          Link This issue relates to HADOOP-2177 [ HADOOP-2177 ]
          Runping Qi made changes -
          Link This issue relates to HADOOP-2933 [ HADOOP-2933 ]
          Robert Chansler made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Arun C Murthy made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Fix Version/s 0.16.0 [ 12312740 ]
          Owen O'Malley made changes -
          Link This issue is related to HADOOP-2211 [ HADOOP-2211 ]
          Owen O'Malley made changes -
          Fix Version/s 0.15.1 [ 12312850 ]
          Priority Blocker [ 1 ] Major [ 3 ]
          Fix Version/s 0.16.0 [ 12312740 ]
          Devaraj Das made changes -
          Assignee Arun C Murthy [ acmurthy ]
          Milind Bhandarkar made changes -
          Field Original Value New Value
          Affects Version/s 0.14.3 [ 12312830 ]
          Fix Version/s 0.15.1 [ 12312850 ]
          Affects Version/s 0.15.0 [ 12312565 ]
          Priority Major [ 3 ] Blocker [ 1 ]
          Fix Version/s 0.16.0 [ 12312740 ]
          Koji Noguchi created issue -

            People

            • Assignee:
              Andy Konwinski
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development