Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2141

speculative execution start up condition based on completion time

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We had one job with speculative execution hang.

      4 reduce tasks were stuck with 95% completion because of a bad disk.

      Devaraj pointed out
      bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.

      It would be nice if speculative execution also starts up when tasks stop making progress.

      Devaraj suggested

      Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check.

        Attachments

        1. hadoop-2141-yahoo-v1.4.8.patch
          25 kB
          Amar Kamat
        2. hadoop-2141-yahoo-v1.4.1.patch
          22 kB
          Amar Kamat
        3. HADOOP-2141-v6.patch
          34 kB
          Andy Konwinski
        4. HADOOP-2141-v5.patch
          31 kB
          Andy Konwinski
        5. HADOOP-2141-v4.patch
          30 kB
          Andy Konwinski
        6. HADOOP-2141-v3.patch
          30 kB
          Andy Konwinski
        7. HADOOP-2141-v2.patch
          30 kB
          Andy Konwinski
        8. HADOOP-2141.v8.patch
          35 kB
          Andy Konwinski
        9. HADOOP-2141.v7.patch
          34 kB
          Andy Konwinski
        10. HADOOP-2141.patch
          28 kB
          Andy Konwinski
        11. 2141.patch
          32 kB
          Devaraj Das
        12. 2141.8.3.patch
          80 kB
          Devaraj Das
        13. 2141.8.2.patch
          80 kB
          Devaraj Das
        14. 2141.7.patch
          76 kB
          Devaraj Das
        15. 2141.6.patch
          76 kB
          Devaraj Das
        16. 2141.5.patch
          75 kB
          Devaraj Das
        17. 2141.4.patch
          69 kB
          Devaraj Das

          Issue Links

            Activity

              People

              • Assignee:
                andyk Andy Konwinski
                Reporter:
                knoguchi Koji Noguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: