Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6143

add configuration for mapreduce speculative execution in MR2

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.1
    • Fix Version/s: 2.7.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      add configuration for mapreduce speculative execution in MR2.
      Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more.

      We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution:

      private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L;
      private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L;
      private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1;
      private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01;
      private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10;

        Attachments

        1. MAPREDUCE-6143.002.patch
          14 kB
          Zhihai Xu
        2. MAPREDUCE-6143.001.patch
          14 kB
          Zhihai Xu
        3. MAPREDUCE-6143.000.patch
          13 kB
          Zhihai Xu

          Activity

            People

            • Assignee:
              zxu Zhihai Xu
              Reporter:
              zxu Zhihai Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: