Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1903

Allow different slowTaskThreshold for mappers and reducers

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: jobtracker
    • Labels:
      None

      Description

      We have been running the new speculative logic in HADOOP-2141 done by Andy in our production cluster.
      One thing that we observed is that there are significantly low number of speculative reducers.
      We have seen usually 100 speculative mappers launched per minute but speculative reducers are usually less than 5.
      But reducers are usually where we get complains about having speculative issues.

      These two types of tasks has different properties and different needs.
      It would be nice if we can configure the slow threshold separately to deal with them separately.

      We can add the following config keys to allow setting them independently.
      MRJobConfig.SPECULATIVE_MAP_SLOWTASK_THRESHOLD
      MRJobConfig.SPECULATIVE_REDUCE_SLOWTASK_THRESHOLD

      Thoughts?

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development