Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2524

Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff.

      Description

      The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.

      1. MAPREDUCE2524-20security-v2.patch
        14 kB
        Thomas Graves
      2. MAPREDUCE2524-patch-20security.txt
        14 kB
        Thomas Graves
      3. MAPREDUCE2524-patch-20security.txt
        14 kB
        Thomas Graves

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development