Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2524

Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff.

      Description

      The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.

      1. MAPREDUCE2524-20security-v2.patch
        14 kB
        Thomas Graves
      2. MAPREDUCE2524-patch-20security.txt
        14 kB
        Thomas Graves
      3. MAPREDUCE2524-patch-20security.txt
        14 kB
        Thomas Graves

        Activity

        Hide
        Owen O'Malley added a comment -

        Hadoop 0.20.204.0 was just released.

        Show
        Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Tom!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Tom!
        Hide
        Thomas Graves added a comment -

        Note, this failed because patch is for branch-0.20-security

        Show
        Thomas Graves added a comment - Note, this failed because patch is for branch-0.20-security
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480599/MAPREDUCE2524-20security-v2.patch
        against trunk revision 1127444.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/313//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480599/MAPREDUCE2524-20security-v2.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/313//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        upload new patch with changes in mapred-default.xml

        Show
        Thomas Graves added a comment - upload new patch with changes in mapred-default.xml
        Hide
        Thomas Graves added a comment -

        reattaching wit license granted.

        Show
        Thomas Graves added a comment - reattaching wit license granted.
        Hide
        Thomas Graves added a comment -

        The javadoc and eclipse failures existed before/without these changes.

        [exec] -1 overall. [exec]
        [exec] +1 @author. The patch does not contain any @author tags. [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec]

        Show
        Thomas Graves added a comment - The javadoc and eclipse failures existed before/without these changes. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec]
        Hide
        Thomas Graves added a comment -

        patch for the branch-0.20-security.

        Show
        Thomas Graves added a comment - patch for the branch-0.20-security.

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development