[MAPREDUCE-3851] Allow more aggressive action on detection of the jetty issue

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.2
• Component/s: tasktracker
• Target Version/s: 1.0.2

Release Note

added new configuration variables to control when TT aborts if it sees a certain number of exceptions:

    // Percent of shuffle exceptions (out of sample size) seen before it's
    // fatal - acceptable values are from 0 to 1.0, 0 disables the check.
    // ie. 0.3 = 30% of the last X number of requests matched the exception,
    // so abort.
      conf.getFloat(
          "mapreduce.reduce.shuffle.catch.exception.percent.limit.fatal", 0);

    // The number of trailing requests we track, used for the fatal
    // limit calculation
      conf.getInt("mapreduce.reduce.shuffle.catch.exception.sample.size", 1000);

• Assignee: Thomas Graves
• Reporter: Kihwal Lee
• Created: 10/Feb/12 22:16
• Updated: 31/Aug/12 13:20
• Resolved: 18/Mar/12 03:27

Description

MAPREDUCE-2529 added the useful failure detection mechanism. In this jira, I propose we add a periodic check inside TT and configurable action to self-destruct. Blacklisting helps but is not enough. Hung jetty still accepts connection and it takes very long time for clients to fail out. Short jobs are delayed for hours because of this. This feature will be a nice companion to MAPREDUCE-3184.