[MAPREDUCE-2529] Recognize Jetty bug 1342 and handle it

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.204.0, 0.23.0
• Fix Version/s: 0.20.204.0, 0.23.0
• Component/s: tasktracker
• Hadoop Flags: Reviewed

Release Note

Added 2 new config parameters:

mapreduce.reduce.shuffle.catch.exception.stack.regex
mapreduce.reduce.shuffle.catch.exception.message.regex

• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 24/May/11 15:05
• Updated: 02/Sep/11 22:13
• Resolved: 05/Jun/11 05:23

Description

We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot of them throw exceptions and eventually fail. The only way we have found to get the TT out of this state is to restart the TT. This jira is to catch this particular exception (or perhaps a configurable regex) and handle it in an automated way to either blacklist or shutdown the TT after seeing it a configurable number of them.