[MAPREDUCE-3992] Reduce fetcher doesn't verify HTTP status code of response

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.1, 1.0.1, 2.0.0-alpha
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: mrv1
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 09/Mar/12 08:42
• Updated: 10/Mar/15 04:32
• Resolved: 24/Mar/12 23:38

Description

Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation:

• the map output servlet gets an IOException after setting the headers but before the first call to flush()
• this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed)
• it will still include the response headers indicating it's a valid response

In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet.