Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3992

Reduce fetcher doesn't verify HTTP status code of response

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 1.0.1, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv1
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation:

      • the map output servlet gets an IOException after setting the headers but before the first call to flush()
      • this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed)
      • it will still include the response headers indicating it's a valid response

      In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet.

      1. mr-3992-branch-1.txt
        1 kB
        Todd Lipcon
      2. mr-3992.txt
        2 kB
        Todd Lipcon

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development