Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3992

Reduce fetcher doesn't verify HTTP status code of response

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.1, 1.0.1, 2.0.0-alpha
    • 0.23.3, 2.0.2-alpha
    • mrv1
    • None
    • Reviewed

    Description

      Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation:

      • the map output servlet gets an IOException after setting the headers but before the first call to flush()
      • this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed)
      • it will still include the response headers indicating it's a valid response

      In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet.

      Attachments

        1. mr-3992-branch-1.txt
          1 kB
          Todd Lipcon
        2. mr-3992.txt
          2 kB
          Todd Lipcon

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment