Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3992

Reduce fetcher doesn't verify HTTP status code of response

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 1.0.1, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv1
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation:

      • the map output servlet gets an IOException after setting the headers but before the first call to flush()
      • this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed)
      • it will still include the response headers indicating it's a valid response

      In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet.

      1. mr-3992.txt
        2 kB
        Todd Lipcon
      2. mr-3992-branch-1.txt
        1 kB
        Todd Lipcon

        Activity

        Todd Lipcon created issue -
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment mr-3992.txt [ 12518085 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.23.1 [ 12318883 ]
        Affects Version/s 0.24.0 [ 12317654 ]
        Target Version/s 1.0.2 [ 12320047 ] 0.24.0, 1.0.2, 0.23.3 [ 12317654, 12320047, 12320060 ]
        Matt Foley made changes -
        Target Version/s 0.23.3, 1.0.2, 0.24.0 [ 12320060, 12320047, 12317654 ] 0.24.0, 0.23.3, 1.0.3 [ 12317654, 12320060, 12320250 ]
        Todd Lipcon made changes -
        Fix Version/s 0.24.0 [ 12317654 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Target Version/s 1.0.3, 0.23.3, 0.24.0 [ 12320250, 12320060, 12317654 ] 0.24.0, 0.23.3, 1.0.3 [ 12317654, 12320060, 12320250 ]
        Todd Lipcon made changes -
        Attachment mr-3992-branch-1.txt [ 12519011 ]
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Target Version/s 1.0.3, 0.23.3, 0.24.0 [ 12320250, 12320060, 12317654 ] 0.24.0, 0.23.3, 1.0.3 [ 12317654, 12320060, 12320250 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 1.0.3, 0.23.3, 0.24.0 [ 12320250, 12320060, 12317654 ] 0.24.0, 0.23.3, 1.0.3 [ 12317654, 12320060, 12320250 ]
        Fix Version/s 1.1.0 [ 12317960 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.0 [ 12320354 ]
        Fix Version/s 0.24.0 [ 12317654 ]
        Fix Version/s 1.1.0 [ 12317960 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Matt Foley made changes -
        Fix Version/s 1.1.0 [ 12317960 ]
        Target Version/s 1.0.3, 0.23.3, 0.24.0 [ 12320250, 12320060, 12317654 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robert Joseph Evans made changes -
        Fix Version/s 0.23.3 [ 12320060 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 1.1.0 [ 12317960 ]
        Fix Version/s 2.0.0-alpha [ 12320354 ]
        Suresh Srinivas made changes -
        Fix Version/s 1.1.0 [ 12317960 ]
        Leitao Guo made changes -
        Fix Version/s 1.1.0 [ 12317960 ]
        Allen Wittenauer made changes -
        Affects Version/s 2.0.0-alpha [ 12320354 ]
        Affects Version/s 0.24.0 [ 12317654 ]

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development