Hadoop Common
  1. Hadoop Common
  2. HADOOP-597

transmission errors to the reduce will cause map output to be considered lost

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.1
    • Component/s: None
    • Labels:
      None

      Description

      When the mapOutput servlet gets an IOException from reading the map output file, it correctly declares the map output lost. However, an IOException writing to the socket should NOT cause the map output to be declared lost. Unfortunately, the current implementation always declares the map output lost. When the map output is lost, the map is re-run on a different computer.

      1. map-output-protect.patch
        2 kB
        Owen O'Malley
      2. map-output-protect-2.patch
        2 kB
        Owen O'Malley

        Activity

        Hide
        Owen O'Malley added a comment -

        Here is a patch that fixes the problem. I use a flag to mark that the fault was on the input and therefore the map output should be declared lost. I'm not really that happy with all of the catches that rethrow, but I didn't see a more straightforward way of getting the right effect.

        Show
        Owen O'Malley added a comment - Here is a patch that fixes the problem. I use a flag to mark that the fault was on the input and therefore the map output should be declared lost. I'm not really that happy with all of the catches that rethrow, but I didn't see a more straightforward way of getting the right effect.
        Hide
        Doug Cutting added a comment -

        Half the number of try/catch blocks would be required if you instead wrapped the outStream operations and use a badOutput boolean, no?

        Show
        Doug Cutting added a comment - Half the number of try/catch blocks would be required if you instead wrapped the outStream operations and use a badOutput boolean, no?
        Hide
        Owen O'Malley added a comment -

        Ok, as suggested by Doug, I inverted the check and wrapped the socket.write call with a specialized check. Thanks!

        Show
        Owen O'Malley added a comment - Ok, as suggested by Doug, I inverted the check and wrapped the socket.write call with a specialized check. Thanks!
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Owen.

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Owen.

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development