Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4136

Hadoop streaming might succeed even through reducer fails

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: contrib/streaming
    • Labels:
      None

      Description

      Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls PipeReducer.close(), but in the mean time the reducer has failed and the process has died. When clientOut_.flush() throws an IOException in PipeMapRed.mapRedFinish() this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

      I've attached a patch that seems to fix it for us.

        Activity

        Wouter de Bie made changes -
        Description Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        I've attached a patch that seems to fix it for us.
        Wouter de Bie made changes -
        Attachment mapreduce-4136.patch [ 12522230 ]
        Wouter de Bie made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Wouter de Bie made changes -
        Field Original Value New Value
        Description Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Wouter de Bie created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Wouter de Bie
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development