Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4136

Hadoop streaming might succeed even through reducer fails

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: contrib/streaming
    • Labels:

      Description

      Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls PipeReducer.close(), but in the mean time the reducer has failed and the process has died. When clientOut_.flush() throws an IOException in PipeMapRed.mapRedFinish() this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

      I've attached a patch that seems to fix it for us.

        Activity

        Wouter de Bie created issue -
        Wouter de Bie made changes -
        Field Original Value New Value
        Description Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Wouter de Bie made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Wouter de Bie made changes -
        Attachment mapreduce-4136.patch [ 12522230 ]
        Wouter de Bie made changes -
        Description Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        We're working on a patch.
        Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

        I've attached a patch that seems to fix it for us.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522230/mapreduce-4136.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2196//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522230/mapreduce-4136.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2196//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        A similar fix was already done for MAPREDUCE-3790. I don't really know if this is the same fix or not, but the patch only went into trunk, and branch-0.23/branch-2. So it might be worth looking at to see if there is some overlap between the two.

        Show
        Robert Joseph Evans added a comment - A similar fix was already done for MAPREDUCE-3790 . I don't really know if this is the same fix or not, but the patch only went into trunk, and branch-0.23/branch-2. So it might be worth looking at to see if there is some overlap between the two.
        Hide
        Matteo Bertozzi added a comment -

        Yes this is fixed with MAPREDUCE-3790, the IOException during clientOut._flush() is now catched.

        Show
        Matteo Bertozzi added a comment - Yes this is fixed with MAPREDUCE-3790 , the IOException during clientOut._flush() is now catched.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12522230/mapreduce-4136.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5493/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12522230/mapreduce-4136.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5493/console This message was automatically generated.
        Allen Wittenauer made changes -
        Labels BB2015-05-TBR
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        22m 3s 1 Wouter de Bie 11/Apr/12 13:22

          People

          • Assignee:
            Unassigned
            Reporter:
            Wouter de Bie
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development