Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4136

Hadoop streaming might succeed even through reducer fails

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 0.20.205.0
    • None
    • contrib/streaming

    Description

      Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls PipeReducer.close(), but in the mean time the reducer has failed and the process has died. When clientOut_.flush() throws an IOException in PipeMapRed.mapRedFinish() this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.

      I've attached a patch that seems to fix it for us.

      Attachments

        1. mapreduce-4136.patch
          2 kB
          Wouter de Bie

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            wouterdebie Wouter de Bie

            Dates

              Created:
              Updated:

              Slack

                Issue deployment