Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4154

streaming MR job succeeds even if the streaming command fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2
    • Fix Version/s: 1.0.3
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Hadoop 1.0.1 behaves as expected - The task fails for streaming MR job if the streaming command fails. But it succeeds in hadoop 1.0.2 .

        Activity

        Hide
        Thejas M Nair added a comment -

        Example of streaming job that succeeds with 1.0.2 -

        sudo -u templeton hadoop fs -rmr /tmp/t.out; sudo -u templeton /usr/bin/hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-1.0.2.jar -input /tmp/nums.txt -output /tmp/t.out -mapper '/bin/ls no_such-file-12e3' -reducer /usr/bin/wc

        Show
        Thejas M Nair added a comment - Example of streaming job that succeeds with 1.0.2 - sudo -u templeton hadoop fs -rmr /tmp/t.out; sudo -u templeton /usr/bin/hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-1.0.2.jar -input /tmp/nums.txt -output /tmp/t.out -mapper '/bin/ls no_such-file-12e3' -reducer /usr/bin/wc
        Hide
        Devaraj Das added a comment -

        Patch for hadoop-1 line. Seems like the fix is already there in trunk via MAPREDUCE-3790. Alternative is to backport the patch for MAPREDUCE-3790 into hadoop-1..

        Show
        Devaraj Das added a comment - Patch for hadoop-1 line. Seems like the fix is already there in trunk via MAPREDUCE-3790 . Alternative is to backport the patch for MAPREDUCE-3790 into hadoop-1..
        Hide
        Thomas Graves added a comment -

        The actual fix in the patch is the same as in MAPREDUCE-3790, just going to pull this one in.

        +1 thanks Devaraj!

        Show
        Thomas Graves added a comment - The actual fix in the patch is the same as in MAPREDUCE-3790 , just going to pull this one in. +1 thanks Devaraj!
        Hide
        Thomas Graves added a comment -

        committed this to branch-1.

        Show
        Thomas Graves added a comment - committed this to branch-1.
        Hide
        Owen O'Malley added a comment -

        I think we should put this into 1.0.3. Thoughts?

        Show
        Owen O'Malley added a comment - I think we should put this into 1.0.3. Thoughts?
        Hide
        Devaraj Das added a comment -

        I think we should put this into 1.0.3. Thoughts?

        +1

        Show
        Devaraj Das added a comment - I think we should put this into 1.0.3. Thoughts? +1
        Hide
        Nathan Roberts added a comment -

        +1 on 1.0.3. I think this fixes other cases where the pipe shuts down before we've finished gathering the output from the reducer.

        Show
        Nathan Roberts added a comment - +1 on 1.0.3. I think this fixes other cases where the pipe shuts down before we've finished gathering the output from the reducer.
        Hide
        Owen O'Malley added a comment -

        I just committed this back to 1.0.3 and moved the log in the branch-1/CHANGES.txt.

        Show
        Owen O'Malley added a comment - I just committed this back to 1.0.3 and moved the log in the branch-1/CHANGES.txt.
        Hide
        Matt Foley added a comment -

        +1. Thanks, Owen and Devaraj.

        Show
        Matt Foley added a comment - +1. Thanks, Owen and Devaraj.
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop-1.0.3.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop-1.0.3.

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Thejas M Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development