Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1068

In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.2
    • Component/s: contrib/streaming
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fix streaming job to show proper message if file is is not present, for -file option.

      Description

      With hadoop-0.20.0 proper error message is not thrown , when streaming job is submitted and if file is not present to be distributed with "-file" option. But with hadoop-0.18.* proper verbose message is thrown if file is not present and it is easy for the users to debug.

      For example:

      With hadoop-0.20.0:
      $ hadoop jar $HADOOP_HOME/hadoop-streaming.jar -Dmapred.job.queue.name=general -input /user/simon/test1.txt -output /user/simon/test.out -mapper "/bin/cat" -file dummy -reducer "/bin/cat"
      Streaming Job Failed!

      With hadoop-0.18.*:
      $ hadoop jar $HADOOP_HOME/hadoop-streaming.jar -input /user/simon/test1.txt -output /user/simon/test.out -mapper "/bin/cat" -file dummy -reducer "/bin/cat"
      09/10/06 10:06:49 ERROR streaming.StreamJob: Unexpected value "Argument : /home/simon/dummy doesn't exist." found while processing -file

      1. patch-1068.txt
        0.9 kB
        Amareshwari Sriramadasu
      2. patch-1068-1.txt
        1 kB
        Amareshwari Sriramadasu

        Activity

        Peeyush Bishnoi created issue -
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch printing the error message.

        Show
        Amareshwari Sriramadasu added a comment - Patch printing the error message.
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Attachment patch-1068.txt [ 12423818 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Manually verifies error message is printed.

        Show
        Amareshwari Sriramadasu added a comment - Manually verifies error message is printed.
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Amareshwari Sriramadasu [ amareshwari ]
        Hide
        Todd Lipcon added a comment -

        I'm not sure this is the right fix. Would it not be better to print out any exception caught at line 128? It currently says '//ignore, since log will already be printed' but this is clearly not true. In working on MAPREDUCE-967 I found that changing that to LOG.fatal("Error in stream job", ex) printed errors as expected.

        Show
        Todd Lipcon added a comment - I'm not sure this is the right fix. Would it not be better to print out any exception caught at line 128? It currently says '//ignore, since log will already be printed' but this is clearly not true. In working on MAPREDUCE-967 I found that changing that to LOG.fatal("Error in stream job", ex) printed errors as expected.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423818/patch-1068.txt
        against trunk revision 831816.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423818/patch-1068.txt against trunk revision 831816. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/218/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        I'm not sure this is the right fix. Would it not be better to print out any exception caught at line 128?

        fail method in streaming is used to fail the job whenever there is an invalid argument, in which it prints the appropriate message. Users might not want to see an exception trace if there is an invalid argument.

        In working on MAPREDUCE-967 I found that changing that to LOG.fatal("Error in stream job", ex) printed errors as expected.

        Sorry, I did not understand how MR-967 is related to command-line parsing errors in Streaming.

        Show
        Amareshwari Sriramadasu added a comment - I'm not sure this is the right fix. Would it not be better to print out any exception caught at line 128? fail method in streaming is used to fail the job whenever there is an invalid argument, in which it prints the appropriate message. Users might not want to see an exception trace if there is an invalid argument. In working on MAPREDUCE-967 I found that changing that to LOG.fatal("Error in stream job", ex) printed errors as expected. Sorry, I did not understand how MR-967 is related to command-line parsing errors in Streaming.
        Hide
        Todd Lipcon added a comment -

        Users might not want to see an exception trace if there is an invalid argument.

        Fair enough for invalid arguments, though I think at least a debug level log with the full stack trace would be helpful for unit tests.

        Sorry, I did not understand how MR-967 is related to command-line parsing errors in Streaming.

        It's not - I had to add a unit test for streaming there because the original patch introduced a regression that didn't already have a test. In working on the streaming unit test, I found that exceptions thrown inside StreamJob never produced a stack trace in the test logs, making it pretty difficult to debug.

        Show
        Todd Lipcon added a comment - Users might not want to see an exception trace if there is an invalid argument. Fair enough for invalid arguments, though I think at least a debug level log with the full stack trace would be helpful for unit tests. Sorry, I did not understand how MR-967 is related to command-line parsing errors in Streaming. It's not - I had to add a unit test for streaming there because the original patch introduced a regression that didn't already have a test. In working on the streaming unit test, I found that exceptions thrown inside StreamJob never produced a stack trace in the test logs, making it pretty difficult to debug.
        Hide
        Amareshwari Sriramadasu added a comment -

        Printing the exception log in debug mode.

        Show
        Amareshwari Sriramadasu added a comment - Printing the exception log in debug mode.
        Amareshwari Sriramadasu made changes -
        Attachment patch-1068-1.txt [ 12423898 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        The patch applies to trunk, branch 0.21 and branch 0.20 cleanly.

        Show
        Amareshwari Sriramadasu added a comment - The patch applies to trunk, branch 0.21 and branch 0.20 cleanly.
        Hide
        Amareshwari Sriramadasu added a comment -

        Re-submitting for hudson

        Show
        Amareshwari Sriramadasu added a comment - Re-submitting for hudson
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423898/patch-1068-1.txt
        against trunk revision 833275.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423898/patch-1068-1.txt against trunk revision 833275. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/229/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        -1 contrib tests.

        Is due to MAPREDUCE-1124

        Show
        Amareshwari Sriramadasu added a comment - -1 contrib tests. Is due to MAPREDUCE-1124
        Hide
        Iyappan Srinivasan added a comment -

        +1 for QA

        bin/hadoop jar contrib/streaming/hadoop-streaming.jar -input /user/test1.txt -output /user/test.out -mapper "/bin/cat"
        -file dummy -reducer "/bin/cat"

        Before patch the Streaming Failed msg only came.

        After patch, it gives,

        File: <path>/dummy does not exist, or is not readable.
        Streaming Job Failed!

        So, failed log file comes properly.

        Show
        Iyappan Srinivasan added a comment - +1 for QA bin/hadoop jar contrib/streaming/hadoop-streaming.jar -input /user/test1.txt -output /user/test.out -mapper "/bin/cat" -file dummy -reducer "/bin/cat" Before patch the Streaming Failed msg only came. After patch, it gives, File: <path>/dummy does not exist, or is not readable. Streaming Job Failed! So, failed log file comes properly.
        Hide
        Sharad Agarwal added a comment -

        I just committed this. Thanks Amareshwari.

        Show
        Sharad Agarwal added a comment - I just committed this. Thanks Amareshwari.
        Sharad Agarwal made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12314045 ]
        Fix Version/s 0.20.1 [ 12314047 ]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #116 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/116/)
        . Fix streaming job to show proper message if file is is not present. Contributed by Amareshwari Sriramadasu.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #116 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/116/ ) . Fix streaming job to show proper message if file is is not present. Contributed by Amareshwari Sriramadasu.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #141 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/141/)
        . Fix streaming job to show proper message if file is is not present. Contributed by Amareshwari Sriramadasu.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #141 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/141/ ) . Fix streaming job to show proper message if file is is not present. Contributed by Amareshwari Sriramadasu.
        Amareshwari Sriramadasu made changes -
        Release Note Fix streaming job to show proper message if file is is not present, for -file option.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Peeyush Bishnoi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development