Hadoop Common
  1. Hadoop Common
  2. HADOOP-3089

streaming should accept stderr from task before first key arrives

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Stderr output from a streaming task is not collected until the MRErrorThread is started by PipeMapRed.startOutputThreads(), which is done on the first call to map() or reduce().

      This makes it difficult to debug failures in starting up the task process. It can also lead to deadlock when a task receives no input keys but produces significant stderr output: the process will block on writing to stderr, while streaming will block waiting for the process to exit.

      We should start the MRErrorThread when the process is forked, and then add the reporter later to enable stderr output serve as a keep-alive.

      1. patch-stderr-3089.txt
        9 kB
        Rick Cox
      2. patch-stderr-3089-2.txt
        9 kB
        Rick Cox

        Issue Links

          Activity

          Ravi Gummadi made changes -
          Link This issue relates to MAPREDUCE-1813 [ MAPREDUCE-1813 ]
          Owen O'Malley made changes -
          Component/s contrib/streaming [ 12310972 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #451 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/ )
          Tom White made changes -
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Rick!

          Show
          Tom White added a comment - I've just committed this. Thanks Rick!
          Hide
          Amareshwari Sriramadasu added a comment -

          +1
          Code looks good.

          Show
          Amareshwari Sriramadasu added a comment - +1 Code looks good.
          Hide
          Rick Cox added a comment -

          Is anyone available to review this? I'd like for this to make the 0.17 freeze.

          thanks

          Show
          Rick Cox added a comment - Is anyone available to review this? I'd like for this to make the 0.17 freeze. thanks
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12378601/patch-stderr-3089-2.txt
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 6 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378601/patch-stderr-3089-2.txt against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 6 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/console This message is automatically generated.
          Rick Cox made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rick Cox made changes -
          Attachment patch-stderr-3089-2.txt [ 12378601 ]
          Hide
          Rick Cox added a comment -

          Same patch, minus 2 commented out lines that shouldn't have been included (and with correct license checkbox).

          Show
          Rick Cox added a comment - Same patch, minus 2 commented out lines that shouldn't have been included (and with correct license checkbox).
          Rick Cox made changes -
          Attachment patch-stderr-3089-2.txt [ 12378600 ]
          Rick Cox made changes -
          Attachment patch-stderr-3089-2.txt [ 12378600 ]
          Hide
          Rick Cox added a comment -

          Same patch, minus two commented out lines that shouldn't have been included.

          Show
          Rick Cox added a comment - Same patch, minus two commented out lines that shouldn't have been included.
          Rick Cox made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12378598/patch-stderr-3089.txt
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 6 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378598/patch-stderr-3089.txt against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 6 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/console This message is automatically generated.
          Rick Cox made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rick Cox made changes -
          Field Original Value New Value
          Attachment patch-stderr-3089.txt [ 12378598 ]
          Hide
          Rick Cox added a comment -

          Patch starts MRErrorThread immediately, adds a reporter later.

          testStderrNoInput() fails without this patch, and passes with it on Linux (RHEL3) and Mac OS X. It does make some assumptions about how large a stderr buffer might be, but that's unfortunately an innate component of this issue.

          testStderrCountsAsProgress() passes, but isn't meaninful because LocalJobRunner does not seem to support mapred.task.timeout.

          Show
          Rick Cox added a comment - Patch starts MRErrorThread immediately, adds a reporter later. testStderrNoInput() fails without this patch, and passes with it on Linux (RHEL3) and Mac OS X. It does make some assumptions about how large a stderr buffer might be, but that's unfortunately an innate component of this issue. testStderrCountsAsProgress() passes, but isn't meaninful because LocalJobRunner does not seem to support mapred.task.timeout .
          Rick Cox created issue -

            People

            • Assignee:
              Rick Cox
              Reporter:
              Rick Cox
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development