Hadoop Common
  1. Hadoop Common
  2. HADOOP-3089

streaming should accept stderr from task before first key arrives

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1, 0.15.2, 0.15.3, 0.16.0, 0.16.1
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Stderr output from a streaming task is not collected until the MRErrorThread is started by PipeMapRed.startOutputThreads(), which is done on the first call to map() or reduce().

      This makes it difficult to debug failures in starting up the task process. It can also lead to deadlock when a task receives no input keys but produces significant stderr output: the process will block on writing to stderr, while streaming will block waiting for the process to exit.

      We should start the MRErrorThread when the process is forked, and then add the reporter later to enable stderr output serve as a keep-alive.

      1. patch-stderr-3089-2.txt
        9 kB
        Rick Cox
      2. patch-stderr-3089.txt
        9 kB
        Rick Cox

        Issue Links

          Activity

          Hide
          Rick Cox added a comment -

          Patch starts MRErrorThread immediately, adds a reporter later.

          testStderrNoInput() fails without this patch, and passes with it on Linux (RHEL3) and Mac OS X. It does make some assumptions about how large a stderr buffer might be, but that's unfortunately an innate component of this issue.

          testStderrCountsAsProgress() passes, but isn't meaninful because LocalJobRunner does not seem to support mapred.task.timeout.

          Show
          Rick Cox added a comment - Patch starts MRErrorThread immediately, adds a reporter later. testStderrNoInput() fails without this patch, and passes with it on Linux (RHEL3) and Mac OS X. It does make some assumptions about how large a stderr buffer might be, but that's unfortunately an innate component of this issue. testStderrCountsAsProgress() passes, but isn't meaninful because LocalJobRunner does not seem to support mapred.task.timeout .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12378598/patch-stderr-3089.txt
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 6 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378598/patch-stderr-3089.txt against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 6 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2056/console This message is automatically generated.
          Hide
          Rick Cox added a comment -

          Same patch, minus two commented out lines that shouldn't have been included.

          Show
          Rick Cox added a comment - Same patch, minus two commented out lines that shouldn't have been included.
          Hide
          Rick Cox added a comment -

          Same patch, minus 2 commented out lines that shouldn't have been included (and with correct license checkbox).

          Show
          Rick Cox added a comment - Same patch, minus 2 commented out lines that shouldn't have been included (and with correct license checkbox).
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12378601/patch-stderr-3089-2.txt
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 6 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12378601/patch-stderr-3089-2.txt against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 6 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2057/console This message is automatically generated.
          Hide
          Rick Cox added a comment -

          Is anyone available to review this? I'd like for this to make the 0.17 freeze.

          thanks

          Show
          Rick Cox added a comment - Is anyone available to review this? I'd like for this to make the 0.17 freeze. thanks
          Hide
          Amareshwari Sriramadasu added a comment -

          +1
          Code looks good.

          Show
          Amareshwari Sriramadasu added a comment - +1 Code looks good.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Rick!

          Show
          Tom White added a comment - I've just committed this. Thanks Rick!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #451 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/ )

            People

            • Assignee:
              Rick Cox
              Reporter:
              Rick Cox
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development