Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1517

streaming should support running on background

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds -background option to run a streaming job in background.

      Description

      StreamJob submit the job and use a while loop monitor the progress.
      I prefer it running on background.

      Just add "&" at the end of command is a alternative solution, but it keeps a java process on client machine.
      When submit hundreds jobs at the same time, the client machine is overloaded.

      Adding a -background option to StreamJob, tell it only submit and don't monitor the progress.

        Activity

        Hide
        Bochun Bai added a comment -

        this patch adds -background to contrib-streaming.

        Show
        Bochun Bai added a comment - this patch adds -background to contrib-streaming.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12436479/contrib-streaming-background.patch
        against trunk revision 912471.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436479/contrib-streaming-background.patch against trunk revision 912471. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/console This message is automatically generated.
        Hide
        Allen Wittenauer added a comment -

        Am I correct that this patch makes -background the default? I'm pretty certain that'll break things.

        Show
        Allen Wittenauer added a comment - Am I correct that this patch makes -background the default? I'm pretty certain that'll break things.
        Hide
        Bochun Bai added a comment -

        Allen: the default behavior remains unchanged.
        Two tests failed:
        1
        org.apache.hadoop.mapred.TestDebugScript.testDebugScript (from TestDebugScript)
        Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

        2
        org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal (from TestMiniMRLocalFS)
        It seems problem of trunk, MAPREDUCE-1510 also failed with the same message.

        Should I resume the progress and make hudson test it again?

        Show
        Bochun Bai added a comment - Allen: the default behavior remains unchanged. Two tests failed: 1 org.apache.hadoop.mapred.TestDebugScript.testDebugScript (from TestDebugScript) Timeout occurred. Please note the time in the report does not reflect the time until the timeout. 2 org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal (from TestMiniMRLocalFS) It seems problem of trunk, MAPREDUCE-1510 also failed with the same message. Should I resume the progress and make hudson test it again?
        Hide
        Bochun Bai added a comment -

        resume progress asking hudson re-test it.

        Show
        Bochun Bai added a comment - resume progress asking hudson re-test it.
        Hide
        Amareshwari Sriramadasu added a comment -

        Bochun, Can you update the patch to trunk and upload again?

        One comment on the patch :

        • Update the -background option in exitUsage() with proper description and specify it as optional.
        Show
        Amareshwari Sriramadasu added a comment - Bochun, Can you update the patch to trunk and upload again? One comment on the patch : Update the -background option in exitUsage() with proper description and specify it as optional.
        Hide
        Bochun Bai added a comment -

        match current(rev 996456) trunk

        Show
        Bochun Bai added a comment - match current(rev 996456) trunk
        Hide
        Bochun Bai added a comment -

        Amareshwari, A new patch for current trunk is now available and the old files are deleted.

        Show
        Bochun Bai added a comment - Amareshwari, A new patch for current trunk is now available and the old files are deleted.
        Hide
        Amareshwari Sriramadasu added a comment -

        A new patch for current trunk is now available and the old files are deleted.

        Thanks Bochun for the patch. But, next time please do not delete the old patch files because they would be useful for tracking the history.

        Changes in the patch look good. Minor nits:

        • Can you change the help message for the option to "Submit the job and don't wait till it completes." Also, log message should say "Job is running in background".
        • Also, can you add the option to the table in src/docs/src/documentation/content/xdocs/streaming.xml in "Streaming Command Options" section.
        Show
        Amareshwari Sriramadasu added a comment - A new patch for current trunk is now available and the old files are deleted. Thanks Bochun for the patch. But, next time please do not delete the old patch files because they would be useful for tracking the history. Changes in the patch look good. Minor nits: Can you change the help message for the option to "Submit the job and don't wait till it completes." Also, log message should say "Job is running in background". Also, can you add the option to the table in src/docs/src/documentation/content/xdocs/streaming.xml in "Streaming Command Options" section.
        Hide
        Bochun Bai added a comment -

        Thanks Amareshwari, text changes are included in this new patch.

        Show
        Bochun Bai added a comment - Thanks Amareshwari, text changes are included in this new patch.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch looks good.

        Can you run test-patch and ant test and upload the results?

        Show
        Amareshwari Sriramadasu added a comment - Patch looks good. Can you run test-patch and ant test and upload the results?
        Hide
        Bochun Bai added a comment -

        test-patch report:

        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 system tests framework. The patch passed system tests framework compile.

        Show
        Bochun Bai added a comment - test-patch report: -1 overall. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile.
        Hide
        Bochun Bai added a comment -

        Added TestStreamingBackground.

        test-patch report is:
        -1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 2 warnings).

        +1 system tests framework. The patch passed system tests framework compile.

        and the "-1 release audit" is:
        $ cat releaseAuditDiffWarnings.txt

        1,2d0
        < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/DelayEchoApp.java
        < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBackground.java

        The patch logs is attached.

        Show
        Bochun Bai added a comment - Added TestStreamingBackground. test-patch report is: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 2 warnings). +1 system tests framework. The patch passed system tests framework compile. and the "-1 release audit" is: $ cat releaseAuditDiffWarnings.txt 1,2d0 < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/DelayEchoApp.java < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBackground.java The patch logs is attached.
        Hide
        Amareshwari Sriramadasu added a comment -

        Can you add apache license headers for src test files, both TestStreamingBackground.java and DelayEchoApp.java? You can copy it from any other src file. Then, you won't see release audit warnings.

        Show
        Amareshwari Sriramadasu added a comment - Can you add apache license headers for src test files, both TestStreamingBackground.java and DelayEchoApp.java? You can copy it from any other src file. Then, you won't see release audit warnings.
        Hide
        Bochun Bai added a comment -

        license header added.

        Show
        Bochun Bai added a comment - license header added.
        Hide
        Amareshwari Sriramadasu added a comment -

        Your patch is missing other files. Can you regenerate the patch properly?

        Show
        Amareshwari Sriramadasu added a comment - Your patch is missing other files. Can you regenerate the patch properly?
        Hide
        Bochun Bai added a comment -

        Previous patch contains only test changes.
        Sorry for that.

        This is a full patch.

        Show
        Bochun Bai added a comment - Previous patch contains only test changes. Sorry for that. This is a full patch.
        Hide
        Bochun Bai added a comment -

        test-patch.sh on trunk Revision: 997967

        +1 overall.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 system tests framework. The patch passed system tests framework compile.

        Show
        Bochun Bai added a comment - test-patch.sh on trunk Revision: 997967 +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile.
        Hide
        Amareshwari Sriramadasu added a comment -

        +1
        All the tests passed with the patch.

        I just committed this. Thanks Bochun !

        Show
        Amareshwari Sriramadasu added a comment - +1 All the tests passed with the patch. I just committed this. Thanks Bochun !
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

          People

          • Assignee:
            Bochun Bai
            Reporter:
            Bochun Bai
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development