Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1517

streaming should support running on background

        Details

        • Type: New Feature New Feature
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 0.22.0
        • Fix Version/s: 0.22.0
        • Component/s: contrib/streaming
        • Labels:
          None
        • Hadoop Flags:
          Reviewed
        • Release Note:
          Adds -background option to run a streaming job in background.

          Description

          StreamJob submit the job and use a while loop monitor the progress.
          I prefer it running on background.

          Just add "&" at the end of command is a alternative solution, but it keeps a java process on client machine.
          When submit hundreds jobs at the same time, the client machine is overloaded.

          Adding a -background option to StreamJob, tell it only submit and don't monitor the progress.

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Bochun Bai added a comment -

            this patch adds -background to contrib-streaming.

            Show
            Bochun Bai added a comment - this patch adds -background to contrib-streaming.
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12436479/contrib-streaming-background.patch
            against trunk revision 912471.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed core unit tests.

            -1 contrib tests. The patch failed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436479/contrib-streaming-background.patch against trunk revision 912471. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/468/console This message is automatically generated.
            Hide
            Permalink
            Allen Wittenauer added a comment -

            Am I correct that this patch makes -background the default? I'm pretty certain that'll break things.

            Show
            Allen Wittenauer added a comment - Am I correct that this patch makes -background the default? I'm pretty certain that'll break things.
            Hide
            Permalink
            Bochun Bai added a comment -

            Allen: the default behavior remains unchanged.
            Two tests failed:
            1
            org.apache.hadoop.mapred.TestDebugScript.testDebugScript (from TestDebugScript)
            Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

            2
            org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal (from TestMiniMRLocalFS)
            It seems problem of trunk, MAPREDUCE-1510 also failed with the same message.

            Should I resume the progress and make hudson test it again?

            Show
            Bochun Bai added a comment - Allen: the default behavior remains unchanged. Two tests failed: 1 org.apache.hadoop.mapred.TestDebugScript.testDebugScript (from TestDebugScript) Timeout occurred. Please note the time in the report does not reflect the time until the timeout. 2 org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal (from TestMiniMRLocalFS) It seems problem of trunk, MAPREDUCE-1510 also failed with the same message. Should I resume the progress and make hudson test it again?
            Hide
            Permalink
            Bochun Bai added a comment -

            resume progress asking hudson re-test it.

            Show
            Bochun Bai added a comment - resume progress asking hudson re-test it.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            Bochun, Can you update the patch to trunk and upload again?

            One comment on the patch :

            • Update the -background option in exitUsage() with proper description and specify it as optional.
            Show
            Amareshwari Sriramadasu added a comment - Bochun, Can you update the patch to trunk and upload again? One comment on the patch : Update the -background option in exitUsage() with proper description and specify it as optional.
            Hide
            Permalink
            Bochun Bai added a comment -

            match current(rev 996456) trunk

            Show
            Bochun Bai added a comment - match current(rev 996456) trunk
            Hide
            Permalink
            Bochun Bai added a comment -

            Amareshwari, A new patch for current trunk is now available and the old files are deleted.

            Show
            Bochun Bai added a comment - Amareshwari, A new patch for current trunk is now available and the old files are deleted.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            A new patch for current trunk is now available and the old files are deleted.

            Thanks Bochun for the patch. But, next time please do not delete the old patch files because they would be useful for tracking the history.

            Changes in the patch look good. Minor nits:

            • Can you change the help message for the option to "Submit the job and don't wait till it completes." Also, log message should say "Job is running in background".
            • Also, can you add the option to the table in src/docs/src/documentation/content/xdocs/streaming.xml in "Streaming Command Options" section.
            Show
            Amareshwari Sriramadasu added a comment - A new patch for current trunk is now available and the old files are deleted. Thanks Bochun for the patch. But, next time please do not delete the old patch files because they would be useful for tracking the history. Changes in the patch look good. Minor nits: Can you change the help message for the option to "Submit the job and don't wait till it completes." Also, log message should say "Job is running in background". Also, can you add the option to the table in src/docs/src/documentation/content/xdocs/streaming.xml in "Streaming Command Options" section.
            Hide
            Permalink
            Bochun Bai added a comment -

            Thanks Amareshwari, text changes are included in this new patch.

            Show
            Bochun Bai added a comment - Thanks Amareshwari, text changes are included in this new patch.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            Patch looks good.

            Can you run test-patch and ant test and upload the results?

            Show
            Amareshwari Sriramadasu added a comment - Patch looks good. Can you run test-patch and ant test and upload the results?
            Hide
            Permalink
            Bochun Bai added a comment -

            test-patch report:

            -1 overall.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 system tests framework. The patch passed system tests framework compile.

            Show
            Bochun Bai added a comment - test-patch report: -1 overall. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile.
            Hide
            Permalink
            Bochun Bai added a comment -

            Added TestStreamingBackground.

            test-patch report is:
            -1 overall.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 2 warnings).

            +1 system tests framework. The patch passed system tests framework compile.

            and the "-1 release audit" is:
            $ cat releaseAuditDiffWarnings.txt

            1,2d0
            < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/DelayEchoApp.java
            < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBackground.java

            The patch logs is attached.

            Show
            Bochun Bai added a comment - Added TestStreamingBackground. test-patch report is: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 2 warnings). +1 system tests framework. The patch passed system tests framework compile. and the "-1 release audit" is: $ cat releaseAuditDiffWarnings.txt 1,2d0 < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/DelayEchoApp.java < [rat:report] !????? /data/home/bochun.bai/code/apache/hadoop-mapred-trunk-testwork/build/hadoop-mapred-PATCH-contrib-streaming-background-with-test.patch/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBackground.java The patch logs is attached.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            Can you add apache license headers for src test files, both TestStreamingBackground.java and DelayEchoApp.java? You can copy it from any other src file. Then, you won't see release audit warnings.

            Show
            Amareshwari Sriramadasu added a comment - Can you add apache license headers for src test files, both TestStreamingBackground.java and DelayEchoApp.java? You can copy it from any other src file. Then, you won't see release audit warnings.
            Hide
            Permalink
            Bochun Bai added a comment -

            license header added.

            Show
            Bochun Bai added a comment - license header added.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            Your patch is missing other files. Can you regenerate the patch properly?

            Show
            Amareshwari Sriramadasu added a comment - Your patch is missing other files. Can you regenerate the patch properly?
            Hide
            Permalink
            Bochun Bai added a comment -

            Previous patch contains only test changes.
            Sorry for that.

            This is a full patch.

            Show
            Bochun Bai added a comment - Previous patch contains only test changes. Sorry for that. This is a full patch.
            Hide
            Permalink
            Bochun Bai added a comment -

            test-patch.sh on trunk Revision: 997967

            +1 overall.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 6 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 system tests framework. The patch passed system tests framework compile.

            Show
            Bochun Bai added a comment - test-patch.sh on trunk Revision: 997967 +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system tests framework. The patch passed system tests framework compile.
            Hide
            Permalink
            Amareshwari Sriramadasu added a comment -

            +1
            All the tests passed with the patch.

            I just committed this. Thanks Bochun !

            Show
            Amareshwari Sriramadasu added a comment - +1 All the tests passed with the patch. I just committed this. Thanks Bochun !
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

            Show
            Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

              People

              • Assignee:
                Bochun Bai
                Reporter:
                Bochun Bai
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development