Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-587

Stream test TestStreamingExitStatus fails with Out of Memory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: contrib/streaming
    • Labels:
      None
    • Environment:

      OS/X, 64-bit x86 imac, 4GB RAM.

    • Release Note:
      Fixed the streaming test TestStreamingExitStatus's failure due to an OutOfMemory error by reducing the testcase's io.sort.mb.

      Description

      contrib/streaming tests are failing a test with an Out of Memory error on an OS/X Mac -same problem does not surface on Linux.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        67d 1h 56m 1 Devaraj Das 30/Jul/09 19:05
        Patch Available Patch Available Resolved Resolved
        4d 12h 44m 1 Devaraj Das 04/Aug/09 07:49
        Resolved Resolved Reopened Reopened
        203d 12h 8m 1 Todd Lipcon 23/Feb/10 18:58
        Reopened Reopened Resolved Resolved
        472d 4h 17m 1 Owen O'Malley 11/Jun/11 00:16
        Resolved Resolved Closed Closed
        157d 1h 33m 1 Arun C Murthy 15/Nov/11 00:49
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Resolution Fixed [ 1 ]
        Hide
        Owen O'Malley added a comment -

        This is already committed to trunk.

        Show
        Owen O'Malley added a comment - This is already committed to trunk.
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12314045 ]
        Vinod Kumar Vavilapalli made changes -
        Release Note Fixed the streaming test TestStreamingExitStatus's failure due ot Out of Memory by reducing the testcase's io.sort.mb. Fixed the streaming test TestStreamingExitStatus's failure due to an OutOfMemory error by reducing the testcase's io.sort.mb.
        Vinod Kumar Vavilapalli made changes -
        Release Note Reduced the io.sort.mb in TestStreamingExitStatus to prevent OOM. Fixed the streaming test TestStreamingExitStatus's failure due ot Out of Memory by reducing the testcase's io.sort.mb.
        Gavin made changes -
        Reporter Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
        Amar Kamat made changes -
        Attachment mr-587-yahoo-y20-v1.1.patch [ 12441626 ]
        Hide
        Amar Kamat added a comment -

        Attached is a patch for Yahoo!'s distribution of Hadoop not to be committed here.

        Show
        Amar Kamat added a comment - Attached is a patch for Yahoo!'s distribution of Hadoop not to be committed here.
        Amar Kamat made changes -
        Attachment mr-587-yahoo-y20-v1.0.patch [ 12441601 ]
        Hide
        Amar Kamat added a comment -

        Attaching an example patch for Yahoo! distribution of Hadoop not to be committed here.

        Show
        Amar Kamat added a comment - Attaching an example patch for Yahoo! distribution of Hadoop not to be committed here.
        Todd Lipcon made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Todd Lipcon added a comment -

        This should be committed to branch-20 as well, since it's just a test bug and causes test failure on some systems.

        Show
        Todd Lipcon added a comment - This should be committed to branch-20 as well, since it's just a test bug and causes test failure on some systems.
        Amar Kamat made changes -
        Release Note Reduced the io.sort.mb in TestStreamingExitStatus to prevent OOM.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #38 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/38/)
        . Fixes a OOM issue in TestStreamingExitStatus. Contributed by Amar Kamat.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #38 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/38/ ) . Fixes a OOM issue in TestStreamingExitStatus. Contributed by Amar Kamat.
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.21.0 [ 12314045 ]
        Resolution Fixed [ 1 ]
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amar!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amar!
        Devaraj Das made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Amar Kamat [ amar_kamat ]
        Hide
        Devaraj Das added a comment -

        Passing through hudson (on Amar's behalf)..

        Show
        Devaraj Das added a comment - Passing through hudson (on Amar's behalf)..
        Amar Kamat made changes -
        Attachment MAPREDUCE-587-v1.0.patch [ 12414990 ]
        Hide
        Amar Kamat added a comment -

        Attaching a patch that solves the problem on my box.

        Show
        Amar Kamat added a comment - Attaching a patch that solves the problem on my box.
        Owen O'Malley made changes -
        Field Original Value New Value
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-5906 MAPREDUCE-587
        Affects Version/s 0.21.0 [ 12313563 ]
        Component/s contrib/streaming [ 12312905 ]
        Component/s contrib/streaming [ 12310972 ]
        Hide
        Steve Loughran added a comment -

        stack trace. Presumably we just need to set the JVM parameters up right, but what are the right values?

        [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec
        [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED
        
        
        09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
        09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1
        09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done
        09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002
        java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:11 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003
        09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:14 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004
        09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:16 INFO streaming.StreamJob: killJob...
        
        Show
        Steve Loughran added a comment - stack trace. Presumably we just need to set the JVM parameters up right, but what are the right values? [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED 09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1 09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done 09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002 java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:11 INFO streaming.StreamJob: killJob... 09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf( Class ) or JobConf#setJar( String ). 09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003 09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:14 INFO streaming.StreamJob: killJob... 09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf( Class ) or JobConf#setJar( String ). 09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004 09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:16 INFO streaming.StreamJob: killJob...
        Steve Loughran created issue -

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development