Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-587

Stream test TestStreamingExitStatus fails with Out of Memory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: contrib/streaming
    • Labels:
      None
    • Environment:

      OS/X, 64-bit x86 imac, 4GB RAM.

    • Release Note:
      Fixed the streaming test TestStreamingExitStatus's failure due to an OutOfMemory error by reducing the testcase's io.sort.mb.

      Description

      contrib/streaming tests are failing a test with an Out of Memory error on an OS/X Mac -same problem does not surface on Linux.

        Activity

        Hide
        Owen O'Malley added a comment -

        This is already committed to trunk.

        Show
        Owen O'Malley added a comment - This is already committed to trunk.
        Hide
        Amar Kamat added a comment -

        Attached is a patch for Yahoo!'s distribution of Hadoop not to be committed here.

        Show
        Amar Kamat added a comment - Attached is a patch for Yahoo!'s distribution of Hadoop not to be committed here.
        Hide
        Amar Kamat added a comment -

        Attaching an example patch for Yahoo! distribution of Hadoop not to be committed here.

        Show
        Amar Kamat added a comment - Attaching an example patch for Yahoo! distribution of Hadoop not to be committed here.
        Hide
        Todd Lipcon added a comment -

        This should be committed to branch-20 as well, since it's just a test bug and causes test failure on some systems.

        Show
        Todd Lipcon added a comment - This should be committed to branch-20 as well, since it's just a test bug and causes test failure on some systems.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #38 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/38/)
        . Fixes a OOM issue in TestStreamingExitStatus. Contributed by Amar Kamat.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #38 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/38/ ) . Fixes a OOM issue in TestStreamingExitStatus. Contributed by Amar Kamat.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amar!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amar!
        Hide
        Devaraj Das added a comment -

        Passing through hudson (on Amar's behalf)..

        Show
        Devaraj Das added a comment - Passing through hudson (on Amar's behalf)..
        Hide
        Amar Kamat added a comment -

        Attaching a patch that solves the problem on my box.

        Show
        Amar Kamat added a comment - Attaching a patch that solves the problem on my box.
        Hide
        Steve Loughran added a comment -

        stack trace. Presumably we just need to set the JVM parameters up right, but what are the right values?

        [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec
        [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED
        
        
        09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
        09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1
        09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done
        09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002
        java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:11 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003
        09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:14 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004
        09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:16 INFO streaming.StreamJob: killJob...
        
        Show
        Steve Loughran added a comment - stack trace. Presumably we just need to set the JVM parameters up right, but what are the right values? [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED 09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1 09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done 09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002 java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:11 INFO streaming.StreamJob: killJob... 09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf( Class ) or JobConf#setJar( String ). 09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003 09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:14 INFO streaming.StreamJob: killJob... 09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf( Class ) or JobConf#setJar( String ). 09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004 09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:16 INFO streaming.StreamJob: killJob...

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development