[MAPREDUCE-1777] In streaming, jobs that used to work, crash in the map phase -- even if the mapper is /bin/cat

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: contrib/streaming
• Assignee: Unassigned
• Reporter: arkady borkovsky
• Created: 15/Dec/07 23:57
• Updated: 19/Aug/10 20:41
• Resolved: 19/Aug/10 20:41

Description

The exception is either "out of memory" of or "broken pipe" – see both stack dumps bellow.

st Hadoop input: |null|
last tool output: |[B@20fa83|
Date: Sat Dec 15 21:02:18 UTC 2007
java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:96)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:107)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

-------------------------------------------------
java.io.IOException: MROutput/MRErrThread
failed:java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect (MapTask.java:347)
at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run (PipeMapRed.java:344)

at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:76)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
1760)