[HADOOP-3075] Streaming reducer stuck after detecting corrupted inputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Runping Qi
• Created: 22/Mar/08 20:10
• Updated: 08/Jul/09 16:52
• Resolved: 06/Jun/08 11:55

Description

A reducer of a streaming job was stuck at 89% after detecting exception from child.
Below is the stack:

2008-03-22 19:29:00,649 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=155701/155680/0 in:710=155701/219 [rec/s] out:710=155680/219 [rec/s]
2008-03-22 19:29:00,667 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: problem advancing
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.java:238)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:314)
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:67)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:394)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2073)
Caused by: java.io.IOException: File is corrupt!
at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1855)
at org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:1972)
at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:3005)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.next(SequenceFile.java:2763)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.readNextKey(ReduceTask.java:275)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.java:236)

The task did not fail but just got stuck.
Speculative execution did not kick in.

The hadoop was built off trunk Version: 0.17.0-dev, r638716

Issue Links

• is related to: HADOOP-3039 Runtime exceptions not killing job