[HADOOP-1740] Certain Pipes tasks fail, after exiting the C++ application

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.1

Environment

Version: 0.15.0-dev, r565628
Compiled: Tue Aug 14 20:55:37 UTC 2007 by hadoopqa
165 nodes

• Assignee: Devaraj Das
• Reporter: Srikanth Kakani
• Created: 20/Aug/07 22:51
• Updated: 08/Jul/09 16:41
• Resolved: 27/Aug/07 17:40

Description

Steps to reproduce:
Run a pipes job. I had 1182 mappers and 300 reducers and produced 4,353,892,559 records and 212,030,552,146 bytes output.

Some of the tasks failed (randomly), all had the same exception (probably a race condition)

task_200708201818_0002_m_000681_0 tip_200708201818_0002_m_000681 node075 FAILED

java.lang.NullPointerException
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:592)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1800)

Last 4KB
Last 8KB
All
task_200708201818_0002_m_000681_1 tip_200708201818_0002_m_000681 node1127 FAILED

java.lang.NullPointerException
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:592)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1800)

Last 4KB
Last 8KB
All
task_200708201818_0002_m_000868_0 tip_200708201818_0002_m_000868 node160 FAILED

java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:405)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:291)

Last 4KB
Last 8KB
All

2007-08-20 19:25:59,909 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2007-08-20 19:26:00,002 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 300
2007-08-20 19:26:01,344 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2007-08-20 19:26:01,345 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2007-08-20 19:49:26,727 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.NullPointerException
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:592)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:190)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1800)