[MAPREDUCE-4857] Fix 126 error during map/reduce phase

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.0.4
• Assignee: Unassigned
• Reporter: Fengdong Yu
• Created: 07/Dec/12 05:31
• Updated: 02/Apr/15 12:21
• Resolved: 02/Apr/15 12:21

Description

There is rare happenings during map or reduce phase, but mostly in map phase, the Exception messages:
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

and error logs are cleaned, so It's very hard to debug.

but I compared DefaultTaskController.java with 0.22, they use "bash command" to start the job scritp, but 1.0.4 use "bash, "-c", command".

I removed "-c", everything is ok, 126 error code never happen again.

I read man document of bash, it indicates when fork a new thread with write command, another thread with "bash -c" also has a writable fd. so I think it could return 126 status occasionally.

So, there is only one line fix for this issue.

Issue Links

• relates to: MAPREDUCE-2374 "Text File Busy" errors launching MR tasks