[HADOOP-3837] hadop streaming does not use progress reporting to detect hung tasks

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.18.0, 0.19.0
• Hadoop Flags: Incompatible change
• Release Note: Changed streaming tasks to adhere to task timeout value specified in the job configuration.
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 25/Jul/08 21:59
• Updated: 08/Jul/09 17:05
• Resolved: 02/Aug/08 00:15

Description

Hadoop streaming (StreamJob.java) sets mapred.task.timeout to 0. This effectively means that if tasks hang (either due to bad user code or machine related issue), these tasks never encounters a timeout. This causes the entire job to hang.

I propose that hadoop streaming not set the timeout to 0. By default, the settings in hadoop-default.xml should be effective for streaming jobs.