[HADOOP-654] jobs fail with some hardware/system failures on a small number of nodes

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.7.2
• Fix Version/s: 0.12.0
• Assignee: Arun Murthy
• Reporter: Yoram Arnon
• Created: 30/Oct/06 18:47
• Updated: 08/Jul/09 16:51
• Resolved: 22/Feb/07 19:27

Description

occasionally, such as when the OS is out of some resource, a node fails only partly. The node is up and running, the task tracker is running and sending heartbeats, but every task fails because the tasktracker can't fork tasks or something.
In these cases, that task tracker keeps getting assigned tasks to execute, and they all fail.
A couple of nodes like that and jobs start failing badly.

The job tracker should avoid assigning tasks to tasktrackers that are misbehaving.

simple approach: avoid tasktrackers that report many more failures than average (say 3X). Simply use the info sent by the TT.
better but harder: track TT failures over time and:
1. avoid those that exhibit a high failure rate
2. tell them to shut down