[HADOOP-2131] Speculative execution should be allowed for reducers only

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.16.0

Environment

Hadoop job, map fetches data from external systems

• Assignee: Amareshwari Sriramadasu
• Reporter: Srikanth Kakani
• Created: 30/Oct/07 23:54
• Updated: 08/Jul/09 16:52
• Resolved: 10/Jan/08 20:09

Description

Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.

Speculative execution only on reducers would be a great way to solve this problem.