[HADOOP-3427] In ReduceTask::fetchOutputs, wait for result can be improved slightly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Release Note: Changed shuffle scheduler policy to wait for notifications from shuffle threads before scheduling more.
• Assignee: Devaraj Das
• Reporter: Devaraj Das
• Created: 21/May/08 12:59
• Updated: 08/Jul/09 16:52
• Resolved: 06/Jun/08 06:27

Description

The getCopyResult call in ReduceTask::fetchOutputs waits for 2 seconds if the results list is empty. This can be improved to wait only when sufficient number of fetches (above a certain threshold) have been scheduled. The threshold should ensure that all the fetcher threads would be busy enough, and in this case, the call to getCopyResult would return only when a fetcher thread wakes it up.