[HADOOP-2980] slow reduce copies - map output locations not being fetched even when map complete

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 0.15.3
• Assignee: Unassigned
• Reporter: Joydeep Sen Sarma
• Created: 08/Mar/08 03:02
• Updated: 17/Jul/14 21:38
• Resolved: 17/Jul/14 21:38

Description

maps are long finished. reduces are stuck looking for map locations. they make progress - but slowly. it almost seems like they get new map locations every minute or so:

2008-03-07 18:50:52,737 INFO org.apache.hadoop.mapred.ReduceTask: task_200803041231_3586_r_000021_0 done copying task_200803041231_3586_m_004620_0 output from hadoop082.sf2p.facebook.com..
2008-03-07 18:50:53,733 INFO org.apache.hadoop.mapred.ReduceTask: task_200803041231_3586_r_000021_0: Got 0 new map-outputs & 0 obsolete map-outputs from tasktracker and 0 map-outputs from previous failures
2008-03-07 18:50:53,733 INFO org.apache.hadoop.mapred.ReduceTask: task_200803041231_3586_r_000021_0 Got 0 known map output location(s); scheduling...
...
2008-03-07 18:51:49,767 INFO org.apache.hadoop.mapred.ReduceTask: task_200803041231_3586_r_000021_0 Got 50 known map output location(s); scheduling...
2008-03-07 18:51:49,767 INFO org.apache.hadoop.mapred.ReduceTask: task_200803041231_3586_r_000021_0 Scheduled 41 of 50 known outputs (0 slow hosts and 9 dup hosts)

they get about 50 locations at a time and this 1 minute delay pattern is surprisingly common ..