[SPARK-751] Consolidate shuffle files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.1
• Assignee: Jason Dai
• Reporter: Reynold Xin
• Created: 21/May/13 00:31
• Updated: 14/Nov/13 18:29
• Resolved: 14/Nov/13 18:29

Description

Right now on each machine, we create M * R temporary files for shuffle, where M = number of map tasks, R = number of reduce tasks.

This can be pretty high when there are lots of mappers and reducers (e.g. 1k map * 1k reduce = 1 million files for a single shuffle). The high number can cripple the file system and significantly slow the system down.

We should cut this number down to O(R) instead of O(M*R).