[SPARK-946] Reduce memory footprint of DiskBlockManager.blockToFileSegmentMap

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.1, 0.9.0
• Fix Version/s: 0.8.1, 0.9.0
• Assignee: Aaron Davidson
• Reporter: Aaron Davidson
• Created: 26/Oct/13 13:53
• Updated: 05/Nov/13 08:13
• Resolved: 05/Nov/13 08:13

Description

blockToFileSegmentMap right now is taking up ~400 bytes per FileSegment. In large shuffles (e.g., >1000 mappers/executor and >1000 reducers), this can lead to several GB used just for this map, which is leading to OOMs.