[SPARK-2711] Create a ShuffleMemoryManager that allocates across spilling collections in the same task

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Target Version/s: 1.1.0
• Assignee: Matei Alexandru Zaharia
• Reporter: Matei Alexandru Zaharia
• Created: 28/Jul/14 01:29
• Updated: 17/Nov/14 23:26
• Resolved: 05/Aug/14 07:31

Description

Right now if there are two ExternalAppendOnlyMaps, they don't compete correctly for memory. This can happen e.g. in a task that is both reducing data from its parent RDD and writing it out to files for a future shuffle, for instance if you do rdd.groupByKey(...).map(...).groupByKey(...) (another key).

Issue Links

• breaks: SPARK-4452 Shuffle data structures can starve others on the same thread for memory
• contains: SPARK-2530 Relax incorrect assumption of one ExternalAppendOnlyMap per thread
• is related to: SPARK-2530 Relax incorrect assumption of one ExternalAppendOnlyMap per thread
• relates to: SPARK-3633 Fetches failure observed after SPARK-2711
• links to: [Github] Pull Request #1707 (mateiz)