[SPARK-2538] External aggregation in Python

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.1
• Fix Version/s: 1.1.0
• Component/s: PySpark
• Labels: pyspark
• Target Version/s: 1.1.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 17/Jul/14 00:02
• Updated: 17/Feb/15 00:43
• Resolved: 25/Jul/14 05:54

Description

For huge reduce tasks, user will got out of memory exception when all the data can not fit in memory.

It should put some of the data into disks and then merge them together, just like what we do in Scala.

Issue Links

• is duplicated by: SPARK-2021 External hashing in PySpark
• is related to: SPARK-2876 RDD.partitionBy loads entire partition into memory
• links to: [Github] Pull Request #1460 (davies)