[SPARK-3466] Limit size of results that a driver collects for each action

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Davies Liu
• Reporter: Matei Alexandru Zaharia
• Created: 10/Sep/14 00:12
• Updated: 02/Nov/14 07:04
• Resolved: 02/Nov/14 07:04

Description

Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data. We should add a spark.driver.maxResultSize setting (or something like that) that will make the driver abort a job if its result is too big. We can set it to some fraction of the driver's memory by default, or to something like 100 MB.

Issue Links

• links to: [Github] Pull Request #3003 (davies)
• links to: Pull Request