[SPARK-3071] Increase default driver memory

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.2
• Fix Version/s: 1.5.0
• Component/s: Spark Core
• Target Version/s: 1.5.0
• Assignee: Ilya Ganelin
• Reporter: Xiangrui Meng
• Created: 15/Aug/14 15:42
• Updated: 02/Jul/15 06:11
• Resolved: 02/Jul/15 06:11

Description

The current default is 512M, which is usually too small because user also uses driver to do some computation. In local mode, executor memory setting is ignored while only driver memory is used, which provides more incentive to increase the default driver memory.

I suggest

1. 2GB in local mode and warn users if executor memory is set a bigger value
2. same as worker memory on an EC2 standalone server

Issue Links

• relates to: SPARK-6120 DecisionTree.save uses too much Java heap space for default spark shell settings
• links to: [Github] Pull Request #7132 (ilganeli)