[SPARK-8776] Increase the default MaxPermSize

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: Spark Core
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 02/Jul/15 03:08
• Updated: 03/Jul/15 05:10
• Resolved: 03/Jul/15 05:09

Description

Since 1.4.0, Spark SQL has isolated class loaders for seperating hive dependencies on metastore and execution, which increases the memory consumption of PermGen. How about we increase the default size from 128m to 256m? Seems the change we need to make is https://github.com/apache/spark/blob/3c0156899dc1ec1f7dfe6d7c8af47fa6dc7d00bf/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L139.

Issue Links

• links to: [Github] Pull Request #7196 (yhuai)