[SPARK-1879] Default PermGen size too small when using Hadoop2 and Hive

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Matei Alexandru Zaharia
• Reporter: Matei Alexandru Zaharia
• Created: 19/May/14 07:20
• Updated: 20/May/14 01:44
• Resolved: 20/May/14 01:44

Description

If you launch a spark-shell with Hadoop 2 and Hive on the classpath, and try to use Hive therein, the PermGen quickly reaches 85 MB after a few commands, at which point Java gives up and freezes. We should pass a MaxPermSize to prevent this. Unfortunately passing this results in a warning on Java 8, but that's still better than not passing it.

I don't think this affects stuff other than the shell; it's just the combination of Scala compiler + Hive + Hadoop 2 that pushes things over the edge.