[SPARK-12384] Allow -Xms to be set differently then -Xmx

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.0
• Component/s: Spark Core, Spark Submit, YARN
• Assignee: Dhruve Ashar
• Reporter: Thomas Graves
• Created: 16/Dec/15 22:10
• Updated: 17/May/20 18:16
• Resolved: 07/Apr/16 15:40

Description

Currently Spark automatically sets the -Xms parameter to be the same as the -Xmx parameter. We should allow the user to set this separately.

The main use case here is if I'm running the spark-shell on a shared gateway. Many users specify a larger memory size then needed and will never use that much memory, so all its doing is preventing other users from potentially using that memory. Allowing it to be less is just more multi-tenant friendly.

I think it makes sense to leave this for cluster mode, although if a user really wants to override I don't see why we shouldn't let them.

Issue Links

• is related to: SPARK-16320 Document G1 heap region's effect on spark 2.0 vs 1.6
• links to: [Github] Pull Request #12115 (dhruve)