[SPARK-701] Wrong SPARK_MEM setting with different EC2 master and worker machine types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.0
• Component/s: EC2
• Assignee: Shivaram Venkataraman
• Reporter: Josh Rosen
• Created: 21/Feb/13 16:33
• Updated: 30/Jun/15 06:07
• Resolved: 14/May/13 15:38

Description

When launching a spark-ec2 cluster using different worker and master machine types, SPARK_MEM in spark-env.sh is set based on the master's memory instead of the worker's. This causes jobs to hang if the master has more memory than the workers (because jobs will request too much memory).