[SPARK-5086] Specifying the master instance type

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: EC2
• Labels: ec2

Environment

aws ec2

• Target Version/s: 1.3.0
• Assignee: Unassigned
• Reporter: inkyu lee
• Created: 05/Jan/15 06:11
• Updated: 15/Apr/15 16:03
• Resolved: 10/Jan/15 17:11

Description

I am using the m2.xlarge as master node, and r3.xlarge as slave nodes since they provide large memories and I do not need quad-core CPU in the master node.

But current ec2 script does not make this setting possible, since this script shares the same AMI. So, I got this error:

ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>Virtualization type 'hvm' is required for instances of type 'r3.xlarge'.</Message></Error></Errors><RequestID>b02b5ce9-116a-40bc-b994-68d106195577</RequestID></Response>
Traceback (most recent call last):
File "./spark_ec2.py", line 1176, in <module>
main()
File "./spark_ec2.py", line 1168, in main
real_main()
File "./spark_ec2.py", line 1017, in real_main
(master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
File "./spark_ec2.py", line 554, in launch_cluster
user_data=user_data_content)
File "/Users/q/Downloads/spaark/spark-1.2.0-bin-hadoop2.4/ec2/lib/boto-2.34.0/boto/ec2/image.py", line 329, in run
tenancy=tenancy, dry_run=dry_run)
File "/Users/q/Downloads/spaark/spark-1.2.0-bin-hadoop2.4/ec2/lib/boto-2.34.0/boto/ec2/connection.py", line 974, in run_instances
verb='POST')
File "/Users/q/Downloads/spaark/spark-1.2.0-bin-hadoop2.4/ec2/lib/boto-2.34.0/boto/connection.py", line 1204, in get_object
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>Virtualization type 'hvm' is required for instances of type 'r3.xlarge'.</Message></Error></Errors><RequestID>b02b5ce9-116a-40bc-b994-68d106195577</RequestID></Response>

So, If you agree with this, I will make a patch to make this possible.

Issue Links

• duplicates: SPARK-1805 Error launching cluster when master and slave machines are of different virtualization types
• is related to: SPARK-6935 spark/spark-ec2.py add parameters to give different instance types for master and slaves