Minor Description
It would be very useful to be able to specify the EC2 VPC in which the Spark cluster should be created.
When creating a Spark cluster on AWS via the spark-ec2 script there is no way to specify a VPC id of the VPC you would like the cluster to be created in. The script always creates the cluster in the default VPC.
In my case I have deleted the default VPC and the spark-ec2 script errors out with the following :
Setting up security groups...
Creating security group test-master
ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>VPCIdNotSpecified</Code><Message>No default VPC for this user</Message></Error></Errors><RequestID>312a2281-81a1-4d3c-ba10-0593a886779d</RequestID></Response>
Traceback (most recent call last):
File "./spark_ec2.py", line 860, in <module>
main()
File "./spark_ec2.py", line 852, in main
real_main()
File "./spark_ec2.py", line 735, in real_main
conn, opts, cluster_name)
File "./spark_ec2.py", line 247, in launch_cluster
master_group = get_or_make_group(conn, cluster_name + "-master")
File "./spark_ec2.py", line 143, in get_or_make_group
return conn.create_security_group(name, "Spark EC2 group")
File "/home/dawson/Develop/spark-1.0.2/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/ec2/connection.py", line 2011, in create_security_group
File "/home/dawson/Develop/spark-1.0.2/ec2/third_party/boto-2.4.1.zip/boto-2.4.1/boto/connection.py", line 925, in get_object
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>VPCIdNotSpecified</Code><Message>No default VPC for this user</Message></Error></Errors><RequestID>312a2281-81a1-4d3c-ba10-0593a886779d</RequestID></Response>
Attachments
Issue Links
- is blocked by
-
SPARK-4890 Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it
-
- Resolved
-
- is duplicated by
-
SPARK-2611 VPC Issue while creating an ec2 cluster
-
- Resolved
-
- links to
