[SPARK-6220] Allow extended EC2 options to be passed through spark-ec2

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Nicholas Chammas
• Created: 09/Mar/15 03:15
• Updated: 23/Jun/15 21:33
• Resolved: 23/Jun/15 21:24

Description

There are many EC2 options exposed by the boto library that spark-ec2 uses.

Over time, many of these EC2 options have been bubbled up here and there to become spark-ec2 options.

Examples:

• spot prices
• placement groups
• VPC, subnet, and security group assignments

It's likely that more and more EC2 options will trickle up like this to become spark-ec2 options.

While major options are well suited to this type of promotion, we should probably allow users to pass through EC2 options they want to use through spark-ec2 in some generic way.

Let's add two options:

• --ec2-instance-option -> boto::run
• --ec2-spot-instance-option -> boto::request_spot_instances

Each option can be specified multiple times and is simply passed directly to the underlying boto call.

For example:

spark-ec2 \
    ...
    --ec2-instance-option "instance_initiated_shutdown_behavior=terminate" \
    --ec2-instance-option "ebs_optimized=True"

I'm not sure about the exact syntax of the extended options, but something like this will do the trick as long as it can be made to pass the options correctly to boto in most cases.

I followed the example of ssh, which supports multiple extended options similarly.

ssh -o LogLevel=ERROR -o UserKnowHostsFile=/dev/null ...

Issue Links

• is related to: SPARK-8576 Add spark-ec2 options to assign launched instances into IAM roles and to set instance-initiated shutdown behavior