[SPARK-5627] Enhance spark-ec2 to return machine-readable output

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Nicholas Chammas
• Created: 05/Feb/15 20:18
• Updated: 09/Jan/16 12:52
• Resolved: 09/Jan/16 12:52

Description

There are some cases where users may want to programmatically invoke spark-ec2 to manage clusters. For example, we might want to programmatically launch clusters as part of a spark-perf run and then destroy them once the performance testing is done.

We should support some of these use cases, perhaps with the explicit caveat that we are not offering API stability for this access.

A good way to offer this might be to follow the Packer model and add a --machine-readable option to spark-ec2.

It would be a lot of work to support such an option for everything that spark-ec2 does, and it probably isn't relevant for most things anyway.

Still, we can phase it in for select things like returning the version and describing clusters.

Issue Links

• relates to: SPARK-5628 Add option to return spark-ec2 version
• relates to: SPARK-5629 Add spark-ec2 action to return info about an existing cluster