[SPARK-5879] spary_ec2.py should expose/return master and slave lists (e.g. write to file)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Florian Verhein
• Created: 18/Feb/15 01:29
• Updated: 16/Mar/15 00:24
• Resolved: 16/Mar/15 00:24

Description

After running spark_ec2.py, it is often useful/necessary to know the master's ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline.

For example, consider a wrapper that launches a cluster, then waits for completion of some application running on it (e.g. polling via ssh), before destroying the cluster.

Some options:

• write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically a subset of the ec2_variables.sh that is temporarily created as part of deploy_files variable substitution)
• launch-variables.json (same info but as json)

Both would be useful depending on the wrapper language.

I think we should incorporate the cluster name for the case that multiple clusters are launched. E.g. <cluster_name>_variables.sh/.json

Thoughts?

Issue Links

• duplicates: SPARK-5629 Add spark-ec2 action to return info about an existing cluster