[SPARK-925] Allow ec2 scripts to load default options from a json file

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.8.0
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Shay Seng
• Created: 08/Oct/13 10:58
• Updated: 16/Jan/16 13:00
• Resolved: 16/Jan/16 13:00

Description

The option list for ec2 script can be a little irritating to type in, especially things like path to identity-file, region , zone, ami etc.
It would be nice if ec2 script looks for an options.json file in the following order: (1) CWD, (2) ~/spark-ec2, (3) same dir as spark_ec2.py

Something like:
def get_defaults_from_options():

1. Check to see if a options.json file exists, if so load it.
2. However, values in the options.json file can only overide values in opts
3. if the Opt values are None or ""
4. i.e. commandline options take presidence
   defaults = {'aws-access-key-id':'','aws-secret-access-key':'','key-pair':'', 'identity-file':'', 'region':'ap-southeast-1', 'zone':'', 'ami':'','slaves':1, 'instance-type':'m1.large'}

1. Look for options.json in directory cluster was called from
2. Had to modify the spark_ec2 wrapper script since it mangles the pwd
   startwd = os.environ['STARTWD']
   if os.path.exists(os.path.join(startwd,"options.json")):
   optionspath = os.path.join(startwd,"options.json")
   else:
   optionspath = os.path.join(os.getcwd(),"options.json")

try:
print "Loading options file: ", optionspath
with open (optionspath) as json_data:
jdata = json.load(json_data)
for k in jdata:
defaults[k]=jdata[k]
except IOError:
print 'Warning: options.json file not loaded'

1. Check permissions on identity-file, if defined, otherwise launch will fail late and will be irritating
   if defaults['identity-file']!='':
   st = os.stat(defaults['identity-file'])
   user_can_read = bool(st.st_mode & stat.S_IRUSR)
   grp_perms = bool(st.st_mode & stat.S_IRWXG)
   others_perm = bool(st.st_mode & stat.S_IRWXO)
   if (not user_can_read):
   print "No read permission to read ", defaults['identify-file']
   sys.exit(1)
   if (grp_perms or others_perm):
   print "Permissions are too open, please chmod 600 file ", defaults['identify-file']
   sys.exit(1)

1. if defaults contain AWS access id or private key, set it to environment.
2. required for use with boto to access the AWS console
   if defaults['aws-access-key-id'] != '':
   os.environ['AWS_ACCESS_KEY_ID']=defaults['aws-access-key-id']
   if defaults['aws-secret-access-key'] != '':
   os.environ['AWS_SECRET_ACCESS_KEY'] = defaults['aws-secret-access-key']

return defaults

Issue Links

• links to: [Github] Pull Request #7906 (marmbrus)