[SPARK-7900] Reduce number of tagging calls in spark-ec2

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.4.0
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Nicholas Chammas
• Created: 27/May/15 16:02
• Updated: 03/Jun/15 18:31
• Resolved: 03/Jun/15 18:31

Description

spark-ec2 currently tags each instance with its own name:

https://github.com/apache/spark/blob/4615081d7a10b023491e25478d19b8161e030974/ec2/spark_ec2.py#L684-L692

Quite often, one of these tagging calls will fail:

Launching instances...
Launched 10 slaves in us-west-2a, regid = r-89656e83
Launched master in us-west-2a, regid = r-07646f0d
Waiting for AWS to propagate instance metadata...
Traceback (most recent call last):
  File "../spark/ec2/spark_ec2.py", line 1395, in <module>
    main()
  File "../spark/ec2/spark_ec2.py", line 1387, in main
    real_main()
  File "../spark/ec2/spark_ec2.py", line 1222, in real_main
    (master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
  File "../spark/ec2/spark_ec2.py", line 667, in launch_cluster
    value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
  File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py",> line 80, in add_tag
    self.add_tags({key: value}, dry_run)
  File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/ec2object.py",> line 97, in add_tags
    dry_run=dry_run
  File "</path/spark/ec2/lib/boto-2.34.0/boto/ec2/connection.py",> line 4202, in create_tags
    return self.get_status('CreateTags', params, verb='POST')
  File "</path/spark/ec2/lib/boto-2.34.0/boto/connection.py",> line 1223, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidInstanceID.NotFound</Code><Message>The instance ID 'i-d3b72524' does not exist</Message></Error></Errors><RequestID>f0936ab5-4d10-46d1-a35d-cefaf8a68adc</RequestID></Response>

This is presumably a problem with AWS metadata taking time to become available on all the servers that spark-ec2 hits as it makes the several tagging calls.

Instead of retrying the tagging calls, we should just reduce them to 2 calls--one for the master, one for the slaves.

Issue Links

• duplicates: SPARK-4983 Add sleep() before tagging EC2 instances to allow instance metadata to propagate