[SPARK-4325] Improve spark-ec2 cluster launch times

• Type: Umbrella
• Status: Resolved
• Priority: Minor
• Resolution: Done
• Component/s: EC2
• Assignee: Nicholas Chammas
• Reporter: Nicholas Chammas
• Created: 10/Nov/14 21:33
• Updated: 13/Jan/16 10:21
• Resolved: 13/Jan/16 10:21

Description

This is an umbrella task to capture several pieces of work related to significantly improving spark-ec2 cluster launch times.

There are several optimizations we know we can make to setup.sh to make cluster launches faster.

There are also some improvements to the AMIs that will help a lot.

Potential improvements:

• Upgrade the Spark AMIs and pre-install tools like Ganglia on them. This will reduce or eliminate SSH wait time and Ganglia init time.
• Replace instances of download; rsync to rest of cluster with parallel downloads on all nodes of the cluster.
• Replace instances of

  for node in $NODES; do
    command
    sleep 0.3
  done
  wait

  with simpler calls to pssh.

• Remove the linear backoff when we wait for SSH availability now that we are already waiting for EC2 status checks to clear before testing SSH.

Issue Links

• contains: SPARK-5189 Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master
• contains: SPARK-3821 Develop an automated way of creating Spark images (AMI, Docker, and others)
• links to: [Github] Pull Request #3195 (nchammas)
• links to: PR to replace some bash-isms in setup.sh with pssh

Sub-Tasks

• Use EC2 status checks to know when to test SSH availability: Resolved
• Use pssh instead of bash-isms and remove unnecessary operations: Resolved
• Bake common tools like ganglia into Spark AMI: Resolved