[SPARK-4890] Upgrade Boto to 2.34.0; automatically download Boto from PyPi instead of packaging it

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: EC2
• Target Version/s: 1.3.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 18/Dec/14 21:41
• Updated: 23/Dec/14 04:12
• Resolved: 20/Dec/14 01:02

Description

We should upgrade to a newer version of Boto (2.34.0), since this is blocking several features. It looks like newer versions of Boto don't work properly when they're loaded from a zipfile since they try to read a JSON file from a path relative to the Boto library sources.

Therefore, I think we should change spark-ec2 to automatically download Boto from PyPi if it's not present in SPARK_EC2_DIR/lib, similar to what we do in the sbt/sbt scripts. This shouldn't ben an issue for users since they already need to have an internet connection to launch an EC2 cluster. By performing the downloading in spark_ec2.py instead of the Bash script, this should also work for Windows users.

Issue Links

• blocks: SPARK-3405 EC2 cluster creation on VPC
• blocks: SPARK-4241 spark_ec2.py support China AWS region: cn-north-1
• links to: [Github] Pull Request #3737 (JoshRosen)
• links to: [Github] Pull Request #3770 (nchammas)