[SPARK-12121] Remote Spark init.sh called from spark_ec2.py points to incorrect prebuilt image URL

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.5.2
• Component/s: EC2
• Assignee: Unassigned
• Reporter: Andre Schumacher
• Created: 03/Dec/15 09:26
• Updated: 09/Feb/16 09:12
• Resolved: 09/Feb/16 09:12

Description

The tar ball for 1.5.2 with Hadoop 1 (the default) has the scala version appended to the file name, which leads to spark_ec2.py failing on starting up the cluster. Here is the record from the S3 contents:

<Contents>
<Key>spark-1.5.2-bin-hadoop1-scala2.11.tgz</Key>
<LastModified>2015-11-10T06:45:17.000Z</LastModified>
<ETag>"056fc68e549db27d986da707f19e39c8-4"</ETag>
<Size>234574403</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>

Maybe one could provide one without the scala suffix (default)?

A workaround is to use set the Hadoop version to a version different from 1 when calling spark-ec2.