[SPARK-8821] The ec2 script doesn't run on python 3 with an utf8 env

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: EC2

Environment

Archlinux, UTF8 LANG env

• Flags: Patch
• Assignee: Simon Hafner
• Reporter: Simon Hafner
• Created: 04/Jul/15 05:39
• Updated: 09/Jul/15 11:36
• Resolved: 07/Jul/15 16:43

Description

Otherwise the script will crash with

• Downloading boto...
  Traceback (most recent call last):
  File "ec2/spark_ec2.py", line 148, in <module>
  setup_external_libs(external_libs)
  File "ec2/spark_ec2.py", line 128, in setup_external_libs
  if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
  File "/usr/lib/python3.4/codecs.py", line 319, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

In case of an utf8 env setting.

Issue Links

• links to: [Github] Pull Request #7215 (reactormonk)