[SPARK-12268] pyspark shell uses execfile which breaks python3 compatibility

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.2, 1.6.0
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: PySpark
• Assignee: Erik Selin
• Reporter: Erik Selin
• Created: 10/Dec/15 20:34
• Updated: 13/Jan/16 20:23
• Resolved: 13/Jan/16 20:22

Description

The pyspark shell allows custom start scripts to run using the PYTHONSTARTUP environment variable. The value specified there will get run at the end of the shell startup by a call to execfile. However, execfile is deprecated in python3 and thus this does not work for python3 users. The simply fix is to follow the 2to3 recommendation and read, compile and exec the file manually as per this PR: https://github.com/apache/spark/pull/10255

Issue Links

• links to: Fix