Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12268

pyspark shell uses execfile which breaks python3 compatibility

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.2, 1.6.0
    • Fix Version/s: 1.6.1, 2.0.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      The pyspark shell allows custom start scripts to run using the PYTHONSTARTUP environment variable. The value specified there will get run at the end of the shell startup by a call to execfile. However, execfile is deprecated in python3 and thus this does not work for python3 users. The simply fix is to follow the 2to3 recommendation and read, compile and exec the file manually as per this PR: https://github.com/apache/spark/pull/10255

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tyro89 Erik Selin
                Reporter:
                tyro89 Erik Selin
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: