Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25079

[PYTHON] upgrade python 3.4 -> 3.6

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.1
    • Fix Version/s: 3.0.0
    • Component/s: Build, PySpark
    • Labels:
      None

      Description

      for the impending arrow upgrade (https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python 3.4 -> 3.6.

      i have been testing this here:  https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/

      my methodology:

      1) upgrade python + arrow to 3.5 and 0.10.0

      2) run python tests

      3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and upgrade centos workers to python3.5

      4) simultaneously do the following: 

        - create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that points to python3.5 (this is currently being tested here:  https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)

        - push a change to python/run-tests.py replacing 3.4 with 3.5

      5) once the python3.5 change to run-tests.py is merged, we will need to back-port this to all existing branches

      6) then and only then can i remove the python3.4 -> python3.5 symlink

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shaneknapp Shane Knapp
                Reporter:
                shaneknapp Shane Knapp
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: