[SPARK-36145] Remove Python 3.6 support in codebase and CI

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.3.0
• Component/s: PySpark
• Labels: release-notes
• Target Version/s: 3.3.0
• Assignee: Maciej Szymkiewicz
• Reporter: Hyukjin Kwon
• Created: 15/Jul/21 01:35
• Updated: 12/Dec/22 18:11
• Resolved: 09/Feb/22 13:01

Description

Python 3.6 is deprecated via SPARK-35938 at Apache Spark 3.2. We should remove it in Spark 3.3.

This JIRA also target to all the changes in CI and development not only user-facing changes.

Issue Links

• is related to: SPARK-36845 Inline type hint files for files in python/pyspark/sql

Sub-Tasks

• Use Python 3.9 in `run-pip-tests` conda environment: Resolved
• Remove Python 3.6 from `pyspark` GitHub Action job: Resolved
• Upgrade Python version from 3.6 to higher version in GitHub linter: Resolved
• Fix lint-python to work with Python 3.9: Resolved
• Update docs to reflect that Python 3.6 is no longer supported: Resolved
• Drop code targetting Python < 3.7: Resolved
• Remove unnecessary usages of OrderedDict: Resolved