[SPARK-37244] Build and test on Python 3.10

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Labels: releasenotes
• Target Version/s: 3.3.0
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 08/Nov/21 21:01
• Updated: 17/Mar/22 21:17
• Resolved: 09/Nov/21 04:10

Description

Python 3.10 introduced breaking changes. We need to update PySpark.

• https://docs.python.org/3/whatsnew/3.10.html

For example, the following.

>>> from collections import Callable
<stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

This is targeting Apache Spark 3.3.0 in 2022.

Issue Links

• is duplicated by: SPARK-37249 ImportError: cannot import name 'Callable' from 'collections'
• is related to: SPARK-37257 Update setup.py for Python 3.10
• relates to: SPARK-37253 try_simplify_traceback should not fail when tb_frame.f_lineno is None
• links to: [Github] Pull Request #34526 (dongjoon-hyun)
• links to: [Github] Pull Request #34529 (dongjoon-hyun)