[SPARK-45981] Improve Python language test coverage

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Project Infra, Tests
• Labels: releasenotes
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 17/Nov/23 20:46
• Updated: 19/Nov/23 09:17
• Resolved: 18/Nov/23 05:31

Description

This umbrella Jira aims to improve Apache Spark 4 test coverage across various Python language versions .

• PySpark
• Spark Connect Python Client

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0

Sub-Tasks

• Add Python GitHub Action Daily Job: Resolved
• Add `Python 3.10` to Infra docker image: Resolved
• Add `Python 3.10` to the Daily Python Github Action job: Resolved
• Add `Python 3.11` to Infra docker image: Resolved
• Add `Python 3.11` to the Daily Python Github Action job: Resolved
• Fix `pyspark.sql.tests.connect.test_connect_basic` in Python 3.11: Resolved
• Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11: Resolved
• Fix `pyspark.pandas.tests.connect.computation.test_parity_apply_func` in Python 3.11: Resolved
• Fix `pyspark.ml.torch.tests.test_distributor` in Python 3.11: Resolved
• Upgrade python to 3.11 for non-python tests: Resolved
• Add `Python 3.12` to Infra docker image: Resolved
• Add `Python 3.12` to the Daily Python Github Action job: Resolved
• Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite: Resolved
• Install `six==1.16.0` explicitly for `pandas` in Python 3.12: Resolved
• Remove `unittest` deprecated alias usage for Python 3.12: Resolved
• Install `torch` nightly only at Python 3.12 in Infra docker image: Resolved
• Upgrade Cloudpickle to 3.0.0: Resolved
• Upgrade `pytorch` for Python 3.12: Resolved
• Fix the doctest in pyspark.pandas.frame.DataFrame.to_dict (Python 3.12): Resolved
• Flaky `pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault` with Python 3.12: Resolved
• Install torchvision for Python 3.12 build: Resolved
• Fix the doctest in pyspark.pandas.series.Series.to_dict (Python 3.12): Resolved
• Fix pyspark.pandas.mlflow.load_model test (Python 3.12): Resolved
• Skip `TorchDistributorLocalUnitTests.test_end_to_end_run_locally` with Python 3.12: Resolved
• Skip 'CrossValidatorTests.test_crossvalidator_with_fold_col' with Python 3.12: Resolved
• Split scheduled Python build: Resolved
• Upgrade memory-profiler>=0.61.0 for Python 3.12: Resolved
• Run Python 3.11 and 3.12 test independently: Resolved
• Install `lxml` in Python 3.12: Resolved
• `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available: Resolved
• Disable `fail-fast` in Python CI: Resolved
• Exclude unittest-xml-reporting in Python 3.12 image: Resolved
• Add unittest-xml-reporting into Python 3.12 image: Resolved
• Add `Python 3.13` to Infra docker image: Resolved
• Add `Python 3.13` Daily Python Github Action job: Resolved
• Add NumPy in Python 3.13 image: Resolved
• Use `grpcio*` 1.67.0 in Python 3.13 image: Resolved
• Install BASIC_PIP_PKGS except `pyarrow` in Python 3.13 image: Resolved
• Make pysaprk-connect tests passing without optional dependencies: Resolved
• Make pysaprk-ml-connect tests passing without optional dependencies: Resolved
• Upgrade PyPy CI to 3.10: Resolved
• Add 'Python 3.9` to the Daily Python GitHub Action job: Resolved
• Use `PyArrow>=18.0.0` for Python 3.13: Resolved
• Use latest Python dependencies for Python client3.5 <> 4.0server job: Resolved
• Use Spark 3.4.4 instead of 3.0.1 in `test_install_spark`: Resolved
• Skip FeatureTests if torch is not installed: Resolved
• Skip PipelineTests if torch is not installed: Resolved
• Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13: Open
• Reeanble `pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault` with Python 3.12: Open
• Remove unnecessary installation `miniconda` from `build_and_test.yml`: Resolved