[SPARK-48068] `mypy` should have `--python-executable` parameter

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 4.0.0, 3.5.1, 3.3.4, 3.4.3
• Fix Version/s: 4.0.0, 3.5.2, 3.4.4
• Component/s: Project Infra, PySpark
• Labels: pull-request-available
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 01/May/24 05:57
• Updated: 01/May/24 19:31
• Resolved: 01/May/24 17:42

Description

We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not true. We need to use `mypy`'s parameter to make it sure.

https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705

root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
3428
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1

Issue Links

• is caused by: SPARK-36989 Migrate type hint data tests
• links to: GitHub Pull Request #46314