[SPARK-34629] Python type hints improvement

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.1.2
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Epic Link: Project Zen
• Assignee: Maciej Szymkiewicz
• Reporter: Hyukjin Kwon
• Shepherd: Maciej Szymkiewicz
• Created: 05/Mar/21 02:13
• Updated: 12/Dec/22 18:10
• Resolved: 23/Dec/21 00:17

Description

We added PySpark type hints at SPARK-32681

However, looks like there are still many missing APIs to type. I maintain a project called [Koalas](https://github.com/databricks/koalas), and I found these errors https://gist.github.com/HyukjinKwon/9faabc5f2680b56007d71ef7cf0ad400
For example, pyspark._version_ and pyspark.sql.Column.contains are missing in the type hints.

I believe this is the same case to other projects that enables mypy in their project (presumably also given SPARK-34544).

This umbrella JIRA targets to identify such cases and improve Python type hints in PySpark.

Sub-Tasks

• Add type hints of pyspark.__version__ and pyspark.sql.Column.contains: Resolved
• pyspark toPandas() should return pd.DataFrame: Resolved
• Add type hints of pyspark.sql.column.* and pyspark.sql.types.*: Resolved
• Improve type hints on pyspark.sql.*: Resolved