[SPARK-32681] PySpark type hints support

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.1.0
• Component/s: PySpark
• Target Version/s: 3.1.0
• Epic Link: Project Zen
• Assignee: Maciej Szymkiewicz
• Reporter: Hyukjin Kwon
• Created: 21/Aug/20 09:00
• Updated: 12/Dec/22 18:10
• Resolved: 04/Dec/20 08:44

Description

https://github.com/zero323/pyspark-stubs demonstrates a lot of benefits to improve usability in PySpark by leveraging Python type hints.

By having the type hints in PySpark we can, for example:

• automatically document the input and output types
• leverage IDE for error detection and auto-completion
• have a cleaner definition and easier to understand.

This JIRA is an umbrella JIRA that targets to port https://github.com/zero323/pyspark-stubs and related items to smoothly run within PySpark.

It was also discussed in the dev mailing list: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html

Sub-Tasks

• Port pyspark-stubs: Resolved
• Post-port removal of non-API stubs: Resolved
• Add type hints guideliness to the documentation: Resolved
• Provide static annotatiions for pyspark.resource.* modules: Resolved
• Make pyspark interface friendly with mypy static analysis: Resolved
• Adjust mypy configuration: Resolved