[SPARK-44733] Add user guide for type mappings between Spark and Python data types

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Philip Dakin
• Reporter: Allison Wang
• Created: 09/Aug/23 02:21
• Updated: 14/Nov/23 03:43
• Resolved: 14/Nov/23 02:26

Description

Currently, the PySpark documentation does not cover the data type mapping between Spark data types and Python data types. This mapping can be useful for users of Python UDFs/UDTFs.

There's a document detailing type mapping in the Spark documentation: https://spark.apache.org/docs/3.4.1/sql-ref-datatypes.html

We should create a dedicated documentation page dedicated to type mappings.

Issue Links

• links to: GitHub Pull Request #43369