[SPARK-44734] Add documentation for type casting rules in Python UDFs/UDTFs

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Allison Wang
• Created: 09/Aug/23 02:27
• Updated: 19/Oct/23 06:52

Description

In addition to type mappings between Spark data types and Python data types (SPARK-44733), we should add the type casting rules for regular and arrow-optimized Python UDFs/UDTFs. 

We currently have this table in code:

• Arrow: https://github.com/apache/spark/blob/master/python/pyspark/sql/pandas/functions.py#L311-L329
• Python UDF: https://github.com/apache/spark/blob/master/python/pyspark/sql/udf.py#L101-L116

We should add a proper documentation page for the type casting rules.