[SPARK-33489] Support null for conversion from and to Arrow type

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Yuya Kanai
• Created: 19/Nov/20 12:06
• Updated: 12/Dec/22 18:10
• Resolved: 25/Jan/21 02:37

Description

I got below error when using from_arrow_type() in pyspark.sql.pandas.types

Unsupported type in conversion from Arrow: null

I noticed NullType exists under pyspark.sql.types so it seems possible to convert from pyarrow null to pyspark null type and vice versa.

Issue Links

• is related to: SPARK-21187 Complete support for remaining Spark data types in Arrow Converters
• links to: [Github] Pull Request #30832 (Cactice)
• links to: [Github] Pull Request #31285 (ueshin)