[SPARK-23446] Explicitly check supported types in toPandas

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 16/Feb/18 07:45
• Updated: 12/Dec/22 17:50
• Resolved: 16/Feb/18 17:42

Description

See:

spark.conf.set("spark.sql.execution.arrow.enabled", "false")
df = spark.createDataFrame([[bytearray("a")]])
df.toPandas()
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
df.toPandas()

     _1
0  [97]
  _1
0  a

We didn't finish binary type in Arrow conversion at Python side. We should disallow it.
Same thine applies to nested timestamps.

Issue Links

• relates to: SPARK-23380 Adds a conf for Arrow fallback in toPandas/createDataFrame with Pandas DataFrame
• links to: [Github] Pull Request #20625 (HyukjinKwon)