[SPARK-15989] PySpark SQL python-only UDTs don't support nested types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: PySpark, SQL
• Assignee: L. C. Hsieh
• Reporter: Vladimir Feinberg
• Created: 16/Jun/16 16:46
• Updated: 02/Apr/18 18:05
• Resolved: 02/Aug/16 17:08

Description

This notebook demonstrates the bug.

The obvious issue is that nested UDTs are not supported if the UDT is Python-only. Looking at the exception thrown, this seems to be because the encoder on the Java end tries to encode the UDT as a Java class, which doesn't exist for the [[PythonOnlyUDT]].

Issue Links

• relates to: SPARK-16062 PySpark SQL python-only UDTs don't work well
• relates to: SPARK-23848 Structured Streaming fails with nested UDTs
• links to: [Github] Pull Request #13778 (viirya)