[SPARK-35525] Define UDTs in schemas using string format

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: SQL
• Labels: SparkSQL UDF UDT schema
• Assignee: Unassigned
• Reporter: Julian Shalaby
• Created: 26/May/21 02:16
• Updated: 26/May/21 06:31

Description

In PySpark where UDTs are public in 3.1.1 for example, you can define a schema using UDTs in the format:

schema = StructType([StructField("Stuff", MyUDT())])

but the format

schema = "Stuff MyUDT"

does not work.

UDTs are officially being made public again in 3.2.0 for Scala, so this issue is pretty important now.