[SPARK-49690] UDT type is not expanded into its StructType in the schema definition

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Labels: SQL pull-request-available
• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Asif
• Created: 18/Sep/24 01:57
• Updated: 11/Dec/24 22:02

Description

A UDT type field does not show up as constituent struct type in the schema. Instead it shows up as UserDefinedType class .

For eg: it shows up as

root
 |-- message: test (nullable = true)

 

But mesage is a field of type UDT with schema represented as 

StructField("intField", IntegerType, nullable = false),
StructField("stringField", StringType, nullable = false)))

so message field should be a struct type, with schema as

root
 |-- message: struct (nullable = true)
 |    |-- intField: integer (nullable = false)
 |    |-- stringField: string (nullable = false)

 

will be opening a PR and along with bug test

Issue Links

• links to: GitHub Pull Request #48174