[SPARK-41879] `DataFrame.collect` should support nested types

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect, PySpark
• Assignee: Apache Spark
• Reporter: Ruifeng Zheng
• Created: 04/Jan/23 07:09
• Updated: 11/Jan/23 03:36
• Resolved: 11/Jan/23 03:36

Description

File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1578, in pyspark.sql.connect.functions.struct
Failed example:
    df.select(struct('age', 'name').alias("struct")).collect()
Expected:
    [Row(struct=Row(age=2, name='Alice')), Row(struct=Row(age=5, name='Bob'))]
Got:
    [Row(struct={'age': 2, 'name': 'Alice'}), Row(struct={'age': 5, 'name': 'Bob'})]

Issue Links

• links to: [Github] Pull Request #39462 (zhengruifeng)