[SPARK-41745] SparkSession.createDataFrame does not respect the column names in the row

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Ruifeng Zheng
• Reporter: Hyukjin Kwon
• Created: 28/Dec/22 12:12
• Updated: 02/Jan/23 08:37
• Resolved: 31/Dec/22 01:04

Description

File "/.../spark/python/pyspark/sql/connect/group.py", line 173, in pyspark.sql.connect.group.GroupedData.pivot
Failed example:
    df1.show()
Differences (ndiff with -expected +actual):
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    - |course|year|earnings|
    + |    _1|  _2|   _3|
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    - |dotNET|2012|   10000|
    ?              ---
    + |dotNET|2012|10000|
    - |  Java|2012|   20000|
    ?              ---
    + |  Java|2012|20000|
    - |dotNET|2012|    5000|
    ?               ---
    + |dotNET|2012| 5000|
    - |dotNET|2013|   48000|
    ?              ---
    + |dotNET|2013|48000|
    - |  Java|2013|   30000|
    ?              ---
    + |  Java|2013|30000|
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    + <BLANKLINE>

Issue Links

• is related to: SPARK-41810 SparkSession.createDataFrame does not respect the column names in the dictionary
• links to: [Github] Pull Request #39313 (zhengruifeng)
• links to: [Github] Pull Request #39330 (grundprinzip)
• links to: [Github] Pull Request #39330 (grundprinzip)
• links to: [Github] Pull Request #39342 (HyukjinKwon)
• links to: [Github] Pull Request #39342 (HyukjinKwon)