[SPARK-13802] Fields order in Row(**kwargs) is not consistent with Schema.toInternal method

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Szymon Matejczyk
• Created: 10/Mar/16 13:55
• Updated: 12/Dec/22 18:10
• Resolved: 03/May/17 12:12

Description

When using Row constructor from kwargs, fields in the tuple underneath are sorted by name. When Schema is reading the row, it is not using the fields in this order.

from pyspark.sql import Row
from pyspark.sql.types import *

schema = StructType([
    StructField("id", StringType()),
    StructField("first_name", StringType())])
row = Row(id="39", first_name="Szymon")
schema.toInternal(row)
Out[5]: ('Szymon', '39')

df = sqlContext.createDataFrame([row], schema)
df.show(1)

+------+----------+
|    id|first_name|
+------+----------+
|Szymon|        39|
+------+----------+

Issue Links

• duplicates: SPARK-12467 Get rid of sorting in Row's constructor in pyspark