[SPARK-44980] Fix inherited namedtuples to work in createDataFrame

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Connect, PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 28/Aug/23 02:06
• Updated: 28/Aug/23 06:47
• Resolved: 28/Aug/23 06:47

Description

from collections import namedtuple
MyTuple = namedtuple("MyTuple", ["zz", "b", "a"])

class MyInheritedTuple(MyTuple):
    pass

df = spark.createDataFrame([MyInheritedTuple(1, 2, 3), MyInheritedTuple(11, 22, 33)])
df.collect()

[Row(zz=None, b=None, a=None), Row(zz=None, b=None, a=None)]

should be

[Row(zz=1, b=2, a=3), Row(zz=11, b=22, a=33)]