[SPARK-41815] Column.isNull returns nan instead of None

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Ruifeng Zheng
• Reporter: Hyukjin Kwon
• Created: 02/Jan/23 08:25
• Updated: 04/Jan/23 23:52
• Resolved: 04/Jan/23 23:52

Description

File "/.../spark/python/pyspark/sql/connect/column.py", line 99, in pyspark.sql.connect.column.Column.isNull
Failed example:
    df.filter(df.height.isNull()).collect()
Expected:
    [Row(name='Alice', height=None)]
Got:
    [Row(name='Alice', height=nan)]

Issue Links

• links to: [Github] Pull Request #39386 (zhengruifeng)
• links to: [Github] Pull Request #39386 (zhengruifeng)
• links to: [Github] Pull Request #39386 (zhengruifeng)