[SPARK-41814] Column.eqNullSafe fails on NaN comparison

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Component/s: Connect
• Assignee: Ruifeng Zheng
• Reporter: Hyukjin Kwon
• Created: 02/Jan/23 08:22
• Updated: 03/Jan/23 12:03
• Resolved: 03/Jan/23 11:54

Description

File "/.../spark/python/pyspark/sql/connect/column.py", line 115, in pyspark.sql.connect.column.Column.eqNullSafe
Failed example:
    df2.select(
        df2['value'].eqNullSafe(None),
        df2['value'].eqNullSafe(float('NaN')),
        df2['value'].eqNullSafe(42.0)
    ).show()
Expected:
    +----------------+---------------+----------------+
    |(value <=> NULL)|(value <=> NaN)|(value <=> 42.0)|
    +----------------+---------------+----------------+
    |           false|           true|           false|
    |           false|          false|            true|
    |            true|          false|           false|
    +----------------+---------------+----------------+
Got:
    +----------------+---------------+----------------+
    |(value <=> NULL)|(value <=> NaN)|(value <=> 42.0)|
    +----------------+---------------+----------------+
    |            true|          false|           false|
    |           false|          false|            true|
    |            true|          false|           false|
    +----------------+---------------+----------------+

Issue Links

• links to: [Github] Pull Request #39360 (zhengruifeng)
• links to: [Github] Pull Request #39360 (zhengruifeng)
• links to: [Github] Pull Request #39363 (HyukjinKwon)
• links to: [Github] Pull Request #39363 (HyukjinKwon)