[SPARK-36740] collection operators should handle duplicated NaN

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Labels: correctness
• Assignee: angerszhu
• Reporter: angerszhu
• Created: 13/Sep/21 12:28
• Updated: 27/Sep/21 04:44
• Resolved: 27/Sep/21 04:44

Description

collection operators should handle duplicated NaN, current OpenHashSet can't handle duplicated NaN

Issue Links

• links to: [Github] Pull Request #34008 (AngersZhuuuu)
• links to: [Github] Pull Request #34008 (AngersZhuuuu)

Sub-Tasks

• array_union should not produce duplicated result: Resolved
• array_distinct should not return duplicated NaN: Resolved
• array_except should handle Double.NaN and Float.NaN: Resolved
• array_intersect should handle Double.NaN and Float.NaN: Resolved
• array_overlap should handle Double.NaN and Float.NaN: Resolved
• ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT: Resolved
• Inset should handle Double.NaN and Float.NaN: Resolved