[SPARK-9076] Improve NaN value handling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 15/Jul/15 19:42
• Updated: 11/Aug/15 08:14
• Resolved: 11/Aug/15 08:14

Description

This is an umbrella ticket for handling NaN values.

For general design, please see https://issues.apache.org/jira/browse/SPARK-9079

Sub-Tasks

• Convert inbound NaN values as null: Closed
• udf7 failed due to null vs nan semantics: Resolved
• udf_asin and udf_acos test failure: Resolved
• Sorting float/double column containing NaNs can lead to "Comparison method violates its general contract!" errors: Resolved
• IsNaN expression: Resolved
• fillna/dropna should also fill/drop NaN values in addition to null values: Resolved
• Design NaN semantics: Resolved
• Equality test on NaN = NaN should return true: Resolved
• NaN should be greater than all other values: Resolved
• UnsafeRow should canonicalize NaN values: Resolved
• Add nanvl expression: Resolved