[SPARK-16462] Spark 2.0 CSV does not cast null values to certain data types properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Assignee: Liwei Lin(Inactive)
• Reporter: Liwei Lin(Inactive)
• Created: 09/Jul/16 14:08
• Updated: 08/Oct/16 14:34
• Resolved: 18/Sep/16 18:26

Description

When casting given string datum to specified type, CSV should return null for nullable types if datum == options.nullValue.

However, for certain data types like Boolean, TimestampType, DateType, CSV in 2.0 does not return null for some "empty" datum. This is a regression comparing to 1.6.

Issue Links

• is duplicated by: SPARK-16903 nullValue in first field is not respected by CSV source when read
• is duplicated by: SPARK-17039 cannot read null dates from csv file
• is duplicated by: SPARK-17290 Spark CSVInferSchema does not always respect nullValue settings
• is duplicated by: SPARK-15144 option nullValue for CSV data source not working for several types.
• is related to: SPARK-16460 Spark 2.0 CSV ignores NULL value in Date format
• relates to: SPARK-14143 Options for parsing NaNs, Infinity and nulls for numeric types
• links to: [Github] Pull Request #14118 (lw-lin)