[SPARK-16981] For CSV files nullValue is not respected for Date/Time data type

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Lev
• Created: 09/Aug/16 22:06
• Updated: 22/Nov/16 15:25
• Resolved: 22/Nov/16 15:25

Description

Test case
val struct = StructType(Seq(StructField("col1", StringType, true),StructField("col2", TimestampType, true), Seq(StructField("col3", StringType, true)))

val cq = sqlContext.readStream
.format("csv")
.option("nullValue", " ")
.schema(struct)
.load(s"somepath")
.writeStream(....)

content of the file
"abc", ,"def"

Result:
Exception is thrown:
scala.MatchError: java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff] (of class java.lang.IllegalArgumentException)

Code analysis:
Problem is caused by code in castTo method of CSVTypeCast object
For all data types except temporal there is the following check:
if (datum == options.nullValue && nullable) {
null
}
But for temporal types it is missing

Issue Links

• relates to: SPARK-16460 Spark 2.0 CSV ignores NULL value in Date format