[SPARK-17039] cannot read null dates from csv file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Barry Becker
• Created: 12/Aug/16 15:30
• Updated: 15/Aug/16 21:14
• Resolved: 13/Aug/16 08:54

Description

I see this exact same bug as reported in this stack overflow post using Spark 2.0.0 (released version).
In scala, I read a csv using

sqlContext.read
.format("csv")
.option("header", "false")
.option("inferSchema", "false")
.option("nullValue", "?")
.option("dateFormat", "yyyy-MM-dd'T'HH:mm:ss")
.schema(dfSchema)
.csv(dataFile)
The data contains some null dates (represented with ?).
The error I get is:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 1 times, most recent failure: Lost task 0.0 in stage 8.0 (TID 10, localhost): java.text.ParseException: Unparseable date: "?"

Issue Links

• duplicates: SPARK-16462 Spark 2.0 CSV does not cast null values to certain data types properly
• is duplicated by: SPARK-17040 cannot read null dates from csv file