I was asked by falaki to create a jira here, previously it was reported as databricks' issue on github https://github.com/databricks/spark-csv/issues/388#issuecomment-255631718
I have problem with spark 2.0.0, spark-csv 1.5.0, and scala 2.11.8.
I have a csv file that I want to convert to parquet. There is a column with timestamps and some of them are missing, those are empty strings (without quotes, and it is not even a spacer, just new line straightaway as that is the last column). I get exception thrown:
The options I use when reading csv
The execution goes through CSVINferSchema.scala (lines 284-287) in *spark-sql_2.11-2.0.0-sources.jar*
it invokes `Date.valueOf(s)` in DateTimeUtils.scala spark-catalyst_2.11-2.0.0-sources.jar that then throws excepion in java.sql.Date.valueOf.
Is that a bug, I am doing something wrong, or there is a way to pass a default value?