Description
Currently spark use Calendar to build the Date when convert from string to Date. But Calendar can not detect the invalid date format (e.g. 2011-02-29).
Although we can use Calendar.setLenient(false) to enable Calendar to detect the invalid date format, but found the error message very confusing. So I suggest to use joda's DateTime to replace Calendar.
Besides that, I found that there's already some format checking logic when casting string to date. And if it is invalid format, it would return None. I don't think it make sense to just return None without telling users. I think by default should just throw exception, and user can set property to allow it return None if invalid format.
if (i == 0 && j != 4) { // year should have exact four digits return None }
Attachments
Issue Links
- relates to
-
SPARK-15379 String to date and timestamp conversion should return null on some invalid date string
- Resolved