[SPARK-9794] ISO DateTime parser is too strict

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.2, 1.3.1, 1.4.1, 1.5.0
• Fix Version/s: 1.6.0
• Component/s: SQL
• Assignee: Kevin Cox
• Reporter: Alex Angelini
• Created: 10/Aug/15 20:04
• Updated: 16/Sep/16 08:26
• Resolved: 16/Sep/15 22:30

Description

The DateTime parser requires 3 millisecond digits, but that is not part of the official ISO8601 spec.

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L132

https://en.wikipedia.org/wiki/ISO_8601

This results in the following exception when trying to parse datetime columns

java.text.ParseException: Unparseable date: "0001-01-01T00:00:00GMT-00:00"

joshrosen rxin

Issue Links

• is duplicated by: SPARK-6385 ISO 8601 timestamp parsing does not support arbitrary precision second fractions
• links to: [Github] Pull Request #8396 (kevincox)