[SPARK-30788] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 11/Feb/20 09:47
• Updated: 08/Mar/22 18:55
• Resolved: 12/Feb/20 12:24

Description

To be absolutely sure that Spark 3.0 is compatible with 2.4 when spark.sql.legacy.timeParser.enabled is set to true, need to support SimpleDateFormat and FastDateFormat as legacy parsers/formatters in TimestampFormatter.

Spark 2.4.x uses the following parsers for parsing/formatting date/timestamp strings:

1. DateTimeFormat in CSV/JSON datasource
2. SimpleDateFormat - is used in JDBC datasource, in partitions parsing.
3. SimpleDateFormat in strong mode (lenient = false). It is used by the date_format, from_unixtime, unix_timestamp and to_unix_timestamp functions.

Spark 3.0 should use the same parsers in those cases.

Issue Links

• links to: GitHub Pull Request #27524