[SPARK-31408] Build Spark’s own datetime pattern definition

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yuanjian Li
• Created: 10/Apr/20 07:40
• Updated: 12/Jun/20 03:36
• Resolved: 10/Apr/20 08:01

Description

This is an umbrella ticket for building Spark's own Datetime patterns and related works.

In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.

However, there are some datetime patterns not compatible between Java 8 and Java 7 APIs, and it's fragile to rely on the JDK API to define Spark's behavior. We should build our own Datetime patterns, which is compatible with Spark 2.4 (the old Java 7 `SimpleDateFormat` API).