[SPARK-26651] Use Proleptic Gregorian calendar

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 2.4.0
• Component/s: SQL
• Labels: release-notes
• Assignee: Maxim Gekk
• Reporter: Maxim Gekk
• Created: 17/Jan/19 20:27
• Updated: 25/Jun/20 20:27
• Resolved: 22/Feb/19 22:58

Description

Spark 2.4 and previous versions use a hybrid calendar - Julian + Gregorian in date/timestamp parsing, functions and expressions. The ticket aims to switch Spark on Proleptic Gregorian calendar, and use java.time classes introduced in Java 8 for timestamp/date manipulations. One of the purpose of switching on Proleptic Gregorian calendar is to conform to SQL standard which supposes such calendar.

Release note:

Spark 3.0 has switched on Proleptic Gregorian calendar in parsing, formatting, and converting dates and timestamps as well as in extracting sub-components like years, days and etc. It uses Java 8 API classes from the java.time packages that based on ISO chronology . Previous versions of Spark performed those operations by using the hybrid calendar (Julian + Gregorian). The changes might impact on the results for dates and timestamps before October 15, 1582 (Gregorian).

Issue Links

• is related to: SPARK-31702 Old POSIXlt, POSIXct and Date become corrupt due to calendar difference
• is related to: SPARK-30688 Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF
• is related to: SPARK-31404 file source backward compatibility after calendar switch
• is related to: SPARK-30760 Port `millisToDays` and `daysToMillis` on Java 8 time API
• relates to: SPARK-18381 Wrong date conversion between spark and python for dates before 1583
• relates to: SPARK-30951 Potential data loss for legacy applications after switch to proleptic Gregorian calendar
• links to: GitHub Pull Request #23722

Sub-Tasks

• Make typed Timestamp/Date literals consistent to casting: Resolved
• Use Proleptic Gregorian calendar in casting UTF8String to date/timestamp types: Resolved
• Caching of DateTimeFormatter: Resolved
• Get rid of spark.sql.legacy.timeParser.enabled: Resolved
• Cast date/timestamp by Date/TimestampFormatter: Resolved
• Use java.time API in timestamp/date functions: Resolved
• CSV schema inferring does not respect spark.sql.legacy.timeParser.enabled: Resolved
• Support new date/timestamp parser in HadoopFsRelationTest: Resolved
• Use java.time API for parsing timestamps and dates from JSON: Resolved
• Use java.time API for parsing timestamps and dates from CSV: Resolved
• Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings: Resolved
• Use Proleptic Gregorian calendar in parsing JDBC lower/upper bounds: Resolved
• Use Timestamp/DateFormatter in CatalogColumnStat: Resolved
• Benchmark for date/time functions and expressions: Resolved
• Port DayWeek, DayOfWeek and WeekDay on Proleptic Gregorian calendar: Resolved
• Get rid of java.util.Calendar in DateTimeUtils: Resolved
• Use java.time API in timestampAddInterval: Resolved
• Benchmark the performance regression: Open