[SPARK-31404] file source backward compatibility after calendar switch

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Unassigned
• Reporter: Wenchen Fan
• Created: 10/Apr/20 03:33
• Updated: 20/Sep/22 17:18
• Resolved: 17/May/20 02:33

Description

In Spark 3.0, we switch to the Proleptic Gregorian calendar by using the Java 8 datetime APIs. This makes Spark follow the ISO and SQL standard, but introduces some backward compatibility problems:
1. may read wrong data from the data files written by Spark 2.4
2. may have perf regression

Issue Links

• relates to: SPARK-30951 Potential data loss for legacy applications after switch to proleptic Gregorian calendar
• relates to: SPARK-26651 Use Proleptic Gregorian calendar
• supercedes: SPARK-34675 TimeZone inconsistencies when JVM and session timezones are different

Sub-Tasks

• Speed up timestamps rebasing: Resolved
• Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing: Resolved
• Speed-up date-time rebasing: Resolved
• fail by default when read/write datetime values and not sure if they need rebase or not: Resolved
• Rebase datetime in parquet/avro according to file metadata: Resolved
• Incorrect rebasing of BCE dates: Resolved
• Regression in loading/saving timestamps from/to ORC files: Resolved
• Perf regression of fromJavaDate: Resolved
• Perf regression of toJavaDate: Resolved
• Fix rebasing of not-existed dates/timestamps: Resolved
• Skip timestamp rebasing after 1900-01-01: Resolved
• Incorrect days conversion by JSON legacy parser: Resolved
• Reading wrong dates from dictionary encoded columns in Parquet files: Resolved
• Reading wrong timestamps from dictionary encoded columns in Parquet files: Resolved
• Check reading date/timestamp from Parquet: plain, dictionary enc, Spark version: Resolved
• Check reading date/timestamp from Avro files w/ and w/o Spark version: Resolved
• Check the metadata key 'org.apache.spark.legacyDateTime' in Avro/Parquet files: Resolved