Description
In Spark 3.0, we switch to the Proleptic Gregorian calendar by using the Java 8 datetime APIs. This makes Spark follow the ISO and SQL standard, but introduces some backward compatibility problems:
1. may read wrong data from the data files written by Spark 2.4
2. may have perf regression
Attachments
Attachments
Issue Links
- relates to
-
SPARK-30951 Potential data loss for legacy applications after switch to proleptic Gregorian calendar
-
- Resolved
-
-
SPARK-26651 Use Proleptic Gregorian calendar
-
- Resolved
-
- supercedes
-
SPARK-34675 TimeZone inconsistencies when JVM and session timezones are different
-
- Reopened
-