[SPARK-46440] Set the rebase configs to the CORRECTED mode by default

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 18/Dec/23 07:26
• Updated: 21/Dec/23 07:52
• Resolved: 18/Dec/23 08:41

Description

Set all rebase related SQL configs to the `CORRECTED` mode by default. Here are the affected configs:

• spark.sql.parquet.int96RebaseModeInWrite
• spark.sql.parquet.datetimeRebaseModeInWrite
• spark.sql.parquet.int96RebaseModeInRead
• spark.sql.parquet.datetimeRebaseModeInRead
• spark.sql.avro.datetimeRebaseModeInWrite
• spark.sql.avro.datetimeRebaseModeInRead

The configs were set to the `EXCEPTION` mode to give users a choice to select proper mode for compatibility with old Spark versions <= 2.4.5. Those versions are not able to detect the rebase mode from meta information in parquet and avro files. Since the versions are out of broad usage, Spark starting from the version 4.0.0 will write/ read ancient datatime without rebasing and any exceptions. This should be more convenient for users.

Issue Links

• links to: GitHub Pull Request #44388