[SPARK-31318] Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 31/Mar/20 17:46
• Updated: 01/Apr/20 05:05
• Resolved: 01/Apr/20 05:05

Description

Currently, Spark provides 2 SQL configs to control rebasing of dates/timestamps in Parquet and Avro datasource: 

spark.sql.legacy.parquet.rebaseDateTime.enabled

spark.sql.legacy.avro.rebaseDateTime.enabled

The configs control rebasing in read and in write. That's can be inconvenient for users who want to read files saved by Spark 2.4 and earlier versions, and save dates/timestamps without rebasing.

The ticket aims to split the configs, and introduce separate SQL configs for read and for write. 

Issue Links

• relates to: SPARK-31159 Incompatible Parquet dates/timestamps with Spark 2.4
• relates to: SPARK-31183 Incompatible Avro dates/timestamps with Spark 2.4
• links to: GitHub Pull Request #28082