[SPARK-47493] Disable spark.sql.parquet.inferTimestampNTZ.enabled by default

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 20/Mar/24 21:38
• Updated: 21/Mar/24 18:01

Description

Since Spark 4.0, the SQL config `spark.sql.parquet.inferTimestampNTZ.enabled` is turned off by default. Consequently, when reading Parquet files that were not produced by Spark, the Parquet reader will no longer automatically recognize data as the TIMESTAMP_NTZ data type. This change ensures backward compatibility with releases of Spark version 3.2 and earlier. It also aligns the behavior of schema inference for Parquet files with that of other data sources such as CSV, JSON, ORC, and JDBC, enhancing consistency across the data sources. To revert to the previous behavior where TIMESTAMP_NTZ types were inferred, set `spark.sql.parquet.inferTimestampNTZ.enabled` to true.

Issue Links

• links to: GitHub Pull Request #45621