[SPARK-39017] Change Java8 datetime support to configurable

• Type: Brainstorming
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: SQL
• Labels: code

Language

scala

• Assignee: Unassigned
• Reporter: Weicheng Wang
• Created: 26/Apr/22 03:25
• Updated: 26/Apr/22 03:25

Description

Background:

In the Spark 3.1.0 there is an improvement introduced to enable Java 8 datetime API by default.

It stops user set this configuration, spark.sql.datetime.java8API.enabled, in command line or configuration file when using Spark SQL shell or Spark Thrift Server.

The only way to set it is in the SQL Session using SET command like:

spark-sql> SET spark.sql.datetime.java8API.enabled=false 

There are a few issues related to this improvement 

• https://github.com/apache/iceberg/issues/2530
• https://github.com/delta-io/delta/issues/760

There is a solution in 3.2.0 for it by having LocalDateConverter uses DateConverter and in the DateConverter it handles both LocalDate and Date types.

 

Discussion:

I think we should give back the ability to user to set this option in configuration file and command line. Both of the improvement defeats the reason why we have a java8API option configurable.

 

Please advise

Thanks