[HUDI-3125] Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.1
• Component/s: spark
• Labels: pull-request-available sev:critical user-support-issues
• Story Points: 1
• Assignee: Yann Byron
• Reporter: Yann Byron

Reviewers

Shiyan Xu

• Created: 29/Dec/21 15:40
• Updated: 09/Jan/22 07:44
• Resolved: 09/Jan/22 07:44

Description

create table h0_p(id int, name string, price double, dt timestamp) using hudi partitioned by(dt) options(type = 'cow', primaryKey = 'id');

insert into h0_p values (3, 'a1', 10, cast('2021-05-08 00:00:00' as timestamp)); 

By default, that run the sql above will throw exception:

Caused by: java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp
    at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8(AvroConversionHelper.scala:306)
    at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8$adapted(AvroConversionHelper.scala:306)
    at scala.Option.map(Option.scala:230) 

We need disable `spark.sql.datetime.java8API.enabled` manually to make it work:

set spark.sql.datetime.java8API.enabled=false; 

And the command must be executed in the runtime. It can't work if provide this by spark-sql command: `spark-sql --conf spark.sql.datetime.java8API.enabled=false`. That's because this config is forced to enable when launch spark-sql.

Issue Links

• links to: GitHub Pull Request #4471