Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3125

Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

    XMLWordPrintableJSON

Details

    • 1

    Description

      create table h0_p(id int, name string, price double, dt timestamp) using hudi partitioned by(dt) options(type = 'cow', primaryKey = 'id');
      
      insert into h0_p values (3, 'a1', 10, cast('2021-05-08 00:00:00' as timestamp)); 

      By default, that run the sql above will throw exception:

      Caused by: java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp
          at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8(AvroConversionHelper.scala:306)
          at org.apache.hudi.AvroConversionHelper$.$anonfun$createConverterToAvro$8$adapted(AvroConversionHelper.scala:306)
          at scala.Option.map(Option.scala:230) 

      We need disable `spark.sql.datetime.java8API.enabled` manually to make it work:

      set spark.sql.datetime.java8API.enabled=false; 

      And the command must be executed in the runtime. It can't work if provide this by spark-sql command: `spark-sql --conf spark.sql.datetime.java8API.enabled=false`. That's because this config is forced to enable when launch spark-sql.

      Attachments

        Issue Links

          Activity

            People

              biyan900116@gmail.com Yann Byron
              biyan900116@gmail.com Yann Byron
              Raymond Xu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: