Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31910

Enable Java 8 time API in Thrift server

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • SQL
    • None

    Description

      Switch to Java 8 time API by turning on the SQL config spark.sql.datetime.java8API.enabled to address the following issues:

      1. Date and timestamp string literals are parsed by using Java 8 time API and Spark's session time zone. Before the changes, date/timestamp values were collected as legacy types `java.sql.Date`/`java.sql.Timestamp`, and the value of such types didn't respect the config `spark.sql.session.timeZone`. To have consistent view, users had to keep JVM time zone and Spark's session time zone in sync.
      2. After the changes, formatting of date values doesn't depend on JVM time zone.
      3. While returning dates/timestamps of Java 8 type, we can avoid dates/timestamps rebasing from Proleptic Gregorian calendar to the hybrid calendar (Julian + Gregorian), and the issues related to calendar switching.
      4. Properly handle negative years (BCE).
      5. Consistent conversion of date/timestamp strings to/from internal Catalyst types in both direction to and from Spark.

      Attachments

        Activity

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: