Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31910

Enable Java 8 time API in Thrift server

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Switch to Java 8 time API by turning on the SQL config spark.sql.datetime.java8API.enabled to address the following issues:

      1. Date and timestamp string literals are parsed by using Java 8 time API and Spark's session time zone. Before the changes, date/timestamp values were collected as legacy types `java.sql.Date`/`java.sql.Timestamp`, and the value of such types didn't respect the config `spark.sql.session.timeZone`. To have consistent view, users had to keep JVM time zone and Spark's session time zone in sync.
      2. After the changes, formatting of date values doesn't depend on JVM time zone.
      3. While returning dates/timestamps of Java 8 type, we can avoid dates/timestamps rebasing from Proleptic Gregorian calendar to the hybrid calendar (Julian + Gregorian), and the issues related to calendar switching.
      4. Properly handle negative years (BCE).
      5. Consistent conversion of date/timestamp strings to/from internal Catalyst types in both direction to and from Spark.

        Attachments

          Activity

            People

            • Assignee:
              maxgekk Max Gekk
              Reporter:
              maxgekk Max Gekk
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: