Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31408

Build Spark’s own datetime pattern definition

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      This is an umbrella ticket for building Spark's own Datetime patterns and related works.

      In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.

      However, there are some datetime patterns not compatible between Java 8 and Java 7 APIs, and it's fragile to rely on the JDK API to define Spark's behavior. We should build our own Datetime patterns, which is compatible with Spark 2.4 (the old Java 7 `SimpleDateFormat` API).

      Attachments

        1.
        		 Disable week-based date filed for parsing Sub-task Resolved Kent Yao 2
        2.
        		 Backward Compatibility for Parsing and Formatting Datetime Sub-task Resolved Yuanjian Li
        3.
        		 Parsing seconds fraction with variable length for timestamp Sub-task Resolved Kent Yao 2
        4.
        		 Remove the unnecessary config spark.sql.legacy.timeParser.enabled Sub-task Resolved Kent Yao 2
        5.
        		 Fix errors and missing parts for datetime pattern document Sub-task Resolved Kent Yao 2
        6.
        		 Remove support for 'e'/'c' as datetime pattern charactar Sub-task Resolved Kent Yao 2
        7.
        		 Raise exception instead of silent change for new DateFormatter Sub-task Resolved Yuanjian Li
        8.
        		 Performance regression with new TimestampFormatter for json and csv Sub-task Resolved Kent Yao 2
        9.
        		 allow missing year/hour when parsing date/timestamp Sub-task Resolved Wenchen Fan
        10.
        		 Fix perf regression of date/timestamp formatting in toHiveString Sub-task Resolved Max Gekk
        11.
        		 Format Java date-time types in Row.jsonValue directly Sub-task Resolved Max Gekk
        12.
        		 Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q' Sub-task Resolved Kent Yao 2
        13.
        		 Fail datetime parsing/formatting if detect the Java 8 bug of stand-alone form Sub-task Resolved Wenchen Fan
        14.
        		 Consistent error handling for datetime formatting functions Sub-task Resolved Kent Yao 2
        15.
        		 Fix silent data change for datetime formatting Sub-task Resolved Kent Yao 2
        16.
        		 First day of week changed for non-MONDAY_START Lacales Sub-task Resolved Kent Yao 2
        17.
        		 Handle am-pm timestamp parsing when hour is missing Sub-task Resolved Kent Yao 2
        18.
        		 Forbid datetime pattern letter u Sub-task Resolved Wenchen Fan
        19.
        		 Fix Parsing day of year when year field pattern is missing Sub-task Resolved Kent Yao 2
        20.
        		 Adjacent value parsing not supported for Localized Patterns because of JDK bug Sub-task Resolved Unassigned
        21.
        		 Fix silent data change for timestamp parsing if overflow happens Sub-task Resolved Kent Yao 2

        Activity

          People

            Unassigned Unassigned
            XuanYuan Yuanjian Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: