Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31408

Build Spark’s own datetime pattern definition

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This is an umbrella ticket for building Spark's own Datetime patterns and related works.

      In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.

      However, there are some datetime patterns not compatible between Java 8 and Java 7 APIs, and it's fragile to rely on the JDK API to define Spark's behavior. We should build our own Datetime patterns, which is compatible with Spark 2.4 (the old Java 7 `SimpleDateFormat` API).

        Attachments

        1.
        Disable week-based date filed for parsing Sub-task Resolved Kent Yao
        2.
        Backward Compatibility for Parsing and Formatting Datetime Sub-task Resolved Yuanjian Li
        3.
        Parsing seconds fraction with variable length for timestamp Sub-task Resolved Kent Yao
        4.
        Remove the unnecessary config spark.sql.legacy.timeParser.enabled Sub-task Resolved Kent Yao
        5.
        Fix errors and missing parts for datetime pattern document Sub-task Resolved Kent Yao
        6.
        Remove support for 'e'/'c' as datetime pattern charactar Sub-task Resolved Kent Yao
        7.
        Raise exception instead of silent change for new DateFormatter Sub-task Resolved Yuanjian Li
        8.
        Performance regression with new TimestampFormatter for json and csv Sub-task Resolved Kent Yao
        9.
        allow missing year/hour when parsing date/timestamp Sub-task Resolved Wenchen Fan
        10.
        Fix perf regression of date/timestamp formatting in toHiveString Sub-task Resolved Maxim Gekk
        11.
        Format Java date-time types in Row.jsonValue directly Sub-task Resolved Maxim Gekk
        12.
        Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q' Sub-task Resolved Kent Yao
        13.
        Fail datetime parsing/formatting if detect the Java 8 bug of stand-alone form Sub-task Resolved Wenchen Fan
        14.
        Consistent error handling for datetime formatting functions Sub-task Resolved Apache Spark
        15.
        Fix silent data change for datetime formatting Sub-task Resolved Kent Yao
        16.
        First day of week changed for non-MONDAY_START Lacales Sub-task Resolved Kent Yao
        17.
        Handle am-pm timestamp parsing when hour is missing Sub-task Resolved Kent Yao
        18.
        Forbid datetime pattern letter u Sub-task Resolved Wenchen Fan
        19.
        Fix Parsing day of year when year field pattern is missing Sub-task Resolved Kent Yao
        20.
        Adjacent value parsing not supported for Localized Patterns because of JDK bug Sub-task Resolved Unassigned
        21.
        Fix silent data change for timestamp parsing if overflow happens Sub-task Resolved Kent Yao

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              XuanYuan Yuanjian Li
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: