XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 3.1.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.

      A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674

      The existing behavior itself in 2.4 is confusing, e.g.

      spark-sql> select to_timestamp('1', 'w');
      1969-12-28 00:00:00
      spark-sql> select to_timestamp('1', 'u');
      1970-01-05 00:00:00
      

      the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.

      spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy');
      2020-01-01 00:00:00
      spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY');
      2019-12-29 00:00:00
      spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w');
      NULL
      spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w');
      2019-12-29 00:00:00
      

      I think we don't need to introduce all the weird behavior from Java.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Qin Yao Kent Yao
                Reporter:
                Qin Yao Kent Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: