Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31868

Wrong results for week-based-year

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Won't Fix
    • 3.0.0, 3.1.0
    • None
    • SQL

    Description

      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd')
               > ;
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#37]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
      spark.sql.legacy.timeParserPolicy	legacy
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#53]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#69]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      1970-01-01 00:00:00
      spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
      spark.sql.legacy.timeParserPolicy	legacy
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#87]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      1968-12-29 00:00:00
      spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
      1969-01-01 00:00:00
      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
      1969-01-01 00:00:00
      
      
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(2000-01-01, YYYY-MM-dd)#73]
      +- *(1) Scan OneRowRelation[]
      
      spark-sql> select to_timestamp('2000-01-01', 'YYYY-MM-dd');
      1970-01-01 00:00:00
      

      Seems that `-28800000000` is static

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Qin Yao Kent Yao 2
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment