Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26740

Statistics for date and timestamp columns depend on system time zone

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      While saving statistics for timestamp/date columns, default time zone is used in conversion of internal type (microseconds or days since epoch) to textual representation. The textual representation doesn't contain time zone. So, when it is converted back to internal types (Long for TimestampType or DateType), the Timestamp.valueOf and Date.valueOf are used in conversions. The methods use current system time zone.
      If system time zone is different while saving and retrieving statistics for timestamp/date columns, restored microseconds/days since epoch will be different.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Maxim Gekk
                Reporter:
                maxgekk Maxim Gekk
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: