Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11724

Casting integer types to timestamp has unexpected semantics

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Minor
        • Resolution: Fixed
        • Affects Version/s: 1.6.0
        • Fix Version/s: 1.6.0
        • Component/s: SQL
        • Labels:
        • Target Version/s:

          Description

          Casting from integer types to timestamp treats the source int as being in millis. Casting from timestamp to integer types creates the result in seconds. This leads to behavior like:

          scala> sql("select cast(cast (1234 as timestamp) as bigint)").show
+---+
|_c0|
+---+
|  1|
+---+

          Double's on the other hand treat it as seconds when casting to and from:

          scala> sql("select cast(cast (1234.5 as timestamp) as double)").show
+------+
|   _c0|
+------+
|1234.5|
+------+

          This also breaks some other functions which return long in seconds, in particular, unix_timestamp.

          scala> sql("select cast(unix_timestamp() as timestamp)").show
+--------------------+
|                 _c0|
+--------------------+
|1970-01-17 10:03:...|
+--------------------+

scala> sql("select cast(unix_timestamp() *1000 as timestamp)").show
+--------------------+
|                 _c0|
+--------------------+
|2015-11-12 23:26:...|
+--------------------+

            Issue Links

            is duplicated by

            Bug - A problem which impairs or prevents the functions of the product. SPARK-13341 Casting Unix timestamp to SQL timestamp fails

            • Major - Major loss of function.
            • Resolved
            links to

            Pull request #9685 [Github] Pull Request #9685 (nongli)

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              apachespark Apache Spark added a comment -

              User 'nongli' has created a pull request for this issue:
              https://github.com/apache/spark/pull/9685

              Show
              apachespark Apache Spark added a comment - User 'nongli' has created a pull request for this issue: https://github.com/apache/spark/pull/9685
              Hide
              Permalink
              yhuai Yin Huai added a comment -

              Issue resolved by pull request 9685
              https://github.com/apache/spark/pull/9685

              Show
              yhuai Yin Huai added a comment - Issue resolved by pull request 9685 https://github.com/apache/spark/pull/9685

                People

                • Assignee:
                  nongli Nong Li
                  Reporter:
                  nongli Nong Li
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development