Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31981

Keep TimestampType when taking an average of a Timestamp

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • SQL
    • None

    Description

      Currently, when you take an average of a Timestamp, you'll end up with a Double, representing the seconds since epoch. This is because of old Hive behavior. I strongly believe that it is better to return a Timestamp.

      root@8c4241b617ec:/# psql postgres postgres
      psql (12.3 (Debian 12.3-1.pgdg100+1))
      Type "help" for help.

      postgres=# CREATE TABLE timestamp_demo (ts TIMESTAMP);
      CREATE TABLE
      postgres=# INSERT INTO timestamp_demo VALUES('2019-01-01 18:22:11');
      INSERT 0 1
      postgres=# INSERT INTO timestamp_demo VALUES('2018-01-01 18:22:11');
      INSERT 0 1
      postgres=# INSERT INTO timestamp_demo VALUES('2017-01-01 18:22:11');
      INSERT 0 1
      postgres=# SELECT AVG(ts) FROM timestamp_demo;
      ERROR: function avg(timestamp without time zone) does not exist
      LINE 1: SELECT AVG(ts) FROM timestamp_demo;

       

      root@bab43a5731e8:/# mysql
      Welcome to the MySQL monitor. Commands end with ; or \g.
      Your MySQL connection id is 9
      Server version: 8.0.20 MySQL Community Server - GPL

      Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.

      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

      mysql> CREATE TABLE timestamp_demo (ts TIMESTAMP);
      Query OK, 0 rows affected (0.05 sec)

      mysql> INSERT INTO timestamp_demo VALUES('2019-01-01 18:22:11');
      Query OK, 1 row affected (0.01 sec)

      mysql> INSERT INTO timestamp_demo VALUES('2018-01-01 18:22:11');
      Query OK, 1 row affected (0.01 sec)

      mysql> INSERT INTO timestamp_demo VALUES('2017-01-01 18:22:11');
      Query OK, 1 row affected (0.01 sec)

      mysql> SELECT AVG(ts) FROM timestamp_demo;
      ---------------------

      AVG(ts)

      ---------------------

      20180101182211.0000

      ---------------------
      1 row in set (0.00 sec)

       

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            fokko Fokko Driesprong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: