Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37935 Migrate onto error classes
  3. SPARK-38481

Substitute Java overflow exception from TIMESTAMPADD by Spark exception

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Currently, Spark SQL can throw Java exceptions from the timestampadd()/date_add()/dateadd() functions, for instance:

      spark-sql> select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03');
      22/03/09 14:47:15 ERROR SparkSQLDriver: Failed in [select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03')]
      java.lang.ArithmeticException: long overflow
      	at java.lang.Math.multiplyExact(Math.java:892) ~[?:1.8.0_292]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:505) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAddMonths(DateTimeUtils.scala:724) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAdd(DateTimeUtils.scala:1197) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      

      That might confuse non-Scala/Java users. Need to wrap such kind of exception by Spark's exception using an error class.

      Attachments

        Issue Links

          Activity

            People

              maxgekk Max Gekk
              maxgekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: