Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37935 Migrate onto error classes
  3. SPARK-38481

Substitute Java overflow exception from TIMESTAMPADD by Spark exception

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Currently, Spark SQL can throw Java exceptions from the timestampadd()/date_add()/dateadd() functions, for instance:

      spark-sql> select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03');
      22/03/09 14:47:15 ERROR SparkSQLDriver: Failed in [select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03')]
      java.lang.ArithmeticException: long overflow
      	at java.lang.Math.multiplyExact(Math.java:892) ~[?:1.8.0_292]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:505) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAddMonths(DateTimeUtils.scala:724) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAdd(DateTimeUtils.scala:1197) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
      

      That might confuse non-Scala/Java users. Need to wrap such kind of exception by Spark's exception using an error class.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment