Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33275

ANSI mode: runtime errors instead of returning null on invalid inputs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Done
    • 3.1.0
    • None
    • SQL
    • None

    Description

      We should respect the ANSI mode in more places. What we have done so far are mostly the overflow check in various operators. This ticket is to track a category of ANSI mode behaviors: operators should throw runtime errors instead of returning null when the input is invalid.

      Attachments

        Issue Links

          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-35030 ANSI SQL compliance

          • Major - Major loss of function.
          • Resolved
          1.
          		 Division by zero on divide-like operations returns incorrect result Sub-task Resolved Lu Lu
          2.
          		 Accessing array elements should failed if index is out of bound. Sub-task Resolved Leanken.Lin
          3.
          		 Accessing map values should fail if key is not found. Sub-task Resolved Leanken.Lin
          4.
          		 ParseUrl should fail if input string is not a valid url Sub-task Resolved XiDuo You
          5.
          		 Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid Sub-task Resolved Leanken.Lin
          6.
          		 Datetime building should fail if the year, month, ..., second combination is invalid Sub-task Resolved zhoukeyong
          7.
          		 Fix the constant folding rule to skip it if the expression fails to execute Sub-task In Progress Unassigned
          8.
          		 ANSI mode: Casting string to date should throw exception on parse error Sub-task Resolved Gengliang Wang
          9.
          		 next_day function should throw runtime exception when receiving invalid input under ANSI mode Sub-task Resolved Chongguang LIU
          10.
          		 ANSI mode: Casting String to Boolean should throw exception on parse error Sub-task Resolved Gengliang Wang
          11.
          		 ANSI mode: Abs throws exception if input is out of range Sub-task Resolved Gengliang Wang
          12.
          		 ANSI mode: IntegralDivide throws exception on overflow Sub-task Resolved Gengliang Wang
          13.
          		 The ANSI flag of Cast should be kept after being copied Sub-task Resolved Gengliang Wang
          14.
          		 The ANSI flags of Sum and Avg should be kept after being copied Sub-task Resolved Gengliang Wang
          15.
          		 ANSI mode: always return null on invalid access to map column Sub-task Resolved Gengliang Wang
          16.
          		 ANSI mode: allow explicitly casting fraction strings as Integral types Sub-task Resolved Gengliang Wang
          17.
          		 Round/Bround should return an error on integral overflow Sub-task Resolved Gengliang Wang
          18.
          		 Conv should return an error if the internal conversion overflows Sub-task Resolved Gengliang Wang

          Activity

            People

              apachespark Apache Spark
              cloud_fan Wenchen Fan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: