Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-46707

Introduce throwable field to mark expressions that may throw a runtime exception

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0
    • 4.0.0
    • SQL

    Description

      For predicate pushdown, currently it is possible that we push down a filter that ends up being evaluated on more rows than before it was pushed down (e.g. if we push the filter through a selective join). In this case, it is possible that we now evaluate the filter on a row that will cause a runtime error to be thrown, when prior to pushing this would not have happened.

      If we mark certain expressions as throwable, we can avoid pushing these through joins, filters, and aggregations to avoid this.

      Attachments

        Activity

          People

            kelvinjiang Kelvin Jiang
            kelvinjiang Kelvin Jiang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: