Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17981

Incorrectly Set Nullability to False in FilterExec

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.0.1, 2.1.0
    • 2.0.3, 2.1.0
    • SQL

    Description

      When FilterExec contains isNotNull, which could be inferred and pushed down or users specified, we convert the nullability of the involved columns if the top-layer expression is null-intolerant. However, this is not true, if the top-layer expression is not a leaf expression, it could still tolerate the null when it has null-tolerant child expression.

      For example, cast(coalesce(a#5, a#15) as double). Although cast is a null-intolerant expression, but obviously coalesce is a null-tolerant.

      When the nullability is wrong, we could generate incorrect results in different cases.

      Attachments

        Activity

          People

            smilegator Xiao Li
            smilegator Xiao Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: