Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17897

not isnotnull is converted to the always false condition isnotnull && not isnotnull

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.0.1
    • Fix Version/s: 2.0.3, 2.1.0
    • Component/s: Optimizer, SQL
    • Labels:
    • Target Version/s:

      Description

      When a logical plan is built containing the following somewhat nonsensical filter:
      Filter (NOT isnotnull($f0#212))

      During optimization the filter is converted into a condition that will always fail:
      Filter (isnotnull($f0#212) && NOT isnotnull($f0#212))

      This appears to be caused by the following check for NullIntolerant:

      https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R63

      Which recurses through the expression and extracts nested IsNotNull calls, converting them to IsNotNull calls on the attribute at the root level:

      https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R49

      This results in the nonsensical condition above.

        Attachments

          Activity

            People

            • Assignee:
              smilegator Xiao Li
              Reporter:
              kuujo Jordan Halterman
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: