[SPARK-25402] Null handling in BooleanSimplification

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.2.2, 2.3.1
• Fix Version/s: 2.2.3, 2.3.2, 2.4.0
• Component/s: SQL
• Target Version/s: 2.2.3, 2.3.2
• Assignee: Xiao Li
• Reporter: Xiao Li
• Created: 11/Sep/18 06:57
• Updated: 13/Sep/18 13:41
• Resolved: 13/Sep/18 02:01

Description

SPARK-20350 introduced a bug BooleanSimplification for null handling. For example, the following case returns a wrong answer.

    val schema = StructType.fromDDL("a boolean, b int")
    val rows = Seq(Row(null, 1))

    val rdd = sparkContext.parallelize(rows)
    val df = spark.createDataFrame(rdd, schema)

    checkAnswer(df.where("(NOT a) OR a"), Seq.empty)

Issue Links

• relates to: SPARK-20350 Apply Complementation Laws during boolean expression simplification
• links to: [Github] Pull Request #22390 (gatorsmile)
• links to: [Github] Pull Request #22403 (gatorsmile)
• links to: [Github] Pull Request #22403 (gatorsmile)