[SPARK-29164] Rewrite coalesce(boolean, booleanLit) as boolean expression

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 19/Sep/19 00:19
• Updated: 12/Dec/22 18:10
• Resolved: 17/Dec/19 02:04

Description

I propose the following expression rewrite optimizations:

coalesce(x: Boolean, true)  -> x or isnull(x)
coalesce(x: Boolean, false) -> x and isnotnull(x)

This pattern appears when translating Dataset filters on Option[Boolean] columns: we might have a typed Dataset filter which looks like

 .filter(_.boolCol.getOrElse(DEFAULT_VALUE))

and the most idiomatic, user-friendly translation of this in Catalyst is to use coalesce(). However, the coalesce() form of this expression is not eligible for Parquet / data source filter pushdown.

(We should write out truth-tables to double-check this rewrite's correctness)

Issue Links

• links to: GitHub Pull Request #25887