[SPARK-49396] PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Optimizer
• Labels: pull-request-available
• Assignee: Avery Qi
• Reporter: Avery Qi
• Created: 26/Aug/24 18:21
• Updated: 06/Sep/24 02:31
• Resolved: 06/Sep/24 02:22

Description

 SimplifyConditionals has a simplification code where if any of the (not first) branches  in CaseWhen is TrueLiteral we remove all the remaining branches including the elseValue.

case CaseWhen(branches, ) if branches.exists(._1 == TrueLiteral) => // a branch with a true condition eliminates all following branches,
// these branches can be pruned away
val (h, t) = branches.span(_._1 != TrueLiteral)
CaseWhen( h :+ t.head, None)}}

Now, the nullability check of CaseWhen checks that
(1) either of the branches including elseValue is nullable or
(2) elseValue is None.

 

The above simplification makes the elseValue as None. Combined with this nullability check makes the CaseWhen switch from non-nullable to nullable.

This ticket aims to fix this issue and remain the nullability of the expression by replacing the elsevalue to the value of the TrueLiteral expression.

Issue Links

• links to: GitHub Pull Request #47877
• links to: GitHub Pull Request #47981