[SPARK-47120] Null comparison push down data filter from subquery produces in NPE in Parquet filter

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Cosmin Dumitru
• Reporter: Cosmin Dumitru
• Created: 21/Feb/24 19:09
• Updated: 28/Feb/24 08:28
• Resolved: 26/Feb/24 00:54

Description

This issue has been introduced in https://github.com/apache/spark/pull/41088  where we convert scalar subqueries to literals and then convert the literals to org.apache.spark.sql.sources.Filters. These filters are then pushed down to parquet.

If the literal is a comparison with null then the parquet filter conversion code throws NPE. 

 

repro code which results in NPE

create table t1(d date) using parquet
create table t2(d date) using parquet
insert into t1 values date'2021-01-01'
insert into t2 values (null)
select * from t1 where 1=1 and d > (select d from t2)

fix PR 

Issue Links

• links to: GitHub Pull Request #45202