[PARQUET-2244] Dictionary filter may skip row-groups incorrectly when evaluating notIn

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.2
• Fix Version/s: 1.13.0
• Component/s: parquet-mr
• Assignee: Yujiang Zhong
• Reporter: Yujiang Zhong
• Created: 15/Feb/23 10:00
• Updated: 12/Apr/23 01:32
• Resolved: 15/Feb/23 10:59

Description

Dictionary filter may skip row-groups incorrectly when evaluating `notIn` on optional columns with null values. Here is an example:

Say there is a optional column `c1` with all pages dict encoded, `c1` has and only has two distinct values: ['foo', null],  and the predicate is  `c1 not in ('foo', 'bar')`. 

Now dictionary filter may skip this row-group that is actually should not be skipped, because there are nulls in the column.

 

This is a bug similar to #1510.