[SPARK-32811] Replace IN predicate of continuous range with boundary checks

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Vu Ho
• Created: 07/Sep/20 09:36
• Updated: 08/Sep/20 10:29

Description

This expression 

select a from t where a in (1, 2, 3, 3, 4)

can be translated to 

select a from t where a >= 1 and a <= 4 

This would speed up parquet row group filter (currently or(or(or(or(or(eq(x, 1), eq(x, 2)), eq(x, 3), eq(x, 4.....)))) and make query more compact

 

Issue Links

• links to: [Github] Pull Request #29661 (hotienvu)
• links to: [Github] Pull Request #29661 (hotienvu)