[IMPALA-10098] Runtime Filters for Set Exclusion or Compliment

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend, Frontend
• Labels: performance runtime-filters
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Shant Hovsepian
• Created: 22/Aug/20 03:24
• Updated: 22/Dec/20 19:51

Description

It would be beneficial to extend runtime filters to push set exclusion down to scan nodes. This would be used to optimize NOT IN, EXCEPT style queries or more generally ANTI JOINS, as well as OUTER JOINs which filter out non null attributes from the nullable side.

This is almost the inverse operation of a traditional bloom filter, other data structures might be more efficient.

This would also compliment Impala's left deep pipelined query planning very well for what otherwise would require complex query plans due to reordering restrictions with ANTI/OUTER joins.