[HIVE-15269] Dynamic Min-Max/BloomFilter runtime-filtering for Tez

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Tez
• Labels: TODOC2.2.0
• Assignee: Deepak Jaiswal
• Reporter: Jason Dere
• Created: 23/Nov/16 07:08
• Updated: 08/May/20 14:44
• Resolved: 24/Jan/17 20:03

Description

If a dimension table and fact table are joined:

select *
from store join store_sales on (store.id = store_sales.store_id)
where store.s_store_name = 'My Store'

One optimization that can be done is to get the min/max store id values that come out of the scan/filter of the store table, and send this min/max value (via Tez edge) to the task which is scanning the store_sales table.
We can add a BETWEEN(min, max) predicate to the store_sales TableScan, where this predicate can be pushed down to the storage handler (for example for ORC formats). Pushing a min/max predicate to the ORC reader would allow us to avoid having to entire whole row groups during the table scan.

Issue Links

• is blocked by: HIVE-15270 ExprNode/Sarg changes to support values supplied during query runtime
• is depended upon by: HIVE-10924 add support for MERGE statement
• is related to: HIVE-16001 add test for merge + runtime filtering
• is related to: HIVE-15698 Vectorization support for min/max/bloomfilter runtime filtering
• is required by: HIVE-16862 Implement a similar feature like "hive.tez.dynamic.semijoin.reduction" in hive on spark
• relates to: HIVE-15802 Changes to expected entries for dynamic bloomfilter runtime filtering
• links to: https://reviews.apache.org/r/55156/