[IMPALA-3051] Runtime filter with RUNTIME_FILTER_MODE=1 doesn't create any filters

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: Impala 2.5.0
• Component/s: Backend
• Labels: performance runtime-filters
• Assignee: Henry Robinson
• Reporter: Mostafa Mokhtar
• Created: 21/Feb/16 22:52
• Updated: 22/Feb/16 06:09
• Resolved: 22/Feb/16 06:09

Description

With

RUNTIME_FILTER_MODE: [1]

no filters are generated at all, Impala is expected to create local filters to maintain previous release behavior

select 
    count(*) as c
from
    store_sales,
    household_demographics,
    time_dim,
    store
where
    store_sales.ss_sold_time_sk = time_dim.t_time_sk
        and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk
        and store_sales.ss_store_sk = store.s_store_sk
        and time_dim.t_hour = 8
        and time_dim.t_minute >= 30
        and household_demographics.hd_dep_count = 5
        and store.s_store_name = 'ese'
order by c
limit 100;

Operator	#Hosts	Avg Time	Max Time	#Rows	Est. #Rows	Peak Mem	Est. Peak Mem	Detail
08:TOP-N	1	208.62us	208.62us	1	1	12.00 KB	-1 B
13:AGGREGATE	1	194.31ms	194.31ms	1	1	92.00 KB	-1 B	FINALIZE
12:EXCHANGE	1	303.11us	303.11us	20	1	0 B	-1 B	UNPARTITIONED
07:AGGREGATE	20	1.53s	1.92s	20	1	18.59 MB	10.00 MB
06:HASH JOIN	20	195.81ms	257.03ms	9.43M	18.53M	12.66 MB	186 B	INNER JOIN, BROADCAST
!--11:EXCHANGE	20	10.01us	19.21us	160	6	0 B	0 B	BROADCAST
! 03:SCAN HDFS	1	7.35ms	7.35ms	8	6	70.00 KB	32.00 MB	tpcds_15000_decimal_parquet.store
05:HASH JOIN	20	1.37s	1.66s	80.90M	100.45M	6.16 MB	9.28 KB	INNER JOIN, BROADCAST
!--10:EXCHANGE	20	23.37us	40.83us	14.40K	720	0 B	0 B	BROADCAST
! 01:SCAN HDFS	1	17.28ms	17.28ms	720	720	273.20 KB	32.00 MB	tpcds_15000_decimal_parquet.household_demographics
04:HASH JOIN	20	55.06s	64.66s	818.31M	1.00B	2.21 MB	19.56 KB	INNER JOIN, BROADCAST
!--09:EXCHANGE	20	41.44us	79.43us	36.00K	1.14K	0 B	0 B	BROADCAST
! 02:SCAN HDFS	1	21.82ms	21.82ms	1.80K	1.14K	1.89 MB	48.00 MB	tpcds_15000_decimal_parquet.time_dim
00:SCAN HDFS	20	9.06s	11.34s	43.20B	43.20B	418.11 MB	264.00 MB	tpcds_15000_decimal_parquet.store_sales

Plan

08:TOP-N [LIMIT=100]
|  order by: count(*) ASC
|  hosts=20 per-host-mem=unavailable
|  tuple-ids=5 row-size=8B cardinality=1
|
13:AGGREGATE [FINALIZE]
|  output: count:merge(*)
|  hosts=20 per-host-mem=unavailable
|  tuple-ids=4 row-size=8B cardinality=1
|
12:EXCHANGE [UNPARTITIONED]
|  hosts=20 per-host-mem=unavailable
|  tuple-ids=4 row-size=8B cardinality=1
|
07:AGGREGATE
|  output: count(*)
|  hosts=20 per-host-mem=10.00MB
|  tuple-ids=4 row-size=8B cardinality=1
|
06:HASH JOIN [INNER JOIN, BROADCAST]
|  hash predicates: store_sales.ss_store_sk = store.s_store_sk
|  runtime filters: RF000 <- store.s_store_sk
|  hosts=20 per-host-mem=186B
|  tuple-ids=0,2,1,3 row-size=80B cardinality=18530203
|
|--11:EXCHANGE [BROADCAST]
|  |  hosts=1 per-host-mem=0B
|  |  tuple-ids=3 row-size=28B cardinality=6
|  |
|  03:SCAN HDFS [tpcds_15000_decimal_parquet.store, RANDOM]
|     partitions=1/1 files=1 size=11.92KB
|     predicates: store.s_store_name = 'ese'
|     table stats: 62 rows total
|     column stats: all
|     hosts=1 per-host-mem=32.00MB
|     tuple-ids=3 row-size=28B cardinality=6
|
05:HASH JOIN [INNER JOIN, BROADCAST]
|  hash predicates: store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk
|  runtime filters: RF001 <- household_demographics.hd_demo_sk
|  hosts=20 per-host-mem=9.28KB
|  tuple-ids=0,2,1 row-size=52B cardinality=100447876
|
|--10:EXCHANGE [BROADCAST]
|  |  hosts=1 per-host-mem=0B
|  |  tuple-ids=1 row-size=12B cardinality=720
|  |
|  01:SCAN HDFS [tpcds_15000_decimal_parquet.household_demographics, RANDOM]
|     partitions=1/1 files=1 size=41.12KB
|     predicates: household_demographics.hd_dep_count = 5
|     table stats: 7200 rows total
|     column stats: all
|     hosts=1 per-host-mem=32.00MB
|     tuple-ids=1 row-size=12B cardinality=720
|
04:HASH JOIN [INNER JOIN, BROADCAST]
|  hash predicates: store_sales.ss_sold_time_sk = time_dim.t_time_sk
|  runtime filters: RF002 <- time_dim.t_time_sk
|  hosts=20 per-host-mem=19.56KB
|  tuple-ids=0,2 row-size=40B cardinality=1004478759
|
|--09:EXCHANGE [BROADCAST]
|  |  hosts=1 per-host-mem=0B
|  |  tuple-ids=2 row-size=16B cardinality=1138
|  |
|  02:SCAN HDFS [tpcds_15000_decimal_parquet.time_dim, RANDOM]
|     partitions=1/1 files=1 size=1.30MB
|     predicates: time_dim.t_hour = 8, time_dim.t_minute >= 30
|     table stats: 86400 rows total
|     column stats: all
|     hosts=1 per-host-mem=48.00MB
|     tuple-ids=2 row-size=16B cardinality=1138
|
00:SCAN HDFS [tpcds_15000_decimal_parquet.store_sales, RANDOM]
   partitions=1824/1824 files=6922 size=1.52TB
   runtime filters: RF000 -> store_sales.ss_store_sk, RF001 -> store_sales.ss_hdemo_sk, RF002 -> store_sales.ss_sold_time_sk
   table stats: 43202446268 rows total
   column stats: all
   hosts=20 per-host-mem=264.00MB
   tuple-ids=0 row-size=24B cardinality=43202446268