[IMPALA-10252] Query returns less number of rows with run-time filtering on integer column in a subquery against functional_parquet schema

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0, Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Frontend
• Labels: correctness
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-1
• Assignee: Tim Armstrong
• Reporter: Qifan Chen
• Created: 18/Oct/20 16:54
• Updated: 08/Dec/20 16:50
• Resolved: 08/Dec/20 16:49

Description

During the work to address IMPALA-6628 (Use unqualified table references in .test files run from test_queries.py), it is found that a query against the functional_parquet database returns 1 row while the same query returns 12 rows when run-time filtering is turned off, or against the functional database. 

 

 

Query: --SET RUNTIME_FILTER_MODE=OFF;
select id, int_col, year, month
 from functional_parquet.alltypessmall s
 where s.int_col = (select count(*) from functional_parquet.alltypestiny t where s.id = t.id)
 order by id
Query submitted at: 2020-10-18 12:41:15 (Coordinator: http://qifan-10229:25000)
Query progress can be monitored at: http://qifan-10229:25000/query_plan?query_id=394a61d8f0002336:fd45e07300000000
+----+---------+------+-------+
| id | int_col | year | month |
+----+---------+------+-------+
| 1 | 1 | 2009 | 1 |
+----+---------+------+-------+

 

 

RUNTIME_FILTER_MODE set to OFF
Query: select id, int_col, year, month 
 from functional_parquet.alltypessmall s 
 where s.int_col = (select count(*) from functional_parquet.alltypestiny t where s.id = t.id) 
 order by id
Query submitted at: 2020-10-18 12:40:58 (Coordinator: http://qifan-10229:25000)
Query progress can be monitored at: http://qifan-10229:25000/query_plan?query_id=304c095f478607fc:7d2d03ff00000000
+----+---------+------+-------+
| id | int_col | year | month |
+----+---------+------+-------+
| 1 | 1 | 2009 | 1 |
| 10 | 0 | 2009 | 1 |
| 20 | 0 | 2009 | 1 |
| 25 | 0 | 2009 | 2 |
| 35 | 0 | 2009 | 2 |
| 45 | 0 | 2009 | 2 |
| 50 | 0 | 2009 | 3 |
| 60 | 0 | 2009 | 3 |
| 70 | 0 | 2009 | 3 |
| 75 | 0 | 2009 | 4 |
| 85 | 0 | 2009 | 4 |
| 95 | 0 | 2009 | 4 |
+----+---------+------+-------+

 

Query against functional database.

Query: select id, int_col, year, month 
 from functional.alltypessmall s 
 where s.int_col = (select count(*) from functional.alltypestiny t where s.id = t.id) 
 order by id
Query submitted at: 2020-10-18 12:35:24 (Coordinator: http://qifan-10229:25000)
Query progress can be monitored at: http://qifan-10229:25000/query_plan?query_id=104bd5d7a6d5fe74:09a6c09000000000
+----+---------+------+-------+
| id | int_col | year | month |
+----+---------+------+-------+
| 1 | 1 | 2009 | 1 |
| 10 | 0 | 2009 | 1 |
| 20 | 0 | 2009 | 1 |
| 25 | 0 | 2009 | 2 |
| 35 | 0 | 2009 | 2 |
| 45 | 0 | 2009 | 2 |
| 50 | 0 | 2009 | 3 |
| 60 | 0 | 2009 | 3 |
| 70 | 0 | 2009 | 3 |
| 75 | 0 | 2009 | 4 |
| 85 | 0 | 2009 | 4 |
| 95 | 0 | 2009 | 4 |
+----+---------+------+-------+

Issue Links

• is caused by: IMPALA-2552 Runtime filter forwarding between operators
• is related to: IMPALA-10382 Predicate with coalesce on both sides of LOJ isn't NULL filtering