[IMPALA-5885] Parquet scanner does not free local allocations in filter contexts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.11.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 31/Aug/17 22:25
• Updated: 30/Aug/18 18:36
• Resolved: 06/Sep/17 00:15

Description

This problem can occur if runtime filter expressions that are evaluated in the scan allocate temporary memory - "local allocations". These accumulate for each scan range and are only
freed upon scan range completion.

A contrived query that exhibits the problem is the following. If I continue adding upper() and lower() to the expression the memory consumption of the scan node will continue to grow - up to 100MB for each extra function call!

set runtime_filter_wait_time_ms=1000000;
select straight_join count(*) from tpch_parquet.lineitem l1 join tpch_parquet.lineitem l2 on upper(lower(upper(lower(l1.l_comment)))) = concat(l2.l_comment, 'foo');
summary;

I think other conjuncts in the scanner may be affected by the same problem, e.g. the min_max conjuncts.