Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5885

Parquet scanner does not free local allocations in filter contexts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
    • Fix Version/s: Impala 2.11.0
    • Component/s: Backend

      Description

      This problem can occur if runtime filter expressions that are evaluated in the scan allocate temporary memory - "local allocations". These accumulate for each scan range and are only
      freed upon scan range completion.

      A contrived query that exhibits the problem is the following. If I continue adding upper() and lower() to the expression the memory consumption of the scan node will continue to grow - up to 100MB for each extra function call!

      set runtime_filter_wait_time_ms=1000000;
      select straight_join count(*) from tpch_parquet.lineitem l1 join tpch_parquet.lineitem l2 on upper(lower(upper(lower(l1.l_comment)))) = concat(l2.l_comment, 'foo');
      summary;
      

      I think other conjuncts in the scanner may be affected by the same problem, e.g. the min_max conjuncts.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: