[DRILL-7372] MethodAnalyzer consumes too much memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0
• Labels: ready-to-commit
• Assignee: Vova Vysotskyi
• Reporter: Vova Vysotskyi
• Reviewer: Arina Ielchiieva
• Created: 11/Sep/19 14:06
• Updated: 14/Nov/19 17:22
• Resolved: 14/Nov/19 17:22

Description

In the scope of DRILL-6524 was added logic for determining whether a variable is assigned in conditional block to prevent incorrect scalar replacement for such cases.

But for some queries, this logic consumes too many memory, for example, for the following query:

SELECT *
FROM cp.`employee.json`
WHERE employee_id+0 < employee_id
  OR employee_id+1 < employee_id
  AND employee_id+2 < employee_id
  OR employee_id+3 < employee_id
  AND employee_id+4 < employee_id
  OR employee_id+5 < employee_id
  AND employee_id+6 < employee_id
  OR employee_id+7 < employee_id
  AND employee_id+8 < employee_id
  OR employee_id+9 < employee_id
  AND employee_id+10 < employee_id
  OR employee_id+11 < employee_id
  AND employee_id+12 < employee_id
  OR employee_id+13 < employee_id
  AND employee_id+14 < employee_id
  OR employee_id+15 < employee_id
  AND employee_id+16 < employee_id
  OR employee_id+17 < employee_id
  AND employee_id+18 < employee_id
  OR employee_id+19 < employee_id
  AND employee_id+20 < employee_id
  OR employee_id+21 < employee_id
  AND employee_id+22 < employee_id
  OR employee_id+23 < employee_id
  AND employee_id+24 < employee_id
  OR employee_id+25 < employee_id
  AND employee_id+26 < employee_id
  OR employee_id+27 < employee_id
  AND employee_id+28 < employee_id
  OR employee_id+29 < employee_id
  AND employee_id+30 < employee_id
  OR employee_id+31 < employee_id
  AND employee_id+32 < employee_id
  OR employee_id+33 < employee_id
  AND employee_id+34 < employee_id
  OR employee_id+35 < employee_id
  AND employee_id+36 < employee_id
  OR employee_id+37 < employee_id
  AND employee_id+38 < employee_id
  OR employee_id+39 < employee_id
  AND employee_id+40 < employee_id
  OR employee_id+41 < employee_id
  AND employee_id+42 < employee_id
  OR employee_id+43 < employee_id
  AND employee_id+44 < employee_id
  OR employee_id+45 < employee_id
  AND employee_id+46 < employee_id
  OR employee_id+47 < employee_id
  AND employee_id+48 < employee_id
  OR employee_id+49 < employee_id
  AND TRUE;

Drill consumes more than 6 GB memory.

One of the issues to fix is to replace Deque<Set<Integer>> localVariablesSet; with Deque<BitSet>, it will reduce memory usage significantly.
Additionally should be investigated why these objects cannot be collected by GC.

Issue Links

• links to: GitHub Pull Request #1887