[IMPALA-3259] Codegen is not cancellable and can use a lot of CPU and memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Component/s: Backend
• Labels: codegen
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Huaisi Xu
• Created: 29/Mar/16 21:39
• Updated: 24/Dec/20 00:09
• Resolved: 24/Dec/20 00:09

Description

memz stucks at

Memory Usage
Memory consumption / limit: 10.43 GB / 20.00 GB
tcmalloc
------------------------------------------------
MALLOC:    10489890000 (10003.9 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    674292952 (  643.1 MiB) Bytes in central cache freelist
MALLOC: +     15303600 (   14.6 MiB) Bytes in transfer cache freelist
MALLOC: +     15594152 (   14.9 MiB) Bytes in thread cache freelists
MALLOC: +     51245248 (   48.9 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  11246325952 (10725.3 MiB) Actual memory used (physical + swap)
MALLOC: +   6235471872 ( 5946.6 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  17481797824 (16671.9 MiB) Virtual address space used
MALLOC:
MALLOC:         565828              Spans in use
MALLOC:            245              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

this query is cancelled because of unreachable impalad because I attached gdb..

I have flushed the query archiving cache by issuing ~30 "show tables" queries. (this works in IMPALA-3254) but the memory in this case stayed high.

I will try to reproduce and get a heap profile.

There is no query running at this moment.

Issue Links

• depends upon: IMPALA-4551 Set limits on size of expression trees
• is duplicated by: IMPALA-3290 Catch std::bad_alloc from LLVM optimisation
• is related to: IMPALA-5296 Codegen for aggregation with 1K grouping columns takes several seconds
• is related to: IMPALA-2651 codegen overhead can be high
• relates to: IMPALA-4164 Codegen does not generate target-specific machine code for cross-compiled functions