[IMPALA-5848] Account for TCMalloc overhead and client cache buffers in MemTracker

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 2.12.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-5
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 25/Aug/17 18:30
• Updated: 14/Dec/17 18:37
• Resolved: 14/Dec/17 18:37

Description

On our stress test cluster I see a significant amount of Untracked memory. I think we can account for a chunk of it, which will further help understanding the state of a cluster.

I see:

  Untracked Memory: Total=4.30 GB

I think some of this memory is TCMalloc caches and metadata (I'm not sure exactly which is counted)

MALLOC:    61821970968 (58958.0 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    808683664 (  771.2 MiB) Bytes in central cache freelist
MALLOC: +      3794688 (    3.6 MiB) Bytes in transfer cache freelist
MALLOC: +    520586840 (  496.5 MiB) Bytes in thread cache freelists
MALLOC: +    182366400 (  173.9 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  63337402560 (60403.3 MiB) Actual memory used (physical + swap)
MALLOC: +  45341532160 (43241.1 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: = 108678934720 (103644.3 MiB) Virtual address space used
MALLOC:
MALLOC:         368033              Spans in use
MALLOC:           3511              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

The client cache also is likely taking up a significant amount of memory, since I believe the clients all have buffers attached:

impala-server.backends.client-cache.total-clients 	1006