[IMPALA-5073] Considering bypassing TCMalloc by default for buffer pool

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Product Backlog
• Epic Color: ghx-label-4
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 14/Mar/17 20:32
• Updated: 02/May/19 16:09

Description

There would be some advantages to switch from from allocating buffers via TCMalloc and instead using mmap directly - e.g. less contention for the page heap lock.

There are also downsides - virtual memory consumption could increase and we may end up mapping and unmapping memory more frequently.

We would also need to wire up the MemTrackers so they include this memory in the process estimate.

Issue Links

• is blocked by: IMPALA-3203 Implement scalable buffer recycling in buffer pool
• is depended upon by: IMPALA-2800 impalad process blocked while releasing memory after a big query
• is related to: IMPALA-5229 Try using TCMalloc + Huge Pages for buffers