[IMPALA-2800] impalad process blocked while releasing memory after a big query

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: kudu performance resource-management
• Target Version: Impala 2.9.0
• Assignee: Tim Armstrong
• Reporter: Todd Lipcon
• Created: 23/Dec/15 22:15
• Updated: 17/Apr/17 22:58
• Resolved: 17/Apr/17 22:58

Description

After running a large join, the impalad process might have allocated tens of GB of memory. When the query finishes, all of this memory is deallocated, and the background MaintenanceThread sees that the heap size is much larger than the actual in-use memory, and calls MallocExtension::instance()->ReleaseFreeMemory(). Inside tcmalloc, this holds a global heap lock while unmapping all of the requested memory.

In one case on my cluster, releasing ~100G of memory caused the tcmalloc page lock to be held for more than 15 seconds. This caused another query to fail with Kudu scan timeouts. Outside the context of Kudu, it's likely still problematic for latency-sensitive SLAs.

Issue Links

• depends upon: IMPALA-5073 Considering bypassing TCMalloc by default for buffer pool
• is blocked by: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
• is related to: IMPALA-3162 Upgrade gperfutils