[IMPALA-8020] Process memory consumption does not include unused reservations

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Backend
• Labels: resource-management
• Target Version: Product Backlog
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 27/Dec/18 18:07
• Updated: 25/Feb/19 06:57

Description

This is being split out from IMPALA-7446, since it's a somewhat separate issue.

The issue is that the process memory consumption only accounts for system memory allocated from the buffer pool, not for reservations that have not been fulfilled yet. Reserved allocations do not check the process memory limit before going through, so it's possible that the same memory could be committed both to a non-reserved allocation and a reserved allocation.

In practice this seems unlikely to cause any real symptoms - it would require:

• the process to be near its memory limit
• a large amount of unused reservation, not offset by free buffers or clean pages.
• a quick spike in the amount of reservation used

This could result in the process going significantly over its memory limit before queries start to fail and free memory.

Issue Links

• is related to: IMPALA-7446 Queries can spill earlier than necessary because of accumulation of free buffers and clean pages