[IMPALA-3204] Integrate buffer pool reservations with memtrackers

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.8.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 18/Mar/16 00:20
• Updated: 12/Sep/16 16:09
• Resolved: 12/Sep/16 16:09

Description

The reservation mechanism for the buffer pool introduces a memory tracking mechanism that is somewhat parallel to the existing MemTracker mechanism. This task is to integrate the two so that:

• Query memory limits include buffer reservations
• Runtime profiles and other diagnostics include buffer pool usage in the memory consumption, down to the exec node level.

My initial thought is that each reservation tracker should optionally have an associated MemTracker. When reservations are granted, it would be counted towards the memory consumption of the tracker. I think we want to count reservations rather than usage, because the MemTrackers are really more geared towards tracking committed memory rather than used memory.