[IMPALA-5658] Report details of process memory maps via metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: supportability
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-1
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 12/Jul/17 23:22
• Updated: 03/Aug/17 11:11
• Resolved: 02/Aug/17 05:29

Description

To help understand the details of process memory consumption, it would be great to have metrics that answered questions like:

• How many anonymous huge pages are allocated?
• How many memory maps is the process using?
• What is the RSS of the process?
• What is the virtual memory consumption of the process?

It looks like /proc/<pid>/smaps is the documented interface for this: https://www.kernel.org/doc/Documentation/filesystems/proc.txt

I'm thinking that maybe we could have the memory maintenance thread periodically scrape /proc/<pid>/smaps and update a set of metrics like:

• Number of mappings
• Total virtual address space
• Total RSS
• Total bytes of anon huge pages

We should also report the transparent huge pages config under:
/sys/kernel/mm/transparent_hugepage/ (or /sys/kernel/mm/redhat_transparent_hugepage if not present).

Issue Links

• breaks: IMPALA-5756 Impala crashes on startup in impala::SpinLock::lock()