[SAMZA-972] Add holistic memory monitoring for SamzaContainer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Assignee: Jagadish
• Reporter: Jagadish
• Created: 11/Jul/16 06:55
• Updated: 27/Jul/16 22:38
• Resolved: 27/Jul/16 22:38

Description

This feature introduces physical memory monitoring in SamzaContainer.

Context:
Often memory used by the SamzaContainer process includes
A. JVM Heap memory: This is where all JVM variables live.
B. Native memory: This memory lives out of the JVM heap and is not visible to the JVM. Examples include used by RocksDb, native libraries that user code depends on etc.

User jobs could be killed by Yarn if their total memory (A+B) exceeds the configured maximum of yarn.container.memory.mb.

Currently, while our existing metrics provide visibility into [A] via JMX, we don't have visibility into [B]. (as it's totally external to the JVM).