[SPARK-8735] Expose metrics for runtime memory usage

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Spark Core, SQL
• Target Version/s: 1.5.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 30/Jun/15 18:30
• Updated: 03/Aug/15 21:22
• Resolved: 03/Aug/15 21:22

Description

Spark has many uses of memory: caching, shuffle, metadata etc. It is useful for users to be able to drill down on the internal memory allocation for memory-intensive operations like aggregations and joins. The goal is to do this for both tungsten and non-tungsten applications.

Issue Links

• relates to: SPARK-9103 Tracking spark's memory usage
• links to: [Github] Pull Request #7770 (andrewor14)