[SPARK-17019] Expose off-heap memory usage in various places

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Target Version/s: 2.2.0
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 11/Aug/16 08:45
• Updated: 06/Apr/17 18:24
• Resolved: 06/Apr/17 18:24

Description

With SPARK-13992, Spark supports persisting data into off-heap memory, but the usage of off-heap is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:

1. Spark UI's executor page will display both on-heap and off-heap memory usage.
2. REST request returns both on-heap and off-heap memory.
3. Also these two memory usage can be obtained programmatically from SparkListener.

Issue Links

• links to: [Github] Pull Request #14617 (jerryshao)