[SPARK-23910] Publish executor memory utilisation in heartbeat events

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Gopala
• Created: 09/Apr/18 06:25
• Updated: 08/Oct/19 05:41
• Resolved: 08/Oct/19 05:41

Description

Is it possible to send the executor memory utilisation in the heart beat events? 
While this detail is pretty easily available in Map Reduce, there doesn't seem to be a way to get this inclusive of all types of memory segments used by Spark ( memory used for storage can be determined, but not that used for execution ) . 
We find this feature very critical to determine the memory utilisation and plan the resources appropriately.