[SPARK-24958] Add executors' process tree total memory information to heartbeat signals

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Reza Safi
• Reporter: Reza Safi
• Created: 28/Jul/18 06:45
• Updated: 13/Dec/18 06:04
• Resolved: 10/Dec/18 17:15

Description

Spark executors' process tree total memory information can be really useful. Currently such information are not available. The goal of this task is to compute such information for each executor, add these information to the heartbeat signals, and compute the peaks at the driver. 

Issue Links

• is a parent of: SPARK-26357 Expose executors' procfs metrics to Metrics system
• is duplicated by: SPARK-21157 Report Total Memory Used by Spark Executors
• links to: [Github] Pull Request #21916 (rezasafi)
• links to: [Github] Pull Request #22612 (rezasafi)
• links to: [Github] Pull Request #22612 (rezasafi)
• links to: GitHub Pull Request #22612