[SPARK-23431] Expose the new executor memory metrics at the stage level

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Terry Kim
• Reporter: Edward Lu
• Created: 14/Feb/18 23:46
• Updated: 14/Jan/21 06:21
• Resolved: 04/Aug/20 13:12

Description

Collect and show the new executor memory metrics for each stage, to provide more information on how memory is used per stage.

Modify the AppStatusListener to track the peak values for JVM used memory, execution memory, storage memory, and unified memory for each executor for each stage.

This is a subtask for SPARK-23206. Please refer to the design doc for that ticket for more details.

Issue Links

• links to: [Github] Pull Request #29020 (imback82)
• links to: [Github] Pull Request #29020 (imback82)
• links to: GitHub Pull Request #23340