[SPARK-48628] Add task peak on/off heap execution memory metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Ziqi Liu
• Reporter: Ziqi Liu
• Created: 14/Jun/24 07:45
• Updated: 21/Aug/24 17:58
• Resolved: 21/Aug/24 15:42

Description

Currently there is no task on/off heap execution memory metrics. There is a peakExecutionMemory  metrics, however, the semantic is a confusing: it only cover the execution memory used by shuffle/join/aggregate/sort, which is accumulated in specific operators.

 

We can easily maintain the whole task-level peak memory in TaskMemoryManager, assuming acquireExecutionMemory is the only one narrow waist for acquiring execution memory.

 

Also it's nice to cleanup/deprecate that poorly-named `peakExecutionMemory`.

 

Creating two followup sub tickets:

• https://issues.apache.org/jira/browse/SPARK-48788 :accumulate task metrics in stage, and display in Spark UI
• https://issues.apache.org/jira/browse/SPARK-48789 : deprecate `peakExecutionMemory` once we have replacement for it.

Issue Links

• causes: SPARK-49228 Investigate ExternalAppendOnlyUnsafeRowArrayBenchmark
• is depended upon by: SPARK-48788 Accumulate task-level execution memory in stage and display in Spark UI
• is depended upon by: SPARK-48789 Deprecate peakExecutionMemory metrics
• links to: GitHub Pull Request #47192
• links to: GitHub Pull Request #47747
• links to: GitHub Pull Request #47776