[SPARK-10543] Peak Execution Memory Quantile should be Per-task Basis

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: Spark Core
• Target Version/s: 1.5.1, 1.6.0
• Assignee: Sen Fang
• Reporter: Sen Fang
• Created: 10/Sep/15 18:57
• Updated: 14/Sep/15 22:06
• Resolved: 14/Sep/15 22:06

Description

Currently the Peak Execution Memory quantiles seem to be cumulative rather than per task basis. For example, I have seen a value of 2TB in one of my jobs on the quantile metric but each individual task shows less than 1GB on the bottom table.

andrewor14 In your PR https://github.com/apache/spark/pull/7770, the screenshot shows the Max Peak Execution Memory of 792.5KB while in the bottom it's about 50KB per task (unless your workload is skewed)

The fix seems straightforward that we use the `update` rather than `value` from the accumulable. I'm happy to provide a PR if people agree this is the right behavior.

Issue Links

• links to: [Github] Pull Request #8726 (saurfang)