[SPARK-3465] Task metrics are not aggregated correctly in local mode

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 09/Sep/14 21:49
• Updated: 12/Sep/14 21:30
• Resolved: 12/Sep/14 21:30

Description

In local mode, after onExecutorMetricsUpdate(), t.taskMetrics will be the same object with that in TaskContext (because there is no serialization for MetricsUpdate in local mode), then all the upcoming changes in metrics will be lost, because updateAggregateMetrics() only counts the difference in these two.

This bug was introduced in https://issues.apache.org/jira/browse/SPARK-2099, cc sandyr]

Issue Links

• links to: [Github] Pull Request #2338 (davies)