[SPARK-10620] Look into whether accumulator mechanism can replace TaskMetrics

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Andrew Or
• Reporter: Patrick Wendell
• Created: 15/Sep/15 16:31
• Updated: 01/May/16 23:40
• Resolved: 30/Jan/16 01:16

Description

This task is simply to explore whether the internal representation used by TaskMetrics could be performed by using accumulators rather than having two separate mechanisms. Note that we need to continue to preserve the existing "Task Metric" data structures that are exposed to users through event logs etc. The question is can we use a single internal codepath and perhaps make this easier to extend in the future.

I think a full exploration would answer the following questions:

• How do the semantics of accumulators on stage retries differ from aggregate TaskMetrics for a stage? Could we implement clearer retry semantics for internal accumulators to allow them to be the same - for instance, zeroing accumulator values if a stage is retried (see discussion here: SPARK-10042).
• Are there metrics that do not fit well into the accumulator model, or would be difficult to update as an accumulator.
• If we expose metrics through accumulators in the future rather than continuing to add fields to TaskMetrics, what is the best way to coerce compatibility?
• Are there any other considerations?
• Is it worth it to do this, or is the consolidation too complicated to justify?

Issue Links

• relates to: SPARK-14626 Simplify accumulators and task metrics
• links to: [Github] Pull Request #10717 (andrewor14)
• links to: [Github] Pull Request #10958 (andrewor14)

Sub-Tasks

• Move *Metrics and *Accum* classes to their own files: Resolved
• Rename 3 fields in ShuffleWriteMetrics: Resolved
• Do not expose var's in TaskMetrics: Resolved
• Implement TaskMetrics using accumulators: Resolved
• Send only accumulator updates, not TaskMetrics, to the driver: Resolved