[SPARK-14626] Simplify accumulators and task metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 14/Apr/16 06:46
• Updated: 01/May/16 23:40
• Resolved: 28/Apr/16 07:26

Description

The goal of the ticket is to simplify both the external interface and the internal implementation for accumulators and metrics. They are unnecessarily convoluted and we should be able to simplify them quite a bit.

This is an umbrella ticket and I will iteratively create new tasks as my investigation goes on. At a high level, I'd would like to create better abstractions for internal implementations, as well as creating a simplified accumulator v2 external interface that doesn't involve a complex type hierarchy.

Issue Links

• is related to: SPARK-10620 Look into whether accumulator mechanism can replace TaskMetrics

Sub-Tasks

• TaskUIData and ExecutorUIData shouldn't be case classes: Resolved
• Remove deprecated APIs in TaskMetrics: Resolved
• Track internal accumulators (metrics) by stage attempt rather than stage: Resolved
• Remove all the Options in TaskMetrics: Resolved
• New accumulator API: Resolved
• create accumulators in TaskMetrics: Resolved
• remove internal flag in Accumulable: Resolved
• Rename NewAccumulator to AccumulatorV2: Resolved
• Deprecate old accumulator API: Resolved
• Remove setValue method on accumulators: Closed
• Support average/count/sum in LongAccumulator/DoubleAccumulator: Resolved
• Break copyAndReset into copy and reset: Resolved
• Move AccumulatorV2 and subclasses into util package: Resolved
• Improve unit test coverage for AccumulatorV2: Resolved
• Update Java API once the Scala one is finalized: Resolved
• Remove AccumulatorV2.localValue and keep only value: Resolved
• Update spark examples with AccumulatorV2: Resolved
• Update document programming-guide accumulator section: Closed