[SPARK-14626] Simplify accumulators and task metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 14/Apr/16 06:46
• Updated: 01/May/16 23:40
• Resolved: 28/Apr/16 07:26

Description

The goal of the ticket is to simplify both the external interface and the internal implementation for accumulators and metrics. They are unnecessarily convoluted and we should be able to simplify them quite a bit.

This is an umbrella ticket and I will iteratively create new tasks as my investigation goes on. At a high level, I'd would like to create better abstractions for internal implementations, as well as creating a simplified accumulator v2 external interface that doesn't involve a complex type hierarchy.

Issue Links

• is related to: SPARK-10620 Look into whether accumulator mechanism can replace TaskMetrics