[SPARK-16861] Refactor PySpark accumulator API to be on top of AccumulatorV2 API

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: PySpark, Spark Core
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 02/Aug/16 21:56
• Updated: 23/Sep/16 08:45
• Resolved: 23/Sep/16 08:44

Description

The PySpark accumulator API is implemented on top of the legacy accumulator API which is now deprecated. To allow this deprecated API to be removed eventually we will need to rewrite the PySpark accumulators on top of the new API. An open question for a possible follow up is if we want to make the Python accumulator API look more like the new accumulator API.

Issue Links

• Is contained by: SPARK-16775 Remove deprecated accumulator v1 APIs
• is duplicated by: SPARK-16580 [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2
• links to: [Github] Pull Request #14467 (holdenk)