[SPARK-3628] Don't apply accumulator updates multiple times for tasks in result stages

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Assignee: Nan Zhu
• Reporter: Matei Alexandru Zaharia
• Created: 21/Sep/14 22:38
• Updated: 25/Mar/15 16:33
• Resolved: 25/Mar/15 16:33

Description

In previous versions of Spark, accumulator updates only got applied once for accumulators that are only used in actions (i.e. result stages), letting you use them to deterministically compute a result. Unfortunately, this got broken in some recent refactorings.

This is related to https://issues.apache.org/jira/browse/SPARK-732, but that issue is about applying the same semantics to intermediate stages too, which is more work and may not be what we want for debugging.

Issue Links

• relates to: SPARK-732 Recomputation of RDDs may result in duplicated accumulator updates
• links to: [Github] Pull Request #2524 (CodingCat)