Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3628

Don't apply accumulator updates multiple times for tasks in result stages

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In previous versions of Spark, accumulator updates only got applied once for accumulators that are only used in actions (i.e. result stages), letting you use them to deterministically compute a result. Unfortunately, this got broken in some recent refactorings.

      This is related to https://issues.apache.org/jira/browse/SPARK-732, but that issue is about applying the same semantics to intermediate stages too, which is more work and may not be what we want for debugging.

        Attachments

          Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-732 Recomputation of RDDs may result in duplicated accumulator updates

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          links to

          Pull request #2524 [Github] Pull Request #2524 (CodingCat)

            Activity

              People

              • Assignee:
                codingcat Nan Zhu
                Reporter:
                matei Matei Alexandru Zaharia
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: