Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3628

Don't apply accumulator updates multiple times for tasks in result stages

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 1.2.0
    • Spark Core
    • None

    Description

      In previous versions of Spark, accumulator updates only got applied once for accumulators that are only used in actions (i.e. result stages), letting you use them to deterministically compute a result. Unfortunately, this got broken in some recent refactorings.

      This is related to https://issues.apache.org/jira/browse/SPARK-732, but that issue is about applying the same semantics to intermediate stages too, which is more work and may not be what we want for debugging.

      Attachments

        Issue Links

          Activity

            People

              codingcat Nan Zhu
              matei Matei Alexandru Zaharia
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: