Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3628

Don't apply accumulator updates multiple times for tasks in result stages

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In previous versions of Spark, accumulator updates only got applied once for accumulators that are only used in actions (i.e. result stages), letting you use them to deterministically compute a result. Unfortunately, this got broken in some recent refactorings.

      This is related to https://issues.apache.org/jira/browse/SPARK-732, but that issue is about applying the same semantics to intermediate stages too, which is more work and may not be what we want for debugging.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                codingcat Nan Zhu
                Reporter:
                matei Matei Alexandru Zaharia
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: