Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23697

Accumulators of Spark 1.x no longer work with Spark 2.x

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 2.2.1, 2.3.0
    • Fix Version/s: 2.0.3, 2.1.3, 2.2.2, 2.3.1, 2.4.0
    • Component/s: Spark Core
    • Labels:
      None
    • Environment:

      Spark 2.2.0
      Scala 2.11

      Description

      I've noticed that accumulators of Spark 1.x no longer work with Spark 2.x failing with

      java.lang.AssertionError: assertion failed: copyAndReset must return a zero value copy

      It happens while serializing an accumulator here

      val copyAcc = copyAndReset()
      assert(copyAcc.isZero, "copyAndReset must return a zero value copy")

      ... although copyAndReset returns zero-value copy for sure, just consider the accumulator below

      val concatParam = new AccumulatorParam[jl.StringBuilder] {
        override def zero(initialValue: jl.StringBuilder): jl.StringBuilder = new jl.StringBuilder()
        override def addInPlace(r1: jl.StringBuilder, r2: jl.StringBuilder): jl.StringBuilder = r1.append(r2)
      }

      So, Spark treats zero value as non-zero due to how isZero is implemented in LegacyAccumulatorWrapper.

      override def isZero: Boolean = _value == param.zero(initialValue)

      All this means that the values to be accumulated must implement equals and hashCode, otherwise isZero is very likely to always return false.

      So I'm wondering whether the assertion 

      assert(copyAcc.isZero, "copyAndReset must return a zero value copy")

      is really necessary and whether it can be safely removed from there?

      If not - is it ok to just override writeReplace for LegacyAccumulatorWrapper to prevent such failures?

        Attachments

          Activity

            People

            • Assignee:
              cloud_fan Wenchen Fan
              Reporter:
              szhemzhitsky Sergey Zhemzhitsky
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: