Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18462

SparkListenerDriverAccumUpdates event does not deserialize properly in history server

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.3, 2.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      The following test fails with a ClassCastException due to oddities in how Jackson object mapping works, breaking the SQL tab in the history server:

      +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
      @@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.ui
      
       import java.util.Properties
      
      +import org.json4s.jackson.JsonMethods._
       import org.mockito.Mockito.mock
      
       import org.apache.spark._
      @@ -35,7 +36,7 @@ import org.apache.spark.sql.execution.{LeafExecNode, QueryExecution, SparkPlanIn
       import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
       import org.apache.spark.sql.test.SharedSQLContext
       import org.apache.spark.ui.SparkUI
      -import org.apache.spark.util.{AccumulatorMetadata, LongAccumulator}
      +import org.apache.spark.util.{AccumulatorMetadata, JsonProtocol, LongAccumulator}
      
      
       class SQLListenerSuite extends SparkFunSuite with SharedSQLContext {
      @@ -416,6 +417,20 @@ class SQLListenerSuite extends SparkFunSuite with SharedSQLContext {
           assert(driverUpdates(physicalPlan.longMetric("dummy").id) == expectedAccumValue)
         }
      
      +  test("roundtripping SparkListenerDriverAccumUpdates through JsonProtocol") {
      +    val event = SparkListenerDriverAccumUpdates(1L, Seq((2L, 3L)))
      +    val actualJsonString = compact(render(JsonProtocol.sparkEventToJson(event)))
      +    val newEvent = JsonProtocol.sparkEventFromJson(parse(actualJsonString))
      +    newEvent match {
      +      case SparkListenerDriverAccumUpdates(executionId, accums) =>
      +        assert(executionId == 1L)
      +        accums.foreach { case (a, b) =>
      +          assert(a == 2L)
      +          assert(b == 3L)
      +        }
      +    }
      +  }
      +
      

        Attachments

          Activity

            People

            • Assignee:
              joshrosen Josh Rosen
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: