[SPARK-18462] SparkListenerDriverAccumUpdates event does not deserialize properly in history server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.3, 2.1.0
• Component/s: SQL
• Target Version/s: 2.0.3, 2.1.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 16/Nov/16 01:23
• Updated: 18/Nov/16 02:45
• Resolved: 18/Nov/16 02:45

Description

The following test fails with a ClassCastException due to oddities in how Jackson object mapping works, breaking the SQL tab in the history server:

+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.ui

 import java.util.Properties

+import org.json4s.jackson.JsonMethods._
 import org.mockito.Mockito.mock

 import org.apache.spark._
@@ -35,7 +36,7 @@ import org.apache.spark.sql.execution.{LeafExecNode, QueryExecution, SparkPlanIn
 import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
 import org.apache.spark.sql.test.SharedSQLContext
 import org.apache.spark.ui.SparkUI
-import org.apache.spark.util.{AccumulatorMetadata, LongAccumulator}
+import org.apache.spark.util.{AccumulatorMetadata, JsonProtocol, LongAccumulator}


 class SQLListenerSuite extends SparkFunSuite with SharedSQLContext {
@@ -416,6 +417,20 @@ class SQLListenerSuite extends SparkFunSuite with SharedSQLContext {
     assert(driverUpdates(physicalPlan.longMetric("dummy").id) == expectedAccumValue)
   }

+  test("roundtripping SparkListenerDriverAccumUpdates through JsonProtocol") {
+    val event = SparkListenerDriverAccumUpdates(1L, Seq((2L, 3L)))
+    val actualJsonString = compact(render(JsonProtocol.sparkEventToJson(event)))
+    val newEvent = JsonProtocol.sparkEventFromJson(parse(actualJsonString))
+    newEvent match {
+      case SparkListenerDriverAccumUpdates(executionId, accums) =>
+        assert(executionId == 1L)
+        accums.foreach { case (a, b) =>
+          assert(a == 2L)
+          assert(b == 3L)
+        }
+    }
+  }
+

Issue Links

• links to: [Github] Pull Request #15922 (JoshRosen)