[SPARK-17042] Repl-defined classes cannot be replicated - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.0.1, 2.1.0
Component/s: Block Manager, Spark Core
Labels:
None

Description

A simple fix is to erase the classTag when using the default serializer, since it's not needed in that case, and the classTag was failing to deserialize on the remote end.

The proper fix is actually to use the right classloader when deserializing the classtags, but that is a much more invasive change for 2.0.

The following test can be added to ReplSuite to reproduce the bug:

  test("replicating blocks of object with class defined in repl") {
    val output = runInterpreter("local-cluster[2,1,1024]",
      """
        |import org.apache.spark.storage.StorageLevel._
        |case class Foo(i: Int)
        |val ret = sc.parallelize((1 to 100).map(Foo), 10).persist(MEMORY_ONLY_2)
        |ret.count()
        |sc.getExecutorStorageStatus.map(s => s.rddBlocksById(ret.id).size).sum
      """.stripMargin)
    assertDoesNotContain("error:", output)
    assertDoesNotContain("Exception", output)
    assertContains(": Int = 20", output)
  }