[SPARK-2610] When spark.serializer is set as org.apache.spark.serializer.KryoSerializer, importing a method causes multiple spark applications creations

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.0.1
• Component/s: Spark Shell
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 21/Jul/14 21:36
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

To reproduce, set

spark.serializer        org.apache.spark.serializer.KryoSerializer

in conf/spark-defaults.conf and launch a spark shell.
Then, execute

class X() { println("What!"); def y = 3 }
val x = new X
import x.y

case class Person(name: String, age: Int)

val serializer = org.apache.spark.serializer.Serializer.getSerializer(null)
val kryoSerializer = serializer.newInstance

val value = kryoSerializer.serialize(Person("abc", 1))
kryoSerializer.deserialize(value): Person
// Once you execute this line, you will see ...
// What!
// What!
// res1: Person = Person(abc,1)

Basically, importing a method of a class causes the constructor of that class been called twice.

It affects our branch 1.0 and master.
For the master, you can use

val serializer = org.apache.spark.serializer.Serializer.getSerializer(None)

to get the serializer.

Issue Links

• relates to: SPARK-2632 Importing a method of class in Spark REPL causes the REPL to pulls in unnecessary stuff.
• relates to: SPARK-7043 KryoSerializer cannot be used with REPL to interpret code in which case class definition and its shipping are in the same line