[SPARK-1813] Add a utility to SparkConf that makes using Kryo really easy

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 13/May/14 00:19
• Updated: 01/Nov/14 15:07
• Resolved: 22/Oct/14 04:53

Description

It would be nice to have a method in SparkConf that makes it really easy to turn on Kryo serialization and register a set of classes.

Using Kryo currently requires all this:

import com.esotericsoftware.kryo.Kryo
import org.apache.spark.serializer.KryoRegistrator

class MyRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo) {
    kryo.register(classOf[MyClass1])
    kryo.register(classOf[MyClass2])
  }
}

val conf = new SparkConf().setMaster(...).setAppName(...)
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.kryo.registrator", "mypackage.MyRegistrator")
val sc = new SparkContext(conf)

It would be nice if it just required this:

SparkConf.setKryo(Array(classOf[MyClass1], classOf[MyClass2]))

Issue Links

• is duplicated by: SPARK-4164 spark.kryo.registrator shall use comma separated value to support multiple registrator
• links to: [Github] Pull Request #789 (sryza)